KR102305673B1 - Method for predicting lane chage intention and route of surrounding vehicles using camera and v2v - Google Patents

Abstract

A system for predicting lane change intention and route of a surrounding vehicle to predict lane change intention and route of a surrounding vehicle in a vehicle according to an embodiment of the present invention comprises: a communication unit obtaining dynamic information of a surrounding vehicle through V2V communication; a sensor unit calculating bounding box coordinate information on the surrounding vehicle recognized by using a camera sensor; and a prediction unit predicting the lane change intention and route of the surrounding vehicle by using the obtained dynamic information of the surrounding vehicle and the calculated bounding box coordinate information as characteristic values of a learning model based on machine learning.

Description

Method for predicting lane change intention and route of surrounding vehicles using camera and V2V

The following description relates to a method and system for predicting a lane change intention and route of a vehicle.

Since the driving pattern of the surrounding vehicles is different according to the individual tendency such as gender, personality, and habit of the driver, the algorithm for predicting lane change intention and route uses a supervised learning algorithm during machine learning.

The existing method uses the distance between the surrounding vehicle and the lane as shown in FIG. 1 as a 'feature value' to be used in the supervised learning algorithm, or changes the lane using the relative distance, relative speed, and relative latitude between the own vehicle and the surrounding vehicle as shown in FIG. Predict whether and the path.

However, in this method, when the lane is erased or when the lane is not clearly recognized due to weather and driving environment (snow/rain/night), the lane change intention and route prediction algorithm using the distance between the surrounding vehicles and the lane is not accurate. There is a problem of falling.

In addition, in order to obtain information such as the relative distance between the own vehicle and the surrounding vehicle, the relative speed, and the relative inclination angle, the environment recognition sensor installed in the own vehicle must be used. It has the disadvantage of being demanding and being greatly influenced by the surrounding driving environment.

It is possible to provide a method and system for predicting machine learning-based driving intentions and routes by using bounding box information and relative position information of surrounding vehicles obtained through a camera sensor and V2X communication.

By using the object bounding box coordinates obtained from the camera sensor as a feature value in a machine learning-based prediction algorithm, it is possible to provide a method and system for predicting the lane change intention and path of a surrounding vehicle.

A system for predicting a lane change intention and a path of a surrounding vehicle in a vehicle for predicting a lane change intention and a path of a surrounding vehicle includes: a communication unit configured to obtain dynamic information of a surrounding vehicle through V2V communication; a sensor unit for calculating bounding box information on the recognized surrounding vehicle using a camera sensor; and a predictor for predicting the lane change intention and route of the surrounding vehicle by using the obtained dynamic information of the surrounding vehicle and the calculated bounding box information as a feature value of a machine learning-based learning model.

The prediction unit learns the learning model by using the obtained dynamic information of the surrounding vehicle and the calculated bounding box information as a feature value of training data, and predicts the lane change intention and route of the surrounding vehicle in the learned learning model and outputting a learning result including an intention to change a lane of a surrounding vehicle or a path prediction of a surrounding vehicle according to input of dynamic information and bounding box coordinate information of the desired surrounding vehicle, wherein the learning model is SVM (Support Vector Machine) , predicts lane change intention of surrounding vehicles using supervised learning algorithms including , RF (Random Forest), and time series data prediction algorithms including RNN (Recurrent Neural Networks) and LSTM (Long Short-Term Memory) The path of nearby vehicles can be predicted.

The prediction unit uses the coordinate information of the bounding box as a feature value of the learning model by using the characteristic of the movement information transition of the bounding box that is moved as the lane of the surrounding vehicle changes, and changes according to the relative distance with the surrounding vehicle Coordinate information of the bounding box may be used as a feature value of the learning model by using the characteristic of the size information of the bounding box.

The sensor unit defines the front screen as (0,0) for the left bottom and (1,1) for the right top of the front screen using the camera sensor, and uses the object recognition algorithm to define the area around the road. The vehicle may be recognized, and bounding box coordinate information of the recognized surrounding vehicle may be calculated.

The communication unit, through the V2V communication, latitude (Latitude), longitude (Longitude), heading (Heading), yaw rate (Yaw rate) of the surrounding vehicle can obtain dynamic information of the surrounding vehicle including values.

The prediction unit predicts the lane change intention of the surrounding vehicle using a supervised learning algorithm including a support vector machine (SVM) and a random forest (RF), and uses Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) It is possible to predict the path of the surrounding vehicle using a time series data prediction algorithm including

By using the object bounding box coordinate information obtained from the camera sensor, it is possible to improve the prediction accuracy in a road environment with an unclear lane and a curved road.

In addition, by using the bounding box obtained through the camera sensor, the change of the position of the surrounding vehicle is not affected by the surrounding environment because the intention and route of the lane change are predicted through the change of the coordinate value.

In addition, it is possible to reduce the amount of calculation for predicting the lane change intention and route of the surrounding vehicle by using the relative distance to the surrounding vehicle and the bounding box for the surrounding vehicle through V2V communication and Carrera sensor fusion.

1 and 2 are examples for explaining whether or not to change a lane of a surrounding vehicle and predicting a route.
3 is a block diagram illustrating a configuration that a processor of a lane change intention and route prediction system according to an embodiment may include.
4 is a flowchart illustrating a method of predicting a lane change intention and a route of a neighboring vehicle using a camera and V2V performed in the lane change intention and route prediction system according to an embodiment.
5 is an example for explaining an object bounding box in a system for predicting a lane change intention and a path according to an embodiment.
6 is an example for explaining a camera sensor coordinate system in a system for predicting a lane change intention and a route according to an embodiment.
7 is an example for explaining predicting a lane change intention of a surrounding vehicle in the lane change intention and route prediction system according to an embodiment.
8 is an example for explaining predicting a path of a surrounding vehicle in the lane change intention and path prediction system according to an embodiment.
9 is a diagram for describing learning data for lane change prediction in the lane change intention and route prediction system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

3 is a block diagram illustrating a configuration that a processor of a lane change intention and path prediction system according to an embodiment may include.

The lane change intention and route prediction system (hereinafter, referred to as a system) 300 is for predicting the lane change intention and route of surrounding vehicles, and includes a communication unit 310, a sensor unit 320 and a prediction unit ( 330) may be included.

The communication unit 310 may acquire dynamic information of the surrounding vehicle through V2V communication. The communication unit 310 may obtain dynamic information of the surrounding vehicle including the latitude, longitude, heading, and yaw rate values of the surrounding vehicle through V2V communication.

The sensor unit 320 may calculate bounding box information about the recognized surrounding vehicle using the camera sensor. The sensor unit 320 uses a camera sensor to define the front screen as (0,0) for the left bottom and (1,1) for the right top, and uses an object recognition algorithm to It is possible to recognize the surrounding vehicle, and calculate bounding box coordinate information of the recognized surrounding vehicle.

The prediction unit 330 may predict the lane change intention and route of the surrounding vehicle by using the obtained dynamic information of the surrounding vehicle and the calculated bounding box information as a feature value of the machine learning-based learning model. The prediction unit 330 trains a learning model by using the acquired dynamic information of the surrounding vehicle and the calculated bounding box information as a feature value of the training data, and predicts the lane change intention and route of the surrounding vehicle in the learned learning model. It may include outputting a learning result including an intention to change a lane of a surrounding vehicle or a path prediction of a surrounding vehicle as the dynamic information of the surrounding vehicle and the bounding box coordinate information are input. At this time, the learning model predicts the lane change intention of surrounding vehicles using a supervised learning algorithm including a support vector machine (SVM) and a random forest (RF), and uses Recurrent Neural Networks (RNN), Long Short-Term (LSTM) Memory) can be used to predict the path of a nearby vehicle using a time-series data prediction algorithm. The prediction unit 330 uses the coordinate information of the bounding box as a feature value of the learning model by using the characteristics of the movement information transition of the bounding box that is moved as the lane of the surrounding vehicle changes, and according to the relative distance with the surrounding vehicle Coordinate information of the bounding box may be used as a feature value of the learning model by using the characteristics of the changing size information of the bounding box.

4 is a flowchart for explaining a method of predicting a lane change intention and route of a neighboring vehicle using a camera and V2V performed in the lane change intention and route prediction system according to an embodiment.

The system may acquire at least one or more surrounding vehicle information related to the surrounding vehicle from the vehicle. The system may learn by inputting the obtained at least one or more surrounding vehicle information into a learning model configured for lane change intention and route prediction. As a result of learning, the system can predict the driving intention and route of surrounding vehicles.

For example, sensor information and dynamic information related to the vehicle and surrounding vehicles may be acquired in the vehicle. In an embodiment, V2V communication between vehicles is possible, and a surrounding vehicle may be recognized based on the vehicle through a camera sensor. For example, a neighboring vehicle existing in a preset range (eg, a 360 degree range) with respect to the vehicle may be recognized. In this case, there are two types of camera sensors: a single camera sensor and a stereo camera sensor. A two-dimensional object on a plane may be recognized through a single camera, and a three-dimensional object may be recognized using a visual difference between two lenses through a stereo camera. For example, the camera sensor may be mounted inside the vehicle under the windshield in the vehicle, and may be mounted in the vicinity of the rearview mirror. A field of view in front of the vehicle may be captured through the camera sensor, and an object (eg, a vehicle) existing within the field of view may be recognized. In this case, in preparation for the external environment of the vehicle, the camera sensor may be mounted inside the vehicle corresponding to an area to be cleaned by a wiper driven outside the windshield. In addition, the position where the camera sensor is mounted is not limited. In addition, at least one camera sensor may be used to photograph the front, side, and rear of the vehicle, and may be installed at different positions.

Specifically, the system may obtain dynamic information of the surrounding vehicle through V2V communication ( 410 ), and may calculate bounding box coordinate information for the recognized surrounding vehicle using a camera sensor ( 411 ). In this case, the surrounding vehicle may be recognized using the camera sensor, and bounding box coordinate information including the recognized surrounding vehicle may be calculated. Steps 410 and 411 may be performed simultaneously or with a time difference. As shown in Table 1, the vehicle-to-vehicle communication information using V2V follows the BSM Set defined in the SAE J2735 standard. In the embodiment, latitude, longitude, heading, and yaw rate values among BSM part 1 may be used as training data feature values. For example, a route may be predicted through a latitude value and a longitude value, and a lane change intention may be predicted through a heading value and a yaw rate value. In order to commercialize vehicle safety service using V2V/V2X communication, standardization of data transmitted and received through communication is required, so the standardization of data structure has been promoted by SAE of the United States through J2735 document. Accordingly, the full width of the vehicle may be transmitted and received by being included in the BSM message defined in the SAE J2735 standard as CAN data. In an embodiment, the structure of data transmitted and received may be designed in a BSM format.

Table 1: Details of the BSM Set as defined in the SAE J2735 standard

Referring to FIG. 5 , it is an example for describing an object bounding box. Fig. 5 (a) is an example showing the camera coordinate system and the bounding box 510, and Fig. 5 (b) is an example showing the bounding box of the surrounding vehicle having changed lanes. In this way, it can be confirmed that the coordinate information of the bounding box of the surrounding vehicle is different depending on the location of the surrounding vehicle.

The system uses the camera sensor to define the front screen as (0,0) for the left bottom and (1,1) for the right top, and the bounding box of the surrounding vehicle through the object recognition algorithm. Coordinate information can be used as a feature value of a machine learning-based lane change intention and path prediction algorithm. If the bounding box obtained through the camera sensor rather than the distance between the surrounding vehicle and the lane is used, the change in the position of the surrounding vehicle can be identified through the change of the coordinate value without being affected by the weather or road environment.

The system according to an embodiment uses V2V to calculate the relative distance, relative speed, and heading values, rather than calculating the relative distance, relative speed, and relative elevation angle with the surrounding vehicle through environmental recognition sensors such as radar, radar, and lidar. Because it is used, it is strong in various weather and driving environments, and the technical effect of being able to acquire information about surrounding vehicles located in non-line-of-sight can be derived. In addition, in order to calculate the relative distance, relative speed, and relative latitude with the surrounding vehicle through environment recognition sensors such as radar and lidar, complex calculation processes such as data association and object tracking are required. However, if the surrounding vehicles are recognized using the V2V communication according to the embodiment, the calculation amount can be reduced.

Referring to FIG. 6 , it is an example for describing a camera sensor coordinate system. FIG. 6(a) is a sensor coordinate system of an existing camera, and FIG. 6(b) is a normalized camera sensor coordinate system. The camera sensor for each pharmaceutical company may have a different coordinate system set for each frame. The system may perform a normalization operation with respect to the camera sensor coordinate system. For example, depending on the camera sensor of each manufacturer, the coordinate system may be displayed differently even if the same location is photographed using a camera. Accordingly, the system can normalize the camera sensor coordinate system for each manufacturer to (0, 0), (0, 1), (1, 0), (1, 1).

The system may recognize a nearby vehicle (eg, a vehicle in front) on the road through an object recognition algorithm on the front screen photographed using the camera sensor. The front screen can be defined as Left-Bottom as (0,0) and Right-Top as (1,1). The system may calculate bounding box coordinate information of the vehicle ahead in the recognized road. In this case, the bounding box coordinate information may be composed of box-shaped two-dimensional or three-dimensional coordinate information including the size of the recognized vehicle, and when at least one or more surrounding vehicles are recognized, a plurality of bounding box coordinates are generated. may be For example, the system may apply Google Tensorflow Object Detection API, YOLO, etc. as an object recognition algorithm to recognize a vehicle in front.

Referring to FIG. 7 , it is an example for explaining predicting the lane change intention of the surrounding vehicle. 7(a) is an example of a neighboring vehicle moving straight, and FIG. 7(b) is an example of a neighboring vehicle undergoing a lane change. The system may use the bounding box coordinate information for the surrounding vehicle as a feature value of a learning model configured for lane change intention and route prediction. The system can predict the intention to change lanes through the movement (direction) information of surrounding vehicles through the bounding box.

If the surrounding vehicle is traveling straight ahead, the bounding box is biased in a specific direction with a preset reference (eg, center) within the camera coordinate system, the surrounding vehicle moves in the adjacent lane (eg, the lane of the surrounding vehicle) As a reference, it can be determined to exist in the left lane or the right lane). The system can confirm that the bounding box, which is biased toward a specific direction, is moved to the center when the surrounding vehicle existing in the adjacent lane tries to change lanes.

Referring to FIG. 7( a ), when the surrounding vehicle is traveling straight, the bounding box is biased to the left with a preset reference (eg, center, center point, etc.) within the camera coordinate system, see FIG. 7(b) Then, when the surrounding vehicle changes lanes, it can be seen that the bounding box, which is biased toward a specific direction based on a preset standard, appears to move to the center within the camera coordinate system.

In this way, the system can learn by using the movement (direction) information of the bounding box as a feature value of a learning model configured for lane change intention and route prediction. In other words, the system may use the coordinate information of the bounding box as a feature value and input it to the learning model. In this case, the coordinate information of the bounding box may include a left bottom, a left top, a right top, and a right bottom.

Referring to FIG. 8 , it is an example for explaining predicting a path of a surrounding vehicle. The size information of the bounding box of the surrounding vehicle may be different depending on the relative distance between the vehicle and the surrounding vehicle. Fig. 8 (a) shows a neighboring vehicle having a relatively long relative distance, and Fig. 8 (b) shows a neighboring vehicle having a close relative distance. In other words, it can be confirmed that the bounding box of the neighboring vehicle having a relatively far distance is smaller than the bounding box of the neighboring vehicle having a close relative distance.

The system uses the characteristic that the bounding box becomes smaller as the relative distance between the vehicle (own vehicle) and the surrounding vehicle increases, and increases as the relative distance between the vehicle and the surrounding vehicle increases. It can be used as a feature value.

In this way, the system may learn by using the size information 810 and 820 of the bounding box as a feature value of a learning model configured for lane change intention and route prediction. In other words, the system may use the coordinate information of the bounding box as a feature value and input it to the learning model. In this case, the coordinate information of the bounding box may include a left bottom, a left top, a right top, and a right bottom.

Referring to FIG. 9 , it is a diagram for explaining learning data for predicting a lane change. The system may predict the lane change intention and route of the surrounding vehicle through the lane change intention and route prediction algorithm using the obtained dynamic information of the surrounding vehicle and the calculated bounding box coordinate information of the surrounding vehicle ( 420 ). Among the machine learning algorithms, for supervised learning, as shown in FIG. 9 , learning data must be composed of feature values and correct answer values (Label). Prediction accuracy is high when the characteristics of the feature value are clear according to the correct answer value.

The system predicts the lane change intention of surrounding vehicles using a supervised learning algorithm including SVM (Support Vector Machine) and RF (Random Forest), and includes RNN (Recurrent Neural Networks) and LSTM (Long Short-Term Memory) It is possible to predict the path of a nearby vehicle using a time series data prediction algorithm. For example, SVM (Support Vector Machine) is a machine learning technique used for linear, non-linear, regression, classification, etc. Basically, it means a method of learning how to divide two groups by measuring the distance between each data in the two groups to be classified, finding the center between the two data, and finding the optimal hyperplane from the center value. . Random Forest (RF) is one of various machine learning machine learning techniques. It is a machine learning method that appropriately learns several observed continuous variable vectors and generates an optimal model by preventing excessive learning. Random Forest collects data and sets each characteristic as a node that determines the driver's driving intention. Nodes decide the decision tree. Random forest is a machine learning method that determines one result by looking at the ratio of the results obtained through the process of bagging the decision tree.

As an example, the system trains a learning model using the acquired dynamic information of surrounding vehicles and the calculated bounding box coordinate information as a feature value of the training data, and uses the learned learning model to predict the lane change intention and route of surrounding vehicles. As the dynamic information of the surrounding vehicle and the bounding box coordinate information are input, the learning result including the intention to change the lane of the surrounding vehicle or the path prediction of the surrounding vehicle may be output. For example, the system may predict whether the lane of the surrounding vehicle will change after a preset time (eg, 3 seconds in the future), and may predict distance information between vehicles (between the own vehicle and the surrounding vehicle) . The system may predict whether a nearby vehicle will intervene in a lane in which the vehicle is present after a preset time and predict how far away the vehicle is from the vehicle.

The system according to an embodiment uses the bounding box coordinate information of the surrounding vehicle obtained from the camera sensor as a feature value in a machine learning-based prediction algorithm, so that the lane change intention and path of surrounding vehicles even in a road environment and curved road in which the lane is unclear. can be accurately predicted.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be embodied in The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (5)

A system for predicting lane change intention and route of a surrounding vehicle for predicting a lane change intention and route of a surrounding vehicle in a vehicle, the system comprising:
a communication unit for acquiring dynamic information of surrounding vehicles through V2V communication;
a sensor unit for calculating bounding box information on the recognized surrounding vehicle using a camera sensor; and
A prediction unit for predicting the lane change intention and route of the surrounding vehicle by using the obtained dynamic information of the surrounding vehicle and the calculated bounding box information as a feature value of a machine learning-based learning model
including,
The sensor unit,
Recognizing surrounding vehicles on the road through an object recognition algorithm with respect to the screen photographed using the camera sensor, and calculating bounding box coordinate information of the recognized surrounding vehicles,
The prediction unit,
The coordinate information of the bounding box is used as a feature value of the learning model by using the characteristics of the movement information transition of the bounding box that is moved as the lane of the surrounding vehicle changes, and the bounding box that changes according to the relative distance to the surrounding vehicle. Using the characteristic of the size information to use the coordinate information of the bounding box as the characteristic value of the learning model.
Lane change intention and route prediction system for surrounding vehicles. According to claim 1,
The prediction unit,
A surrounding vehicle for learning the learning model by using the obtained dynamic information of the surrounding vehicle and the calculated bounding box information as a feature value of training data, and predicting the lane change intention and route of the surrounding vehicle in the learned learning model outputting a learning result including an intention to change a lane of a surrounding vehicle or a path prediction of a surrounding vehicle according to the input of dynamic information and bounding box coordinate information of
The learning model predicts the lane change intention of a surrounding vehicle using a supervised learning algorithm including a support vector machine (SVM) and a random forest (RF), and uses a Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) ) to predict the path of surrounding vehicles using a time series data prediction algorithm that includes
Lane change intention and route prediction system for surrounding vehicles, characterized in that. delete According to claim 1,
The sensor unit,
Using the camera sensor to define the front screen as (0,0) as the lower left and the upper right as (1,1)
Lane change intention and route prediction system for surrounding vehicles, characterized in that. According to claim 1,
The communication unit,
Obtaining dynamic information of surrounding vehicles including latitude, longitude, heading, and yaw rate values of surrounding vehicles through the V2V communication
Lane change intention and route prediction system for surrounding vehicles, characterized in that.

Images