CN111002980B - Road obstacle trajectory prediction method and system based on deep learning - Google Patents
Road obstacle trajectory prediction method and system based on deep learning Download PDFInfo
- Publication number
- CN111002980B CN111002980B CN201911260123.8A CN201911260123A CN111002980B CN 111002980 B CN111002980 B CN 111002980B CN 201911260123 A CN201911260123 A CN 201911260123A CN 111002980 B CN111002980 B CN 111002980B
- Authority
- CN
- China
- Prior art keywords
- vehicle
- information
- obstacle
- image
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units, or advanced driver assistance systems for ensuring comfort, stability and safety or drive control systems for propelling or retarding the vehicle
- B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
- B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
Abstract
The invention provides a track prediction method of a road obstacle based on deep learning, which comprises the following steps: collecting vehicle information, obstacle information around the vehicle and road information; converting vehicle information, obstacle information around the vehicle and road information into a Frenet coordinate system, and setting a record cache in the Frenet coordinate system; adding the vehicle information, the road information and the cached obstacle information around the vehicle to a database to obtain an image training data set; inputting the samples in the image training data set into a deep neural network model to generate an obstacle trajectory prediction model; the motion trajectory of an obstacle around the vehicle is predicted using the prediction model. The training sample acquired by the method is more real, and the movement track of the obstacles around the vehicle can be predicted more accurately based on the obstacle information with constantly changing road conditions; in addition, the obstacle trajectory prediction model obtained by the method is not single.
Description
Technical Field
The invention belongs to the technical field of vehicles, and particularly relates to a road obstacle trajectory prediction method and system based on deep learning.
Background
For over a century recently, the appearance of automobiles replaces the traditional transportation mode, so that the life of people is more convenient. In recent years, with the development of science and technology, especially the rapid development of intelligent computing, the research of the automatic driving automobile technology becomes a focus of all industries. The '12 leading edge technologies for determining future economy' report issued by McKensin discusses the influence degree of the 12 leading edge technologies on the future economy and society, and analyzes and estimates the respective economic and social influence of the 12 technologies in 2025, wherein the automatic driving automobile technology is ranked at the 6 th position, and the influence of the automatic driving automobile technology in 2025 is estimated as follows: economic benefits are about $ 0.2-1.9 trillion per year, and social benefits can recover 3-15 million lives per year.
Generally, systems for autonomous vehicle operation are generally divided into five modules for sensing, locating, predicting, planning, and executing. The perception is to identify and cognize the surrounding environment of the vehicle; the positioning is the judgment of the position of the vehicle; the prediction is the calculation of the motion track of the vehicle or other obstacles; planning is to measure and make a decision on the vehicle running path; the execution is to realize the control of the transverse direction and the longitudinal direction of the vehicle.
In the conventional scheme for predicting the driving track of the obstacle, the track of the obstacle is generally predicted according to the position, the speed and the posture of each obstacle, and firstly, the intention tendency of the obstacle is predicted (such as whether a vehicle can change lanes or continue to move straight in the lane); then, predicting the vehicle track by using the position of the obstacle in the high-precision map; thirdly, predicting the track of the obstacle by using different polynomial equations and fitting, and finally obtaining the running track of the obstacle; however, in this method, when the movement intention of each obstacle is predicted, the intention or position of another vehicle is not effectively taken into consideration, and even when a high-precision map is used, the vehicle is caused to travel in the center of the lane according to a fixed rule.
In addition, some technical schemes use a reinforcement learning mode, firstly, the same vehicle model is used for simulating the vehicles in all the simulation training systems; then, different reward and penalty equations are specified, and a vehicle model which can run in a plurality of individual simulation systems is obtained by utilizing the learning of the vehicle model; however, the disadvantages of this method are: the reward and penalty equations are difficult to set and time-consuming to train; in addition, the model obtained after training is not a model of multivariate behavior, so that each vehicle model in the simulation system is excessively unified.
In summary, the two obstacle trajectory prediction methods cannot better predict the trajectory of the obstacle, so that the simulation system of the automatic driving vehicle cannot simulate real road conditions.
Disclosure of Invention
The invention provides a road obstacle trajectory prediction method and system based on deep learning, and aims to solve at least one technical problem in the prior art.
In a first aspect, an embodiment of the present invention provides a road obstacle trajectory prediction method based on deep learning, where the method includes the following steps:
collecting vehicle information, obstacle information around the vehicle and road information;
converting vehicle information, obstacle information around the vehicle and road information into a Frenet coordinate system, and arranging a record cache in the Frenet coordinate system, wherein the record cache is used for caching the position information of the obstacles around the vehicle so as to enable identification serial numbers of the obstacles around the vehicle to be consistent at different moments;
adding the vehicle information, the road information and the cached obstacle information around the vehicle to a database to obtain an image training data set;
inputting the samples in the image training data set into a deep neural network model to generate an obstacle trajectory prediction model;
and predicting the motion trail of the obstacles around the vehicle by using the obstacle trail prediction model.
Further, a driving path of the vehicle is planned according to the movement track of the obstacle around the vehicle to generate planning information.
Further, after the driving path of the vehicle is planned, the traveling direction of the vehicle is controlled by using the planning information.
Further, the image training data set is obtained by the following sub-steps:
acquiring first information of a vehicle, first information of obstacles around the vehicle and first road information;
converting the first information of the vehicle, the first information of obstacles around the vehicle and the first road information into an IMU coordinate system of the vehicle to obtain second information of the vehicle, the second information of the obstacles around the vehicle and the second road information;
sampling second information of the vehicle and second information of obstacles around the vehicle respectively within a preset time range to obtain sampling information;
converting the sampling information and the second road information into a Frenet coordinate system, setting the recording cache in the Frenet coordinate system, and caching the position information of the obstacles around the vehicle by the recording cache to obtain the cached obstacle information around the vehicle;
and adding the vehicle information and the road information in the Frenet coordinate system and the cached obstacle information around the vehicle to a database to obtain the image training data set.
Further, the generating of the obstacle trajectory prediction model includes the following sub-steps:
inputting the samples in the image training data set into a deep neural network model, and extracting the image characteristics of the samples in the image training data set by using the deep neural network model;
compressing image features of samples in the image training data set to obtain an image feature array;
decoding the image feature array to obtain obstacle trajectory prediction image information, and generating an obstacle trajectory prediction model based on the obstacle trajectory prediction image information.
Further, the deep neural network model comprises a convolution layer, a residual error network layer and a deconvolution network layer;
inputting the samples in the image training dataset into a convolutional layer to obtain the image characteristics of the samples in the image training dataset;
inputting the image characteristics into the residual error network layer to be compressed to obtain an image characteristic array;
and inputting the image feature array into a deconvolution network layer to obtain the obstacle track prediction model.
Further, the convolutional layer comprises three convolutional neural network layers, and a random deactivation layer is arranged between every two adjacent convolutional neural network layers; wherein each convolutional neural network layer uses a leaky linear rectification function;
the residual error network layer comprises five residual error network layers, and each residual error network layer is sequentially arranged into a convolutional neural network layer, a random inactivation layer, a convolutional neural network layer and a random inactivation layer; wherein each convolutional neural network layer in the residual network layer uses a leaky linear rectification function;
the deconvolution network layer comprises three layers of deconvolution; wherein each deconvolution network layer uses a linear rectification function.
Further, the generating of the obstacle trajectory prediction model further includes the sub-steps of:
and performing gradient optimization on the obstacle track prediction image information to generate an optimized obstacle track prediction model.
Further, after the obstacle track prediction model is obtained, road information and vehicle information in output data of the deep neural network model are removed.
In a second aspect, an embodiment of the present invention provides a road obstacle trajectory prediction system based on deep learning, where the system includes an acquisition module, a conversion module, an addition module, a generation module, and a prediction module;
the acquisition module is used for acquiring vehicle information, obstacle information around the vehicle and road information;
the conversion module is used for executing the following operations:
converting vehicle information, obstacle information around the vehicle and road information into a Frenet coordinate system, and arranging a record cache in the Frenet coordinate system, wherein the record cache is used for caching the position information of the obstacles around the vehicle so as to enable identification serial numbers of the obstacles around the vehicle to be consistent at different moments;
the adding module is used for adding the vehicle information, the road information and the cached obstacle information around the vehicle into a database to obtain an image training data set;
the generating module inputs the samples in the image training data set into a deep neural network model to generate an obstacle track prediction model;
the prediction module predicts a motion trajectory of an obstacle around the vehicle using the obstacle trajectory prediction model.
According to the road obstacle trajectory prediction method and system based on deep learning, obstacle information and road information around a vehicle are considered, compared with the prior art that obstacle features are artificially selected, the acquired training samples are more real;
and inputting the collected training samples into the deep neural network model, so that the obstacle track prediction model can be obtained through learning based on obstacle information with constantly changing road conditions, the movement track of obstacles around the vehicle in the future N seconds is predicted, the simulation system enables the obstacles around the vehicle to run along the predicted track, and the obstacle track prediction model is obtained through gradual iterative testing, so that the obstacle track prediction model obtained through the training learning method is not single any more.
Drawings
Fig. 1 is a schematic flowchart of a road obstacle trajectory prediction method based on deep learning according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a Frenet coordinate system;
FIG. 3A is a schematic view of a scene of input data of a deep neural network model according to an embodiment of the present invention;
fig. 3B is a schematic diagram of a predicted image of output data of the deep neural network model according to the embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a deep neural network model provided in an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a residual error network layer according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a road obstacle trajectory prediction system based on deep learning according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of an autonomous vehicle system according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that functional, methodological, or structural equivalents or substitutions made by these embodiments are within the scope of the present invention.
Example one
Fig. 1 is a schematic flowchart of a method for predicting a road obstacle trajectory based on deep learning according to an embodiment of the present invention; referring to fig. 1, the method comprises the steps of:
s100, collecting vehicle information, obstacle information around the vehicle and road information; the distance between the obstacles around the vehicle and the vehicle is within a preset distance range;
s200, converting the vehicle information, the obstacle information around the vehicle and the road information into a Frenet coordinate system, and arranging a record cache in the Frenet coordinate system, wherein the record cache is used for caching the position information of the obstacles around the vehicle so as to make the identification serial numbers of the obstacles around the vehicle consistent at different moments;
s300, adding the vehicle information, the road information and the cached obstacle information around the vehicle into a database to obtain an image training data set;
s400, inputting the samples in the image training data set into a deep neural network model to generate an obstacle track prediction model;
and S500, predicting the motion trail of the obstacles around the vehicle by using the obstacle trail prediction model.
Further, a driving path of the vehicle is planned according to the movement track of the obstacle around the vehicle to generate planning information.
Further, after the driving path of the vehicle is planned, the traveling direction of the vehicle (for example, the lateral direction and the longitudinal direction of the vehicle) is controlled by using the planning information.
The following will specifically describe a specific implementation process of the deep learning-based road obstacle trajectory prediction method of the present invention.
S100, collecting vehicle information, obstacle information around the vehicle and road information; the distance between the obstacles around the vehicle and the vehicle is within a preset distance range;
the obstacles around the vehicle comprise other vehicles around the vehicle, traffic signboards and the like, and in the embodiment, the other vehicles around the vehicle are mainly used as the obstacle examples to predict the movement track of the obstacles; wherein step S100 comprises the following sub-steps S110-S130;
s110, collecting first information of a vehicle, first information of obstacles around the vehicle and first road information;
the first information of the vehicle comprises image information, position information, speed and attitude information of the vehicle;
the first information of the obstacles around the vehicle comprises image information, speed, posture and position information of the obstacles; for example, the initial position and posture of the obstacle around the vehicle may be set, and then the image information, speed, posture and position information of the obstacle may be obtained by using the obstacle detecting and tracking module.
The first road information includes lane line information.
Specifically, an image acquisition device (such as an image acquisition device like a binocular camera) acquires image information of a vehicle, a speed of the vehicle, and image information and speed of obstacles around the vehicle;
the method comprises the steps that a laser radar collects position information of a vehicle and obstacles around the vehicle;
an IMU (inertial measurement unit) measures attitude information of a vehicle and attitude information of obstacles around the vehicle;
the image acquisition apparatus also acquires first road information (for example, an image acquisition device such as a binocular camera) which is used to further position the vehicle or the position of an obstacle around the vehicle, and sets the pixel value of the first road information to L;
the GPS locator collects position information of the vehicle and position information of obstacles around the vehicle.
The vehicle position information, the position information of the obstacle around the vehicle, and the first road information are all in the ground as a coordinate system.
S120, converting the first information of the vehicle, the first information of the obstacles around the vehicle and the first road information into an IMU coordinate system of the vehicle to obtain second information of the vehicle, the second information of the obstacles around the vehicle and the second road information;
specifically, first position information, first attitude information and first speed of the vehicle, first position information, first attitude information and first road information of an obstacle around the vehicle are converted into an IMU coordinate system of the vehicle to obtain second position information, second attitude information and second speed of the vehicle, second position information, second attitude information and second speed of the obstacle around the vehicle and second road information;
s130, sampling second information of the vehicle and second information of obstacles around the vehicle respectively within a preset time range to obtain sampling information;
specifically, based on the second information of the vehicle and the second information of the obstacles around the vehicle, M pieces of vehicle information and M pieces of information of the obstacles around the vehicle are sampled within the first M seconds; sampling information is input into M video channels, and a pixel value on a video is set to be P by using attitude information and size information of a vehicle in video information of the vehicle and obstacles around the vehicle.
S200, converting vehicle information, obstacle information around the vehicle and road information into a Frenet coordinate system, wherein a record cache is arranged in the Frenet coordinate system and used for caching position information of obstacles around the vehicle so as to enable identification serial numbers of the obstacles around the vehicle to be consistent at different moments, namely if the distance between the obstacle and the vehicle is within a preset distance range, the same obstacle can continue to the identification serial number of the previous obstacle by utilizing the cache;
since the position information of the obstacles around the vehicle and the second road information acquired by the image acquisition device (e.g., a binocular camera, etc.) are not accurate enough, the mark numbers of the respective obstacles cannot be accurately identified when the obstacles are tracked, and thus different position information of the same obstacle in the map cannot be acquired.
Referring to FIG. 2, FIG. 2 is a schematic view of the Frenet coordinate system; in the context of automatic driving of vehicles, trajectory planning is road-based and performed in the Frenet coordinate system, which significantly simplifies the problem compared to the cartesian coordinate system. In the Frenet coordinate system, the center line of the link is used as a reference line, namely, named as a link center reference line, as shown in the right drawing of fig. 2. The vehicle is used as an origin, the s direction of a coordinate axis is the direction along a road center reference line and is called Longitudinal (Longitudinal), the 1 direction of the coordinate axis is the normal direction passing through the current point on the road center reference line and is called transverse (Lateral), and the s direction is perpendicular to the l direction.
The Frenet coordinate system uses the center line of the road as a reference line, and a tangent vector and a normal vector of the reference line are used for establishing the coordinate system.
Specifically, the sampling information and the second road information are converted into a Frenet coordinate system, a record cache is arranged in the Frenet coordinate system, and the record cache performs cache processing on position information of obstacles around the vehicle to obtain obstacle information around the vehicle after the cache processing, so that identification serial numbers of the obstacles at different times are consistent.
S300, adding the vehicle information, the road information and the cached obstacle information around the vehicle into a database to obtain an image training data set;
specifically, vehicle information, road information and cached obstacle information around the vehicle in the Frenet coordinate system are added to the database to obtain the image training data set.
Preferably, the reduction of all samples in the image training data set, i.e. the reduction of all samples in the image training data set to one or more images x by y, saves time for training the samples of the image training data set due to the reduced size or pixels of the images.
S400, inputting the samples in the image training data set into a deep neural network model to generate an obstacle track prediction model;
specifically, the step S400 includes the following steps S410 to S430;
s410: inputting the samples in the image training data set into a deep neural network model, namely taking the sampling information and the second road information as input data of the deep neural network model, and extracting the image characteristics of the samples in the image training data set by using the deep neural network model;
s420: compressing image features of samples in the image training data set to obtain an image feature array; wherein the image feature array has dimensions that are lower than dimensions of image features of samples in the image training dataset;
s430: decoding the image feature array to obtain obstacle trajectory prediction image information, and generating a vehicle trajectory prediction model based on the obstacle trajectory prediction image information; wherein the obstacle trajectory prediction image information is predicted position information of an obstacle around the vehicle.
Wherein, referring to fig. 4, the deep neural network model includes a convolutional layer, a residual block (residual block), and a deconvolution network layer;
specifically, inputting a sample in the image training dataset into a convolutional layer to obtain an image feature of the sample in the image training dataset;
inputting the image characteristics into the residual error network layer to be compressed to obtain an image characteristic array;
and inputting the image feature array into a deconvolution network layer to obtain the vehicle track prediction model.
Preferably, in one embodiment, referring to fig. 4 and 5, the convolutional layers have three convolutional neural network layers, namely convolutional neural network layers 1, 2 and 3, random deactivation layers (dropout) are included between adjacent convolutional layers, random deactivation layer 1 is included between convolutional neural network layers 1 and 2, and random deactivation layer 2 is included between convolutional neural network layers 2 and 3; wherein each convolutional neural network layer uses a leaky linear rectification function (leak-relu);
the residual error network layer has five layers, namely residual error network layers 1, 2, 3, 4 and 5; a random inactivation layer 3 is arranged between the residual error network layer 1 and the convolution neural network layer 3; in fig. 5, x represents input data of the residual network layer, y represents output data of the residual network layer, each of the residual network layers includes two convolution layers and two random inactivation layers, for example, the structure of each of the residual network layers can be specifically configured as a convolutional neural network layer 4-a random inactivation layer 4-a convolutional neural network layer 5-a random inactivation layer 5; wherein each of the residual network layers uses a leaky linear rectification function (leak-relu);
the deconvolution network layer also comprises three deconvolution network layers, namely deconvolution network layers 1, 2 and 3; wherein a random deactivation layer 6 is included between the deconvolution network layers 1 and 2, wherein each deconvolution network layer uses a linear rectification function (relu).
Since the input data (i.e., the samples in the image training data set) and the output data of the deconvolution network layer are relatively similar, the size of the predicted image of the vehicle trajectory can be obtained by adding the deconvolution output data of the last layer of the deconvolution network layer to the data of the samples in the image training data set, and in this way, the burden of the obstacle trajectory prediction model during training can be reduced.
Further, after generating the obstacle trajectory prediction model, step 400 further includes the following substep S440;
s440: optimizing the obstacle track prediction image information to generate an optimized obstacle track prediction model;
for example, each pixel of the image in the obstacle trajectory prediction image information is optimized by using a normalization function (e.g., softmax function), and weights of each convolution layer, residual block, and deconvolution network layer of the obstacle trajectory prediction model are normalized and calculated, thereby generating the optimized vehicle trajectory prediction model.
Preferably, the embodiment further optimizes the obstacle trajectory prediction model by using a distributed learning method, and the optimization step includes the following sub-steps S441-S443:
s441, dispersing the weight parameters of each neuron of each layer of the convolution layer, the residual error network layer and the deconvolution network layer of the obstacle track prediction model on a plurality of graphic display cards (GPU) to obtain a main obstacle track prediction model and a plurality of auxiliary obstacle track prediction models;
and S442, synchronously graduating the multiple slave obstacle trajectory prediction models to the master model for optimization, wherein each GPU is provided with one obstacle trajectory prediction model, each GPU reads different input data (namely samples in the image training data set) to calculate gradients, and the gradients calculated by each GPU are optimized on the master obstacle trajectory prediction model, and because the gradients to be optimized by the master obstacle trajectory prediction model are calculated from more input data (namely samples in the image training data set), the probability of each slave obstacle trajectory prediction model to a global optimization point can be increased by synchronously graduating the multiple slave obstacle trajectory prediction models to the master model for optimization.
And S443, synchronizing the parameters of the master model to the plurality of slave obstacle track prediction models.
Because the samples in the image training data set can be optimized by adopting the gradient optimization mode, the samples in the image training data set can be studied in a targeted manner by the distributed learning method when the obstacle trajectory prediction model is trained.
In the input data of the deep neural network model, the vehicle information and the obstacle information around the vehicle are sampled to help the model to more accurately predict the future running track of the obstacle; the input road information (i.e., lane line information) may enable the obstacle trajectory prediction model to more effectively identify the location of the obstacle in the road (i.e., lane).
Preferably, after obtaining the obstacle trajectory prediction model, the road information and the vehicle information in the output data of the deep neural network model are removed, because: the road information and the vehicle information are not required to be predicted in the present embodiment and the amount of computation by the obstacle trajectory prediction model in predicting the movement trajectory of the obstacle around the vehicle can also be reduced.
It should be noted that all samples in the image training data set are bird's-eye views and use the vehicle as a coordinate origin; fig. 3A is a scene schematic diagram of input data of a deep neural network model according to an embodiment of the present invention; fig. 3B is a schematic diagram of a predicted image of output data of the deep neural network model according to the embodiment of the present invention; wherein, the A vehicle is a vehicle, the B vehicle is an obstacle around the vehicle (for example, other vehicles around the vehicle), and the A vehicle is a vehicle1、A2Respectively representing the position of the vehicle at the sampling moments t1 and t 2; b is1、B2Respectively indicate the obstacle at t1、t2The location of the sampling instant.
And S500, predicting the motion trail of the obstacles around the vehicle by using the obstacle trail prediction model.
Further, the movement locus of the obstacle around the vehicle is converted from the Frenet coordinate system of the vehicle into the geodetic coordinate system as the obstacle position of the next frame in the sample of the image training data set.
Further, planning a running path of the vehicle according to the motion trail of the obstacles around the vehicle to generate planning information; and uses the planning information to control the direction of travel of the vehicle (e.g., lateral, longitudinal direction of the vehicle).
Example two
Fig. 6 is a schematic structural diagram of a road obstacle trajectory prediction system based on deep learning according to an embodiment of the present invention; referring to fig. 6, the system includes a track prediction system for road obstacles based on deep learning,
the system comprises an acquisition module, a conversion module, an adding module, a generation module and a prediction module;
the acquisition module is used for acquiring vehicle information, obstacle information around the vehicle and road information;
the conversion module is used for executing the following operations:
converting vehicle information, obstacle information around the vehicle and road information into a Frenet coordinate system, and arranging a record cache in the Frenet coordinate system, wherein the record cache is used for caching the position information of the obstacles around the vehicle so as to enable identification serial numbers of the obstacles around the vehicle to be consistent at different moments;
the adding module is used for adding the vehicle information, the road information and the cached obstacle information around the vehicle into a database to obtain an image training data set;
the generating module inputs the samples in the image training data set into a deep neural network model to generate an obstacle track prediction model;
the prediction module predicts a motion trajectory of an obstacle around the vehicle using the obstacle trajectory prediction model.
The specific implementation of each module is the same as that of each method step in the first embodiment, and is not described herein again.
EXAMPLE III
Fig. 7 is a schematic structural diagram of an autonomous vehicle system according to an embodiment of the present invention, referring to fig. 6, the autonomous vehicle system includes a positioning system, a sensing system, a road obstacle trajectory prediction system, a motion planning system, and a vehicle control system, where the positioning system is configured to position a position of a vehicle, and obtained positioning information is input to the obstacle trajectory prediction system;
the sensing system is used for sensing information of obstacles around the vehicle, and the obtained sensing information is input to the obstacle track prediction system; the information of the obstacles around the vehicle includes image information, speed, attitude and position information of the obstacles;
the road obstacle trajectory prediction system inputs the predicted movement trajectory of obstacles around the vehicle into a movement planning module of the vehicle;
the motion planning system plans a driving path of the vehicle according to the motion track of the obstacle to generate planning information;
the execution system receives the planning information to control the traveling direction of the vehicle;
the road obstacle trajectory prediction system is the trajectory prediction system of the road obstacle based on deep learning described in the second embodiment, and details are not repeated here.
Example four
Fig. 8 is a schematic structural diagram of an embodiment of an electronic device according to the present invention, and referring to fig. 8, in the embodiment, an electronic device is provided, including but not limited to an electronic device such as a smart phone, a fixed phone, a tablet computer, a notebook computer, a wearable device, and the like, and the electronic device includes: a processor and a memory, said memory storing computer readable instructions which, when executed by said processor, implement the method of the invention as described above.
EXAMPLE five
In the present embodiment, a computer-readable storage medium is provided, which may be a ROM (e.g., read only memory, FLASH memory, transfer device, etc.), an optical storage medium (e.g., CD-ROM, DVD-ROM, paper card, etc.), a magnetic storage medium (e.g., magnetic tape, magnetic disk drive, etc.), or other types of program storage; the computer-readable storage medium has stored thereon a computer program which, when executed by a processor or a computer, performs the method of the invention described above.
The invention has the following advantages:
in the road obstacle trajectory prediction method based on deep learning, the obstacle information and the road information around the vehicle are considered, and compared with the prior art in which the obstacle characteristics are artificially selected, the acquired training sample is more real;
the obstacle track training model can be obtained through learning based on the acquired training samples and the input deep neural network model and based on the obstacle information with constantly changing road conditions, so that the motion track of obstacles around the vehicle in the future N seconds is predicted, the simulation system enables the obstacles around the vehicle to run along the predicted track, and the obstacle track prediction model is obtained through gradual iterative testing, so that the obstacle track prediction model obtained through the training and learning method is not single any more.
Because the vehicle needs to make corresponding movement according to the movement track of the obstacles around the vehicle, the prediction method can more intelligently plan the vehicle to move more accurately.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (7)
1. A track prediction method of a road obstacle based on deep learning is characterized by comprising the following steps:
collecting vehicle information, obstacle information around the vehicle and road information;
converting vehicle information, obstacle information around the vehicle and road information into a Frenet coordinate system, and arranging a record cache in the Frenet coordinate system, wherein the record cache is used for caching the position information of the obstacles around the vehicle so as to enable identification serial numbers of the obstacles around the vehicle to be consistent at different moments;
adding the vehicle information, the road information and the cached obstacle information around the vehicle to a database to obtain an image training data set;
inputting the samples in the image training data set into a deep neural network model to generate an obstacle trajectory prediction model;
predicting the motion trail of the obstacles around the vehicle by using the obstacle trail prediction model;
the generating of the obstacle trajectory prediction model comprises the sub-steps of:
inputting the samples in the image training data set into a deep neural network model, and extracting the image characteristics of the samples in the image training data set by using the deep neural network model;
compressing image features of samples in the image training data set to obtain an image feature array;
decoding the image feature array to obtain obstacle trajectory prediction image information, and generating an obstacle trajectory prediction model based on the obstacle trajectory prediction image information;
the deep neural network model comprises a convolution layer, a residual error network layer and a deconvolution network layer;
inputting the samples in the image training dataset into a convolutional layer to obtain the image characteristics of the samples in the image training dataset;
inputting the image characteristics into the residual error network layer to be compressed to obtain an image characteristic array;
inputting the image feature array into a deconvolution network layer to obtain the obstacle track prediction model;
the convolutional layer comprises three convolutional neural network layers, and a random inactivation layer is arranged between every two adjacent convolutional neural network layers; wherein each convolutional neural network layer uses a leaky linear rectification function;
the residual error network layer comprises five residual error network layers, and each residual error network layer is sequentially arranged into a convolutional neural network layer, a random inactivation layer, a convolutional neural network layer and a random inactivation layer; wherein each convolutional neural network layer in the residual network layer uses a leaky linear rectification function;
the deconvolution network layer comprises three layers of deconvolution; wherein each deconvolution network layer uses a linear rectification function.
2. The trajectory prediction method according to claim 1, characterized in that a travel path of a vehicle is planned according to a motion trajectory of an obstacle around the vehicle to generate planning information.
3. The trajectory prediction method according to claim 2, wherein after the travel path of the vehicle is planned, the traveling direction of the vehicle is controlled using the planning information.
4. The trajectory prediction method according to claim 1, characterized in that the image training dataset is obtained by the following sub-steps:
acquiring first information of a vehicle, first information of obstacles around the vehicle and first road information;
converting the first information of the vehicle, the first information of obstacles around the vehicle and the first road information into an IMU coordinate system of the vehicle to obtain second information of the vehicle, the second information of the obstacles around the vehicle and the second road information;
sampling second information of the vehicle and second information of obstacles around the vehicle respectively within a preset time range to obtain sampling information;
converting the sampling information and the second road information into a Frenet coordinate system, setting the recording cache in the Frenet coordinate system, and caching the position information of the obstacles around the vehicle by the recording cache to obtain the cached obstacle information around the vehicle;
and adding the vehicle information and the road information in the Frenet coordinate system and the cached obstacle information around the vehicle to a database to obtain the image training data set.
5. The trajectory prediction method of claim 1, wherein the generating an obstacle trajectory prediction model further comprises the sub-steps of:
and performing gradient optimization on the obstacle track prediction image information to generate an optimized obstacle track prediction model.
6. The trajectory prediction method according to any one of claims 1 to 5, characterized in that road information and vehicle information in the output data of the deep neural network model are eliminated after the obstacle trajectory prediction model is obtained.
7. A track prediction system of a road barrier based on deep learning is characterized by comprising an acquisition module, a conversion module, an addition module, a generation module and a prediction module;
the acquisition module is used for acquiring vehicle information, obstacle information around the vehicle and road information;
the conversion module is used for executing the following operations:
converting vehicle information, obstacle information around the vehicle and road information into a Frenet coordinate system, and arranging a record cache in the Frenet coordinate system, wherein the record cache is used for caching the position information of the obstacles around the vehicle so as to enable identification serial numbers of the obstacles around the vehicle to be consistent at different moments;
the adding module is used for adding the vehicle information, the road information and the cached obstacle information around the vehicle into a database to obtain an image training data set;
the generating module inputs the samples in the image training data set into a deep neural network model to generate an obstacle track prediction model;
the prediction module predicts a motion trajectory of an obstacle around a vehicle using the obstacle trajectory prediction model;
the generating of the obstacle trajectory prediction model comprises the sub-steps of:
inputting the samples in the image training data set into a deep neural network model, and extracting the image characteristics of the samples in the image training data set by using the deep neural network model;
compressing image features of samples in the image training data set to obtain an image feature array;
decoding the image feature array to obtain obstacle trajectory prediction image information, and generating an obstacle trajectory prediction model based on the obstacle trajectory prediction image information;
the deep neural network model comprises a convolution layer, a residual error network layer and a deconvolution network layer;
inputting the samples in the image training dataset into a convolutional layer to obtain the image characteristics of the samples in the image training dataset;
inputting the image characteristics into the residual error network layer to be compressed to obtain an image characteristic array;
inputting the image feature array into a deconvolution network layer to obtain the obstacle track prediction model;
the convolutional layer comprises three convolutional neural network layers, and a random inactivation layer is arranged between every two adjacent convolutional neural network layers; wherein each convolutional neural network layer uses a leaky linear rectification function;
the residual error network layer comprises five residual error network layers, and each residual error network layer is sequentially arranged into a convolutional neural network layer, a random inactivation layer, a convolutional neural network layer and a random inactivation layer; wherein each convolutional neural network layer in the residual network layer uses a leaky linear rectification function;
the deconvolution network layer comprises three layers of deconvolution; wherein each deconvolution network layer uses a linear rectification function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911260123.8A CN111002980B (en) | 2019-12-10 | 2019-12-10 | Road obstacle trajectory prediction method and system based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911260123.8A CN111002980B (en) | 2019-12-10 | 2019-12-10 | Road obstacle trajectory prediction method and system based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111002980A CN111002980A (en) | 2020-04-14 |
| CN111002980B true CN111002980B (en) | 2021-04-30 |
Family
ID=70115134
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911260123.8A Active CN111002980B (en) | 2019-12-10 | 2019-12-10 | Road obstacle trajectory prediction method and system based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111002980B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2594079A (en) * | 2020-04-16 | 2021-10-20 | Daimler Ag | A method for tracking at least one object in the surroundings of an autonomous motor vehicle with a Frenet frame, as well as an assistance system |
| GB2595857A (en) * | 2020-06-08 | 2021-12-15 | Daimler Ag | A method for object tracking for an at least partially autonomous motor vehicle, as well as an assistance system |
| CN111845774B (en) * | 2020-07-20 | 2021-12-03 | 上海大学 | Automatic driving automobile dynamic trajectory planning and tracking method based on transverse and longitudinal coordination |
| CN112163446B (en) * | 2020-08-12 | 2023-04-28 | 浙江吉利汽车研究院有限公司 | Obstacle detection method and device, electronic equipment and storage medium |
| CN112306059B (en) * | 2020-10-15 | 2024-02-27 | 北京三快在线科技有限公司 | Training method, control method and device for control model |
| CN112417756B (en) * | 2020-11-13 | 2023-11-17 | 清华大学苏州汽车研究院（吴江） | Interactive simulation test system of automatic driving algorithm |
| CN112572472B (en) * | 2020-12-08 | 2021-12-14 | 重庆大学 | Automatic driving collision prediction method based on Frenet coordinate system |
| CN113641734B (en) * | 2021-08-12 | 2024-04-05 | 驭势科技（北京）有限公司 | Data processing method, device, equipment and medium |
| CN113753077A (en) * | 2021-08-17 | 2021-12-07 | 北京百度网讯科技有限公司 | Method and device for predicting movement locus of obstacle and automatic driving vehicle |
| CN113753078B (en) * | 2021-08-17 | 2023-09-01 | 北京百度网讯科技有限公司 | Obstacle track prediction method and device, electronic equipment and automatic driving vehicle |
| CN114237236A (en) * | 2021-12-06 | 2022-03-25 | 广州小鹏自动驾驶科技有限公司 | Vehicle forward predicted path verification method and device, vehicle and storage medium |
| CN114485712B (en) * | 2022-01-30 | 2023-07-21 | 重庆长安汽车股份有限公司 | Method for processing obstacle multiple invasion host vehicle track during ST diagram construction |
| CN115164931B (en) * | 2022-09-08 | 2022-12-09 | 南开大学 | System, method and equipment for assisting blind person in going out |
| CN115534939B (en) * | 2022-12-06 | 2023-04-14 | 广汽埃安新能源汽车股份有限公司 | Vehicle control method, device, electronic equipment and computer readable medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20190023389A (en) * | 2017-08-29 | 2019-03-08 | 인하대학교 산학협력단 | Multi-Class Multi-Object Tracking Method using Changing Point Detection |
| CN110349179A (en) * | 2019-07-12 | 2019-10-18 | 安徽大学 | Visual tracking method and device outside a kind of visible red based on more adapters |
Family Cites Families (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8605947B2 (en) * | 2008-04-24 | 2013-12-10 | GM Global Technology Operations LLC | Method for detecting a clear path of travel for a vehicle enhanced by object detection |
| US8793046B2 (en) * | 2012-06-01 | 2014-07-29 | Google Inc. | Inferring state of traffic signal and other aspects of a vehicle's environment based on surrogate data |
| US10026506B1 (en) * | 2015-02-06 | 2018-07-17 | Brain Trust Innovations I, Llc | System, RFID chip, server and method for capturing vehicle data |
| DE102016009763A1 (en) * | 2016-08-11 | 2018-02-15 | Trw Automotive Gmbh | Control system and control method for determining a trajectory and generating associated signals or control commands |
| DE102017208245A1 (en) * | 2017-05-16 | 2018-11-22 | Continental Automotive Gmbh | Method and device for the goal-based prediction of dynamic objects |
| US10482337B2 (en) * | 2017-09-29 | 2019-11-19 | Infineon Technologies Ag | Accelerating convolutional neural network computation throughput |
| CN107967486B (en) * | 2017-11-17 | 2020-08-28 | 江苏大学 | Method for recognizing behaviors of surrounding vehicles |
| US10338223B1 (en) * | 2017-12-13 | 2019-07-02 | Luminar Technologies, Inc. | Processing point clouds of vehicle sensors having variable scan line distributions using two-dimensional interpolation and distance thresholding |
| CN109927719B (en) * | 2017-12-15 | 2022-03-25 | 百度在线网络技术（北京）有限公司 | Auxiliary driving method and system based on obstacle trajectory prediction |
| DE102018203063A1 (en) * | 2018-03-01 | 2019-09-05 | Honda Motor Co., Ltd. | Collision risk-prediction unit |
| US10414395B1 (en) * | 2018-04-06 | 2019-09-17 | Zoox, Inc. | Feature-based prediction |
| US11126873B2 (en) * | 2018-05-17 | 2021-09-21 | Zoox, Inc. | Vehicle lighting state determination |
| US20190361454A1 (en) * | 2018-05-24 | 2019-11-28 | GM Global Technology Operations LLC | Control systems, control methods and controllers for an autonomous vehicle |
| US10497258B1 (en) * | 2018-09-10 | 2019-12-03 | Sony Corporation | Vehicle tracking and license plate recognition based on group of pictures (GOP) structure |
| CN109919074B (en) * | 2019-03-01 | 2022-11-04 | 中国科学院合肥物质科学研究院 | Vehicle sensing method and device based on visual sensing technology |
| CN110244734B (en) * | 2019-06-20 | 2021-02-05 | 中山大学 | Automatic driving vehicle path planning method based on deep convolutional neural network |
| CN110415266A (en) * | 2019-07-19 | 2019-11-05 | 东南大学 | A method of it is driven safely based on this vehicle surrounding vehicles trajectory predictions |
| CN110335467B (en) * | 2019-07-24 | 2021-08-27 | 山东交通学院 | Method for realizing highway vehicle behavior detection by using computer vision |
-
2019
- 2019-12-10 CN CN201911260123.8A patent/CN111002980B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20190023389A (en) * | 2017-08-29 | 2019-03-08 | 인하대학교 산학협력단 | Multi-Class Multi-Object Tracking Method using Changing Point Detection |
| CN110349179A (en) * | 2019-07-12 | 2019-10-18 | 安徽大学 | Visual tracking method and device outside a kind of visible red based on more adapters |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111002980A (en) | 2020-04-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111002980B (en) | Road obstacle trajectory prediction method and system based on deep learning | |
| KR102292277B1 (en) | LIDAR localization inferring solutions using 3D CNN networks in autonomous vehicles | |
| KR102335389B1 (en) | Deep Learning-Based Feature Extraction for LIDAR Position Estimation of Autonomous Vehicles | |
| JP6738932B2 (en) | System and method for training machine learning models located on a simulation platform | |
| KR102350181B1 (en) | LIDAR Position Estimation Using RNN and LSTM to Perform Temporal Smoothing in Autonomous Vehicles | |
| US20230144209A1 (en) | Lane line detection method and related device | |
| CN110377025A (en) | Sensor aggregation framework for automatic driving vehicle | |
| CN110239562A (en) | The real-time perception adjustment based on surrounding vehicles behavior of automatic driving vehicle is adjusted with driving | |
| CN109429518A (en) | Automatic Pilot traffic forecast based on map image | |
| CN110389580A (en) | Method for planning the drift correction in the path of automatic driving vehicle | |
| US20210206387A1 (en) | Methodologies, systems, and components for incremental and continual learning for scalable improvement of autonomous systems | |
| CN112286206B (en) | Automatic driving simulation method, system, equipment, readable storage medium and platform | |
| CN110119138A (en) | For the method for self-locating of automatic driving vehicle, system and machine readable media | |
| EP3968609B1 (en) | Control method, vehicle, and server | |
| CN114248778B (en) | Positioning method and positioning device of mobile equipment | |
| CN113592905A (en) | Monocular camera-based vehicle running track prediction method | |
| CN114127810A (en) | Vehicle autonomous level function | |
| CN110992424B (en) | Positioning method and system based on binocular vision | |
| CN110864670B (en) | Method and system for acquiring position of target obstacle | |
| CN116859931A (en) | Training method of track planning model, vehicle control mode and device | |
| JP2020050047A (en) | Object detection device | |
| US20220228866A1 (en) | System and method for providing localization using inertial sensors | |
| US11938939B1 (en) | Determining current state of traffic light(s) for use in controlling an autonomous vehicle | |
| CN117689907A (en) | Vehicle tracking method, device, computer equipment and storage medium | |
| Zhou | High Definition Map as an Infrastructure for Urban Autonomous Driving |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |