CN112990049A - AEB emergency braking method and device for automatic driving of vehicle - Google Patents
AEB emergency braking method and device for automatic driving of vehicle Download PDFInfo
- Publication number
- CN112990049A CN112990049A CN202110326396.9A CN202110326396A CN112990049A CN 112990049 A CN112990049 A CN 112990049A CN 202110326396 A CN202110326396 A CN 202110326396A CN 112990049 A CN112990049 A CN 112990049A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- sensing result
- camera
- target
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units, or advanced driver assistance systems for ensuring comfort, stability and safety or drive control systems for propelling or retarding the vehicle
- B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
- B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Abstract
The invention provides an AEB emergency braking method and device for automatic driving of a vehicle, wherein the method comprises the following steps: clustering point cloud data acquired by a laser radar through 2D spatial hash and 2D voxel grids to acquire a point cloud sensing result, and performing target identification on image data acquired by a camera through a target detection neural network to acquire an image sensing result; performing information fusion on the point cloud sensing result and the image sensing result by adopting an information fusion neural network to obtain a dense depth map; and performing data fusion on the dense depth map and the point cloud sensing result according to the transformation matrix to obtain a 3D sensing result of the target and input the 3D sensing result into a decision layer of the AEB. According to the method, faster point cloud neighbor query is realized through 2D spatial hashing in the aspect of point cloud clustering, the number of points in the spatial hashing is reduced by using a 2D voxel grid, the calculation load is reduced, and a more accurate 3D sensing result can be obtained by adopting a multi-level fusion algorithm for data fusion.
Description
Technical Field
The invention relates to the technical field of automatic driving, in particular to an AEB (automatic Braking system) Emergency Braking method and an AEB Emergency Braking device for automatic driving of a vehicle.
Background
At present, most of automatic driving AEB systems detect obstacles in front of a vehicle in real time by using a laser radar as an obstacle detection sensor, and report to a control subsystem to perform emergency braking. However, the point cloud data of the laser radar is sparse, the sensing difficulty is high, and false detection and missing detection are easily caused. And sparse point clouds have difficulty providing object class information relative to the image. Therefore, the AEB algorithm realized by the single sensor is often unreliable in sensing result, which brings a plurality of unstable factors to the AEB decision module for decision making, and affects the safety and comfort.
In the related technology, a data fusion algorithm based on millimeter wave radar and camera vision is adopted for AEB braking, however, the millimeter wave radar finds and determines the position of a target by utilizing the reflection of the target to electromagnetic waves, and the external environment full of clutter often brings false alarm problems to the sensing of the millimeter wave radar. And the millimeter wave radar can not perceive the pedestrian, and can not carry out accurate modeling to all peripheral obstacles. Therefore, the fusion of the millimeter waves and the camera is difficult to provide a stable and accurate three-dimensional sensing result.
Disclosure of Invention
The invention aims to solve the technical problems and provides an AEB emergency braking method for automatic driving of a vehicle, which adopts a method of joint perception of multiple sensors to fuse multiple perception information, can improve the stability and accuracy of perception results, has good accuracy, real-time performance and fault tolerance under more complex and multiple traffic scenes, realizes faster point cloud neighbor query by 2D spatial hashing in the aspect of point cloud clustering, reduces the number of points in the spatial hashing by using a 2D voxel grid, reduces the calculation load, enables a point cloud perception algorithm to obtain higher performance, adopts a multilevel fusion algorithm for data fusion, and can obtain a more accurate 3D perception result.
The invention also provides an AEB emergency braking device for automatic driving of a vehicle.
The technical scheme adopted by the invention is as follows:
an embodiment of the first aspect of the invention provides an AEB emergency braking method for automatic vehicle driving, which comprises the following steps: calibrating a laser radar and a camera to obtain a transformation matrix between a laser radar coordinate system and a camera coordinate system; clustering point cloud data acquired by the laser radar through 2D spatial hash and 2D voxel grids to acquire a point cloud sensing result, and performing target identification on image data acquired by the camera through a target detection neural network to acquire an image sensing result; performing information fusion on the point cloud sensing result and the image sensing result by adopting an information fusion neural network to obtain a dense depth map; performing data fusion on the dense depth map and the point cloud sensing result according to the transformation matrix to obtain a 3D sensing result of the target; and inputting the 3D perception result of the target into a decision layer of the AEB so that the decision layer judges whether to trigger emergency braking according to the 3D perception result of the target.
The AEB emergency braking method for automatic vehicle driving proposed by the present invention further has the following additional technical features:
according to an embodiment of the present invention, calibrating a laser radar and a camera to obtain a transformation matrix between a laser radar coordinate system and a camera coordinate system includes: detecting the angular points of the chessboard patterns in each picture by adopting a chessboard pattern calibration method, and estimating a projection matrix P of the camera by using a linear least square method; acquiring an internal reference matrix K of the camera according to the projection matrix P; calibrating external parameters of a camera by adopting a camera and laser radar combined calibration method based on Target plane constraint to obtain an external parameter matrix (R, t) of the camera relative to the laser radar; and acquiring a transformation matrix between the laser radar coordinate system and a camera coordinate system according to the internal reference matrix K and the external reference matrix [ R, t ].
According to one embodiment of the invention, the AEB emergency braking method for automatic driving of a vehicle described above further includes: preprocessing the point cloud data acquired by the laser radar and the image data acquired by the camera, wherein the preprocessing comprises the following steps: filtering denoising and time alignment.
According to one embodiment of the present invention, the target detection neural network includes: yolov3 framework.
According to one embodiment of the invention, information fusion is performed on the point cloud sensing result and the image sensing result by using an information fusion neural network to obtain a dense depth map, and the method comprises the following steps: and processing the point cloud sensing result and the image sensing result through a parallel encoder of NASN, and fusing the point cloud sensing result and the image sensing result into a shared decoder to output the filled dense depth map.
According to an embodiment of the invention, the data fusion of the dense depth map and the point cloud sensing result according to the transformation matrix to obtain a 3D sensing result of the target includes: projecting the dense depth map into the lidar coordinate system according to the transformation matrix; matching the dense depth map with the point cloud sensing result according to the projected result; and if the matching is successful, performing data fusion on the dense depth map and the point cloud sensing result.
An embodiment of a second aspect of the present invention provides an AEB emergency brake device for automatic vehicle driving, comprising: the calibration module is used for calibrating the laser radar and the camera so as to obtain a transformation matrix between a laser radar coordinate system and a camera coordinate system; the first perception module is used for clustering point cloud data collected by the laser radar through 2D spatial hash and 2D voxel grids to obtain a point cloud perception result; the second perception module is used for carrying out target recognition on the image data collected by the camera through a target detection neural network so as to obtain an image perception result; the first fusion module is used for carrying out information fusion on the point 5 cloud perception result and the image perception result by adopting an information fusion neural network so as to obtain a dense depth map; the second fusion module is used for carrying out data fusion on the dense depth map and the point cloud sensing result according to the transformation matrix so as to obtain a 3D sensing result of the target;
and the judging module is used for inputting the 3D perception result of the target into the decision layer of the AEB so that the decision layer judges whether to trigger emergency braking according to the 3D perception result of the target.
The invention has the beneficial effects that:
the invention adopts a method of multi-sensor combined sensing to fuse various sensing information, which can improve the stability and accuracy of sensing results, thereby having good accuracy, real-time performance and fault tolerance under more complex and multi-traffic scenes, realizing faster point cloud neighbor query by 2D spatial hashing in the aspect of point cloud clustering, reducing the number of points in the spatial hashing by using a 2D voxel grid, reducing the calculation load, and leading the point cloud sensing algorithm to obtain higher performance, and the data fusion adopts a multi-level fusion algorithm, thus obtaining a more accurate 3D sensing result.
Drawings
FIG. 1 is a flow diagram of an AEB emergency braking method for vehicle autonomous driving according to one embodiment of the present invention;
FIG. 2 is a schematic diagram of a lidar calibrated with a camera in accordance with one embodiment of the present disclosure;
FIG. 3 is a schematic diagram of clustering point cloud data collected by a lidar via 2D spatial hashing and a 2D voxel grid according to one specific example of the present disclosure;
FIG. 4 is a schematic diagram of the Yolov3 framework according to one embodiment of the invention;
FIG. 5 is a schematic illustration of image perception according to a specific example of the present invention;
FIG. 6 is a schematic diagram of the generation of a dense depth map according to one specific example of the invention;
FIG. 7 is a schematic illustration of the 3D perception results of a target according to a specific example of the present invention;
FIG. 8 is a schematic illustration of the pre-processing of point cloud data according to one specific example of the invention;
fig. 9 is a block schematic diagram of an AEB emergency braking device for vehicle autonomous driving according to one embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The present inventors have made their present invention based on the recognition and recognition of the following problems:
the sensor fusion technology utilizes a plurality of sensors with complementary characteristics to enhance the perception capability and reduce the cost, and becomes an emerging research topic, and particularly, the deep learning technology improves the performance of a camera-laser radar fusion algorithm. The camera and lidar have complementary characteristics, which make the fusion model more efficient and popular than other sensor fusion configurations.
Vision-based perception systems achieve satisfactory performance at low cost, however, single-camera perception systems do not provide reliable 3D geometry, and in addition, camera-based perception systems struggle with complex or harsh lighting conditions, which limits their all-weather operation capabilities. In contrast, lidar can provide highly accurate three-dimensional geometries and is invariant to ambient light.
The following describes an AEB emergency braking method for vehicle automatic driving, an AEB emergency braking apparatus for vehicle automatic driving, according to an embodiment of the present invention, with reference to the accompanying drawings.
FIG. 1 is a flow diagram of an AEB emergency braking method for vehicle autonomous driving according to one embodiment of the present invention. As shown in fig. 1, the method comprises the steps of:
and S1, calibrating the laser radar and the camera to obtain a transformation matrix between a laser radar coordinate system and a camera coordinate system.
Further, according to an embodiment of the present invention, calibrating the laser radar and the camera to obtain the transformation matrix between the laser radar coordinate system and the camera coordinate system may include: detecting the angular points of the chessboard patterns in each picture by adopting a chessboard pattern calibration method, and estimating a projection matrix P of the camera by using a linear least square method; acquiring an internal reference matrix K of the camera according to the projection matrix P;
calibrating external parameters of the camera by adopting a camera and laser radar combined calibration method based on Target plane constraint to obtain an external parameter matrix (R, t) of the camera relative to the laser radar; and acquiring a transformation matrix between the laser radar coordinate system and the camera coordinate system according to the internal reference matrix K and the external reference matrix [ R, t ].
Specifically, firstly, a checkerboard calibration method is adopted to detect the angular points of the checkerboard patterns in each picture, a camera projection matrix P is estimated by using a linear least square method, an internal reference matrix K of the camera is obtained, and the precision of the K is improved through nonlinear optimization to obtain the internal reference matrix of the camera. And then calibrating the external parameters of the camera by adopting a camera and laser radar combined calibration method based on Target plane constraint to obtain an external parameter matrix (R, t) of the camera relative to the laser radar. With the transformation matrix from the radar coordinate system to the image plane coordinate system, the 360-degree point cloud can be mapped to the image plane to obtain the point cloud picture. For example, as shown in the example diagram of fig. 2 in which the laser radar and the camera are calibrated, the calibrated point cloud image may be used for data fusion with an image at a later stage.
And S2, clustering the point cloud data acquired by the laser radar through 2D spatial hash and 2D voxel grids to obtain a point cloud sensing result, and performing target identification on the image data acquired by the camera through a target detection neural network to obtain an image sensing result.
Specifically, clustering is to divide a data set into different clusters according to a certain specific standard, so that data in the same cluster are as similar as possible, and data differences not in the same cluster are as large as possible. The three-dimensional point cloud is generally clustered by using characteristic attributes thereof, and each point space or local space is subjected to characteristic extraction or conversion to obtain various attributes, such as: normal vectors, density, distance, height, intensity, etc., to segment point clouds of different attributes. Common methods in point cloud processing include Euclidean clustering, density clustering and super-clustering, and aiming at different scenes, the super-clustering is slowest from the time consumption angle of the algorithm and the Euclidean clustering is fastest; from the angle of algorithm effect, the Euclidean distance is most violent and direct, and the super-body clustering has the best characteristic on objects in special forms.
Euclidean clustering takes Euclidean distance as a judgment criterion, for a point P in the space, k points closest to the point P are searched in the field through KD-Tree, wherein the point with the distance smaller than a set threshold value is put into a set Q, and if elements in the Q are not increased, the clustering process is ended; otherwise, selecting points except the point P in the set Q, and repeating the steps.
The invention adopts an improved Euclidean clustering algorithm, realizes faster point cloud query through 2D hash, divides input point cloud into blocks, obtains hash value through mapping of each block to form a hash table, each item in the hash table comprises a pointer pointing to the block, can use integer plane coordinates (x, y) to retrieve the corresponding block from the hash table, and then carries out linear search on the point cloud stored in the block. In addition, the number of points in the hash is reduced by using a 2D voxel grid, the calculation load is reduced, and the point cloud sensing algorithm obtains higher performance. The point cloud Euclidean clustering algorithm uses the distance between neighbors as a judgment standard, and the point clouds of different objects are respectively combined and aggregated to form a plurality of point sets, so that a point cloud sensing result is finally obtained. Fig. 3 is a schematic diagram of clustering point cloud data acquired by a lidar via 2D spatial hashing and a 2D voxel grid according to a specific example of the present invention.
Target detection technology is also a major research direction in the field of computer vision. It is widely applied in the fields of safety, military, traffic, medical treatment and the like.
Object detection is primarily the identification and localization of multiple objects of interest in a digital image or video. Conventional target detection can be divided into three steps: firstly, selecting a candidate region in an image, then extracting visual features such as Haar, HOG and the like, and finally classifying the image based on a support vector machine model, an RF model and other common classifiers. With the development of deep learning, the target detection algorithm is also shifted from the traditional algorithm based on manual features to the detection technology based on artificial neural network. The latter development has mainly focused on two directions: two stage algorithms such as R-CNN series and one stage algorithms such as YOLO, SSD, etc. The real-time performance of the algorithm is considered and the performance is considered. We use the one stage object detection algorithm yolov 3. Yolov3 achieves target location and identification by laying a priori box on the map and regressing the center point, length, width, and category of the object.
According to one embodiment of the present invention, an object detection neural network includes: yolov3 framework.
As shown in fig. 4, the target detection network for two-dimensional images employs Yolov3 framework; the method adopts the Darknet53 as a feature extraction network, comprises a layer 0 to a layer 74, and consists of a series of 1x1 and 3x3 convolutional layers, wherein each convolutional layer is followed by a BN layer and a Leaky-ReLU layer, and the Darknet53 also adopts residual connection; layers 75 to 105 behind Darknet53 are feature interaction layers of the yolo network, and the boundary frame prediction is respectively carried out in three dimensions, wherein in the first dimension, a feature map is subjected to 32 times of downsampling and is suitable for detecting a target with a larger size in an image; in the second scale, the feature map is subjected to 16 times of downsampling, has a medium receptive field and is suitable for detecting a medium-sized target; in the third scale, the feature map is subjected to 8 times of downsampling, the resolution of the feature map is high, and the feature map is suitable for detecting small-sized targets; and predicting under three scales to obtain a boundary frame and a label of the target of interest in the image, so as to obtain an image perception result.
In order to obtain better performance of the neural network, the invention uses a transfer learning method in network training. The pre-training weights are extracted from the general target detection model trained on the coco data set, and then the specific target is intensively trained on the data set which is built by collecting the pre-training weights by the user, so that the model has more excellent performance in a specific scene. Meanwhile, data are enhanced during training, and the enhancing method comprises horizontal image turning, scaling, HSV color gamut transformation and a Mosaic enhancing method. In the Mosaic method, four pictures are randomly cut and spliced to form training data, so that the background of the pictures is enriched, and the size of the training batch is increased indirectly.
As shown in fig. 5, the target detection neural network obtained through the aforementioned construction and training can obtain a bounding box (bounding box) and an associated label (label) of an object of interest in an image, including a person and a vehicle.
And S3, performing information fusion on the point cloud sensing result and the image sensing result by using an information fusion neural network to obtain a dense depth map.
Further, according to an embodiment of the present invention, performing information fusion on the point cloud sensing result and the image sensing result by using an information fusion neural network to obtain a dense depth map, including:
the point cloud sensing result and the image sensing result are processed by a parallel encoder of NASN (a kind of image recognition framework) and then are fused into a shared decoder to output a filled dense depth map.
In particular, the neural network for dense point cloud image generation employs an automatic encoder network that can perform depth generation from a sparse point cloud image (i.e., the point cloud sensing result described above) and an RGB (Red, Green, Blue, Red-Green-Blue) image (i.e., the image sensing result described above) without applying a validity mask. The RGB image and the sparse point cloud image are firstly processed by two NASN-based parallel encoders, then are fused into a shared decoder, and finally the filled dense depth image is output. The coding part of the network adopts a mobile version of NASN, and changes are made, so that a BN layer after a first stranded convolutional layer is removed, and a problem occurs in the calculation of an average value of a batcnorm layer due to a large number of 0 values in a sparse matrix. The decoder section uses a skip connection (skip pass) between feature maps of the same size as the encoder, using a general deconvolution layer. The neural network can improve the uneven distribution of the point cloud obtained by laser radar scanning. The generation of dense depth values is done by the high resolution image as an auxiliary condition.
Through the information fusion neural network, a dense depth map can be obtained from a sparse point cloud map. The dense and regular depth values are more beneficial to later-stage acquisition of a subsequent 3D perception result of the target. FIG. 6 is a schematic diagram of the generation of a dense depth map according to one specific example of the invention.
And S4, performing data fusion on the dense depth map and the point cloud sensing result according to the transformation matrix to obtain a 3D sensing result of the target.
Further, according to an embodiment of the present invention, the data fusion of the dense depth map and the point cloud sensing result according to the transformation matrix to obtain the 3D sensing result of the target may include: projecting the dense depth map into a laser radar coordinate system according to the transformation matrix; matching the dense depth map with the point cloud sensing result according to the projected result; and if the matching is successful, performing data fusion on the dense depth map and the point cloud sensing result.
Specifically, the step S3 describes a feature layer image and point cloud data fusion method, and the generation of the dense depth map is completed based on the sparse point cloud map and with the high-resolution image as an auxiliary condition. The obtained dense depth map can project a two-dimensional image target detection result into a 3D coordinate system, then the two-dimensional image target detection result is matched with a 3D perception result obtained by point cloud clustering, if the matching is successful, the two data are fused to obtain a more accurate 3D perception result, and the result comprises information such as target category, coordinates, size, direction angle and the like. The sensing result can be shown with reference to fig. 7.
And S5, inputting the 3D sensing result of the target into a decision layer of an AEB (automatic emergency braking) so that the decision layer can judge whether to trigger emergency braking according to the 3D sensing result of the target.
Specifically, the method and the device use a Honda model to calculate the safe distance as the discrimination standard of an AEB decision layer. The calculation formula is as follows:
dw＝tHondavrel+dHonda
wherein d (w) represents an early warning safety inter-vehicle distance, d (br) represents a braking safety inter-vehicle distance, v represents a vehicle speed, v (rel) represents a relative speed of the two vehicles, v2 represents a target vehicle speed, a1 and a2 are maximum deceleration of the vehicle and the target vehicle respectively, t1 and t2 are system delay time and braking time respectively, t (Honda) and d (Honda) are Honda parameters related to the early warning distance respectively, and t (Honda) is generally taken as 2.2, and d (Honda) is taken as 6.2. Through setting up the early warning distance, AEB can remind driver the place ahead to have the collision danger through modes such as reputation early warning in advance to can be more timely take more vigorous braking, improve braking effect.
Therefore, the method for combining and sensing the multiple sensors is adopted to fuse the multiple sensing information, the stability and the accuracy of the sensing result can be improved, so that the method has good accuracy, real-time performance and fault tolerance under more complex and multi-traffic scenes, the point cloud neighbor query is faster realized through 2D spatial hashing in the aspect of point cloud clustering, the number of points in the spatial hashing is reduced by using a 2D voxel grid, the calculation load is reduced, the point cloud sensing algorithm obtains higher performance, and the data fusion adopts a multi-level fusion algorithm, so that a more accurate 3D sensing result can be obtained.
According to an embodiment of the present invention, the AEB emergency braking method for vehicle autonomous driving described above may further include: the method comprises the following steps of preprocessing point cloud data acquired by a laser radar and image data acquired by a camera, wherein the preprocessing comprises the following steps: filtering denoising and time alignment.
Specifically, the point cloud data preprocessing comprises point cloud coordinate system conversion, point cloud ground filtering and point cloud voxel filtering. And point cloud coordinate conversion is used for mapping the point cloud under the original radar coordinate system to the unified coordinate system. And (4) separating ground points and non-ground points in the point cloud data by point cloud ground filtering, wherein the non-ground points are used for subsequent obstacle clustering. A point cloud voxel grid filtering algorithm creates a 3D voxel grid on the input point cloud data, and in each voxel, all existing points will be approximated by their centroids, thereby achieving down-sampling of the point cloud. As shown in fig. 8, the first image on the left is an original point cloud, the second is a point cloud after ground filtering, and the third is a point cloud after voxel filtering and downsampling.
Therefore, the AEB emergency braking method for automatic vehicle driving, provided by the invention, establishes a data set for a specific application scene in the aspect of image recognition, and trains a neural network by adopting a transfer learning and image enhancement algorithm, so that an image perception model obtains better performance. In the aspect of point cloud clustering, a point cloud clustering algorithm is simplified and improved, faster point cloud neighbor query is realized through 2D spatial hashing, and a 2D voxel grid is also used for reducing the number of points in the spatial hashing and reducing the calculation load, so that the point cloud sensing algorithm obtains higher performance.
And a method for joint sensing of multiple sensors is adopted to fuse multiple sensing information, so that the stability and accuracy of sensing results are improved. Through verification, the scheme has good accuracy, instantaneity and fault tolerance under more complex and multi-traffic scenes.
In the aspect of sensor fusion, the designed fusion method has great advantages compared with the traditional fusion algorithm, the traditional fusion algorithm is usually fused on one data layer in a signal layer, a characteristic layer or a result layer, and the information fusion scheme designed by the invention is a multi-level fusion algorithm. Firstly, fusing an image and point cloud in a feature layer, respectively inputting the image and the point cloud into a parallel encoder for processing, then fusing the image and the point cloud into a shared decoder, and outputting a filled dense depth map. And then, fusing the image sensing result and the point cloud sensing result in a result layer, projecting a two-dimensional image target detection result into a three-dimensional unified coordinate system by using a dense depth map, and then matching with a target obtained by point cloud clustering. And if the matching is successful, fusing the data of the two to obtain more accurate 3D perception information.
Based on the perception result obtained by the fusion, the AEB decision layer calculates the safety distance, including the early warning safety distance and the braking safety distance, and by setting the early warning distance, the AEB system can properly decelerate or remind the driver that the front part of the driver has collision danger through the modes of acousto-optic early warning and the like, so that more powerful braking can be adopted more timely, and the braking effect is improved.
In summary, according to the AEB emergency braking method for vehicle automatic driving of the embodiment of the present invention, a method of multiple sensor joint sensing is adopted to fuse multiple sensing information, so that stability and accuracy of a sensing result can be improved, and thus, the method has good accuracy, real-time performance and fault tolerance in a more complex and multiple traffic scene, faster point cloud neighbor query is realized by 2D spatial hashing in point cloud clustering, a 2D voxel grid is used to reduce the number of points in spatial hashing, and computational burden is reduced, so that a point cloud sensing algorithm obtains higher performance, and a multi-level fusion algorithm is adopted for data fusion, so that a more accurate 3D sensing result can be obtained.
Corresponding to the AEB emergency braking method for the automatic driving of the vehicle, the invention also provides an AEB emergency braking device for the automatic driving of the vehicle. Since the device embodiment of the present invention corresponds to the method embodiment described above, details that are not disclosed in the device embodiment may refer to the method embodiment described above, and are not described again in the present invention.
Fig. 9 is a block schematic diagram of an AEB emergency braking apparatus for vehicle autonomous driving according to one embodiment of the present invention, as shown in fig. 9, the apparatus including: the device comprises a calibration module 1, a first perception module 2, a second perception module 3, a first fusion module 4, a second fusion module 5 and a judgment module 6.
The calibration module 1 is used for calibrating the laser radar and the camera to acquire a transformation matrix between a laser radar coordinate system and a camera coordinate system; the first perception module 2 is used for clustering the point cloud data collected by the laser radar through 2D spatial hash and 2D voxel grids to obtain a point cloud perception result; the second perception module 3 is used for carrying out target recognition on the image data collected by the camera through a target detection neural network so as to obtain an image perception result; the first fusion module 4 is used for performing information fusion on the point cloud sensing result and the image sensing result by using an information fusion neural network to obtain a dense depth map; the second fusion module 5 is used for performing data fusion on the dense depth map and the point cloud sensing result according to the transformation matrix to obtain a 3D sensing result of the target; the judging module 6 is configured to input the 3D sensing result of the target into a decision layer of the AEB, so that the decision layer judges whether to trigger emergency braking according to the 3D sensing result of the target.
According to the AEB emergency braking device for automatic vehicle driving, disclosed by the embodiment of the invention, a method of joint sensing of multiple sensors is adopted to fuse multiple sensing information, so that the stability and accuracy of a sensing result can be improved, and the AEB emergency braking device has good accuracy, real-time performance and fault tolerance under more complex and multiple traffic scenes, faster point cloud neighbor query is realized through 2D spatial hashing in the aspect of point cloud clustering, the number of points in the spatial hashing is reduced by using a 2D voxel grid, the calculation load is reduced, the point cloud sensing algorithm obtains higher performance, and a multi-level fusion algorithm is adopted for data fusion, so that a more accurate 3D sensing result can be obtained.
In order to achieve the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements an AEB emergency braking method for vehicle autonomous driving, the method comprising: calibrating a laser radar and a camera to obtain a transformation matrix between a laser radar coordinate system and a camera coordinate system; clustering point cloud data acquired by a laser radar through 2D spatial hash and 2D voxel grids to acquire a point cloud sensing result, and performing target identification on image data acquired by a camera through a target detection neural network to acquire an image sensing result; performing information fusion on the point cloud sensing result and the image sensing result by adopting an information fusion neural network to obtain a dense depth map; performing data fusion on the dense depth map and the point cloud sensing result according to the transformation matrix to obtain a 3D sensing result of the target; and inputting the 3D sensing result of the target into a decision layer of the AEB so that the decision layer judges whether to trigger emergency braking according to the 3D sensing result of the target.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
It will be understood by those skilled in the art that all or part of the steps carried by the method for implementing the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, and when the program is executed, the program includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (7)
1. An AEB emergency braking method for automatic vehicle driving, comprising the steps of:
calibrating a laser radar and a camera to obtain a transformation matrix between a laser radar coordinate system and a camera coordinate system;
clustering point cloud data acquired by the laser radar through 2D spatial hash and 2D voxel grids to acquire a point cloud sensing result, and performing target identification on image data acquired by the camera through a target detection neural network to acquire an image sensing result;
performing information fusion on the point cloud sensing result and the image sensing result by adopting an information fusion neural network to obtain a dense depth map;
performing data fusion on the dense depth map and the point cloud sensing result according to the transformation matrix to obtain a 3D sensing result of the target;
and inputting the 3D perception result of the target into a decision layer of the AEB so that the decision layer judges whether to trigger emergency braking according to the 3D perception result of the target.
2. The AEB emergency braking method for vehicle autopilot according to claim 1 wherein calibrating the lidar to the camera to obtain a transformation matrix between the lidar coordinate system and the camera coordinate system comprises:
detecting the angular points of the chessboard patterns in each picture by adopting a chessboard pattern calibration method, and estimating a projection matrix P of the camera by using a linear least square method;
acquiring an internal reference matrix K of the camera according to the projection matrix P;
calibrating external parameters of a camera by adopting a camera and laser radar combined calibration method based on Target plane constraint to obtain an external parameter matrix (R, t) of the camera relative to the laser radar;
and acquiring a transformation matrix between the laser radar coordinate system and a camera coordinate system according to the internal reference matrix K and the external reference matrix [ R, t ].
3. The AEB emergency braking method for vehicle autonomous driving of claim 1, further comprising:
preprocessing the point cloud data acquired by the laser radar and the image data acquired by the camera, wherein the preprocessing comprises the following steps: filtering denoising and time alignment.
4. The AEB emergency braking method for vehicle autopilot of claim 1 wherein the object detection neural network comprises: yolov3 framework.
5. The AEB emergency braking method for vehicle autonomous driving according to claim 1, wherein information fusion is performed on the point cloud sensing result and the image sensing result by using an information fusion neural network to obtain a dense depth map, comprising:
and processing the point cloud sensing result and the image sensing result through a parallel encoder of NASN, and fusing the point cloud sensing result and the image sensing result into a shared decoder to output the filled dense depth map.
6. The AEB emergency braking method for vehicle autonomous driving of claim 1, wherein data fusing the dense depth map and the point cloud perception result according to the transformation matrix to obtain a 3D perception result of a target comprises:
projecting the dense depth map into the lidar coordinate system according to the transformation matrix;
matching the dense depth map with the point cloud sensing result according to the projected result;
and if the matching is successful, performing data fusion on the dense depth map and the point cloud sensing result.
7. An AEB emergency brake device for automatic vehicle driving, comprising:
the calibration module is used for calibrating the laser radar and the camera so as to obtain a transformation matrix between a laser radar coordinate system and a camera coordinate system;
the first perception module is used for clustering point cloud data collected by the laser radar through 2D spatial hash and 2D voxel grids to obtain a point cloud perception result;
the second perception module is used for carrying out target recognition on the image data collected by the camera through a target detection neural network so as to obtain an image perception result;
the first fusion module is used for carrying out information fusion on the point 5 cloud perception result and the image perception result by adopting an information fusion neural network so as to obtain a dense depth map;
the second fusion module is used for carrying out data fusion on the dense depth map and the point cloud sensing result according to the transformation matrix so as to obtain a 3D sensing result of the target;
and the judging module is used for inputting the 3D perception result of the target into the decision layer of the AEB so that the decision layer judges whether to trigger emergency braking according to the 3D perception result of the target.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110326396.9A CN112990049A (en) | 2021-03-26 | 2021-03-26 | AEB emergency braking method and device for automatic driving of vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110326396.9A CN112990049A (en) | 2021-03-26 | 2021-03-26 | AEB emergency braking method and device for automatic driving of vehicle |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112990049A true CN112990049A (en) | 2021-06-18 |
Family
ID=76334545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110326396.9A Withdrawn CN112990049A (en) | 2021-03-26 | 2021-03-26 | AEB emergency braking method and device for automatic driving of vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112990049A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114677446A (en) * | 2022-03-21 | 2022-06-28 | 华南理工大学 | Vehicle detection method, device and medium based on roadside multi-sensor fusion |
| CN116862922A (en) * | 2023-06-20 | 2023-10-10 | 运来智能装备(无锡)有限公司 | Target positioning method, system and medium based on image segmentation and radar information fusion |
-
2021
- 2021-03-26 CN CN202110326396.9A patent/CN112990049A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114677446A (en) * | 2022-03-21 | 2022-06-28 | 华南理工大学 | Vehicle detection method, device and medium based on roadside multi-sensor fusion |
| CN116862922A (en) * | 2023-06-20 | 2023-10-10 | 运来智能装备(无锡)有限公司 | Target positioning method, system and medium based on image segmentation and radar information fusion |
| CN116862922B (en) * | 2023-06-20 | 2024-03-19 | 运来智能装备(无锡)有限公司 | Target positioning method, system and medium based on image segmentation and radar information fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106650708B (en) | Automatic driving obstacle vision detection method and system | |
| KR102109941B1 (en) | Method and Apparatus for Vehicle Detection Using Lidar Sensor and Camera | |
| US11747444B2 (en) | LiDAR-based object detection and classification | |
| JP5922257B2 (en) | Vehicle periphery monitoring device | |
| CN110879994A (en) | Three-dimensional visual inspection detection method, system and device based on shape attention mechanism | |
| CN111753609A (en) | Target identification method and device and camera | |
| CN113192091A (en) | Long-distance target sensing method based on laser radar and camera fusion | |
| CN112990049A (en) | AEB emergency braking method and device for automatic driving of vehicle | |
| Li et al. | Automatic parking slot detection based on around view monitor (AVM) systems | |
| CN114118252A (en) | Vehicle detection method and detection device based on sensor multivariate information fusion | |
| CN113408324A (en) | Target detection method, device and system and advanced driving assistance system | |
| Doval et al. | Traffic sign detection and 3D localization via deep convolutional neural networks and stereo vision | |
| Zimmer et al. | Infradet3d: Multi-modal 3d object detection based on roadside infrastructure camera and lidar sensors | |
| CN117111055A (en) | Vehicle state sensing method based on thunder fusion | |
| Perrollaz et al. | Using obstacles and road pixels in the disparity-space computation of stereo-vision based occupancy grids | |
| Petrovai et al. | Obstacle detection using stereovision for Android-based mobile devices | |
| CN112529011A (en) | Target detection method and related device | |
| CN112733678A (en) | Ranging method, ranging device, computer equipment and storage medium | |
| Giosan et al. | Superpixel-based obstacle segmentation from dense stereo urban traffic scenarios using intensity, depth and optical flow information | |
| CN116968758A (en) | Vehicle control method and device based on three-dimensional scene representation | |
| CN114648639B (en) | Target vehicle detection method, system and device | |
| Dai et al. | A driving assistance system with vision based vehicle detection techniques | |
| US20240151855A1 (en) | Lidar-based object tracking | |
| CN112651405B (en) | Target detection method and device | |
| CN114648549A (en) | Traffic scene target detection and positioning method fusing vision and laser radar |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210618 |
|
| WW01 | Invention patent application withdrawn after publication |