WO2022237210A1 - 障碍物信息生成 - Google Patents

障碍物信息生成 Download PDF

Info

Publication number
WO2022237210A1
WO2022237210A1 PCT/CN2022/070563 CN2022070563W WO2022237210A1 WO 2022237210 A1 WO2022237210 A1 WO 2022237210A1 CN 2022070563 W CN2022070563 W CN 2022070563W WO 2022237210 A1 WO2022237210 A1 WO 2022237210A1
Authority
WO
WIPO (PCT)
Prior art keywords
obstacle
data
fusion
point cloud
information
Prior art date
Application number
PCT/CN2022/070563
Other languages
English (en)
French (fr)
Inventor
黄超
袁梓峰
姚为龙
Original Assignee
上海仙途智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海仙途智能科技有限公司 filed Critical 上海仙途智能科技有限公司
Publication of WO2022237210A1 publication Critical patent/WO2022237210A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles

Definitions

  • the present application relates to the field of computer technology, and in particular to a method, device, device and computer-readable storage medium for generating obstacle information.
  • the perception module is responsible for receiving the original information of each sensor, constructing a real-time vehicle body environment model for the intelligent driving vehicle, and using the vehicle body environment model to provide information such as the position, shape, category, and motion state of obstacles for downstream modules. operations and decision-making.
  • the existing mainstream sensors of vehicles include lidar, millimeter-wave radar, cameras, etc.
  • Different sensors have different advantages in environment modeling, but any single sensor has certain defects.
  • the obstacle information identified by lidar data has a 3D cube frame; the obstacle information identified by millimeter wave radar data has 2D position information and velocity information full of false positives; the obstacle information identified by camera data
  • the information is only the 2D rectangular frame and category information on the plane, or the obstacle information identified based on the deep learning algorithm has 3D cube frame and category information with low precision.
  • the present application provides an obstacle information generation method, device, device, and computer-readable storage medium, capable of generating accurate and stable obstacle information in a vehicle driving environment in real time.
  • a method for generating obstacle information including: using at least two types of radar sensors to separately collect the surrounding environment information of the target vehicle, and based on the collected surrounding environment information, generate To describe the point cloud data of each obstacle in the environment; use the camera to collect the surrounding image information of the target vehicle, and based on the collected surrounding image information, generate image data used to describe each obstacle in the image; use the generated point cloud data and Perform fusion processing of obstacle data on the image data to obtain the description data of each initial fusion obstacle; use the description data and generated image data of each initial fusion obstacle to perform fusion processing of obstacle data to obtain each target fusion obstacle descriptive data.
  • An obstacle information generating device comprising: a point cloud data generating unit, configured to use at least two types of radar sensors to respectively collect surrounding environment information of a target vehicle, and generate information for describing each object in the environment based on the collected surrounding environment information.
  • the point cloud data of the obstacle is used to use the camera to collect the surrounding image information of the target vehicle, and based on the collected surrounding image information, generate image data for describing each obstacle in the image;
  • the first fusion processing unit used to use the generated point cloud data and image data to perform fusion processing of obstacle data to obtain the description data of each initial fusion obstacle;
  • the second fusion processing unit is used to use the description data of each initial fusion obstacle and generate The image data of the obstacle data is fused, and the description data of each target fusion obstacle is obtained.
  • An electronic device comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the above method for generating obstacle information by invoking the computer program.
  • a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above method for generating obstacle information is realized.
  • At least two types of radar sensors and cameras are used to perceive the body environment of the target vehicle, so as to obtain the point cloud data of each obstacle detected by each radar sensor and the camera corresponding The detected image data of each obstacle, and then, by performing secondary fusion processing on the detected obstacle data, the description data of each target fusion obstacle can be obtained. It can be seen that the application can calculate the accurate and stable obstacle position, shape, category, motion state and other information in the vehicle driving environment in real time, and the perception effect is more accurate and feature information richer than that of any single sensor.
  • FIG. 1 is a schematic flowchart of a method for generating obstacle information shown in the present application
  • FIG. 2 is a schematic diagram of the composition of modules for generating obstacle information shown in the present application
  • FIG. 3 is a schematic diagram of the composition of an obstacle information generating device shown in the present application.
  • FIG. 4 is a schematic structural diagram of an electronic device shown in the present application.
  • first, second, third, etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present application, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word “if” as used herein may be interpreted as “at” or “when” or “in response to a determination.”
  • lidar can provide three-dimensional point cloud information with the sensor as the center of the circle, with very high accuracy, good at ranging and describing the outline of obstacles, and can provide the main vehicle with the most direct reflection of the true shape of objective things.
  • the sensor has good robustness in different weathers, but the disadvantage is that it cannot know the color and texture of obstacles; millimeter-wave radar can perceive a long distance, especially sensitive to metal objects (such as vehicles), and the observation distance Farther than lidar, the disadvantage is that it detects more noise and is easily affected by the weather; the camera has high resolution, and the acquired image data can provide rich color and texture information, but it is easily affected by bad weather conditions, and the accuracy of distance measurement is poor. It is not suitable for object motion estimation (including position, velocity and other information).
  • the embodiment of the present application provides a method for generating obstacle information, specifically a method for vehicle body environment perception based on multi-sensor data, which can maximize the use of the advantages of different sensors and calculate in real time Accurate and stable obstacle position, shape, category, motion status and other information.
  • FIG. 1 it is a schematic flow chart of a method for generating obstacle information provided by an embodiment of the present application.
  • the method includes the following steps S101-S104:
  • S101 Use at least two types of radar sensors to respectively collect surrounding environment information of the target vehicle, and generate point cloud data for describing obstacles in the environment based on the collected surrounding environment information.
  • the target vehicle may be an intelligent driving vehicle, and at least two types of radar sensors may be installed on the target vehicle.
  • the embodiment of the present application does not limit the type of radar sensor, and each type of radar sensor may be It is sufficient to detect obstacles within a certain range around the target vehicle.
  • the "at least two types of radar sensors" in S101 may include: a lidar sensor and a millimeter wave radar sensor.
  • the surrounding environment information of the target vehicle can be collected by the laser radar sensor, so as to generate three-dimensional laser point cloud data according to the surrounding environment information, and the laser point cloud data can describe the surrounding environment of the vehicle obstacles in the .
  • the laser point cloud data generation equipment includes but not limited to single-line lidar, multi-line lidar, and binocular stereo cameras capable of generating point cloud data.
  • the surrounding environment information of the target vehicle can be collected by the millimeter-wave radar sensor, so as to generate three-dimensional millimeter-wave point cloud data according to the surrounding environment information, and the millimeter-wave point cloud data can be Describe the individual obstacles in the environment around the vehicle.
  • some millimeter-wave devices can also directly generate information such as obstacle location, category, and motion status with the help of algorithms provided by suppliers, and intelligent driving software can use this information when necessary.
  • module 21 and module 24 can use the laser radar sensor to detect and track the obstacles around the vehicle, and generate each laser radar obstacle Similar to the laser point cloud data, modules 23 and 26 can use millimeter wave radar sensors to detect and track obstacles around the vehicle, and generate millimeter wave point cloud data of each millimeter wave radar obstacle in the surrounding environment.
  • the detection ranges of the lidar sensor and the millimeter-wave radar sensor around the vehicle may be different, the obstacles detected by the lidar sensor and the millimeter-wave radar sensor may be all or partially the same.
  • S102 Using the camera to collect surrounding image information of the target vehicle, and based on the collected surrounding image information, generate image data for describing each obstacle in the image.
  • a camera may also be installed on the target vehicle, and the camera device includes but is not limited to a monocular camera, a binocular camera, a depth camera or a more advanced image acquisition device and the like.
  • the monocular camera can be used to collect the surrounding image information of the target vehicle, and the detection model based on deep learning is used to extract the image features of the image collected by the monocular camera to generate a binary image.
  • One-dimensional or more than two-dimensional image data which can describe various obstacles in the surrounding environment of the vehicle.
  • module 22 and module 25 can use the camera to detect and track obstacles around the vehicle, and generate image data of various visual obstacles in the surrounding environment.
  • the obstacles detected by the camera and the radar sensor may be all or partially the same.
  • S103 Use the generated point cloud data and image data to perform fusion processing of obstacle data to obtain description data of each initially fused obstacle.
  • the point cloud data of each obstacle detected by different radar sensors is obtained, and the obstacle is detected by a camera to obtain the point cloud data of each obstacle
  • the initial data fusion processing can be performed based on these data. Specifically, the point cloud data and image data of the same obstacle can be fused.
  • each obstacle after fusion processing is defined as is the initial fusion obstacle, after preliminary data fusion processing, the description data of each initial fusion obstacle can be obtained.
  • the "use the generated point cloud data and image data to perform fusion processing of obstacle data" in S103 may include the following steps A1-A3: Step A1: For each type The radar sensor, each obstacle detected by the radar sensor is defined as each point cloud obstacle, and each obstacle detected by the camera is defined as each visual obstacle.
  • the obstacles detected by the lidar sensor and the millimeter-wave radar sensor can be defined as point cloud obstacles.
  • the description form of the point cloud obstacle can be (p, v), p represents the 3D/2D point cloud set contained in the point cloud obstacle, and v represents the speed of the point cloud obstacle in the ego vehicle coordinate system.
  • the obstacles detected by the monocular camera can be defined as visual obstacles.
  • the two-dimensional image can obtain visual obstacles through the object detection algorithm based on deep learning.
  • the optional detection models include but not limited to YOLO, SSD and RetinaNet, etc.; in addition, the visual obstacles can be Is it a 2D obstacle or a 3D obstacle?
  • the description form of a 2D visual obstacle is (u, v, w, h, t, c), where (u, v) is the center point of the obstacle in the image coordinate system
  • the position above, w and h are the width and length of the 2D frame respectively
  • t is the category of obstacles (such as cars, pedestrians, trees, etc.)
  • c is a floating point number from 0 to 1, describing the confidence of the obstacle , obstacles with too little confidence can be filtered out
  • a description form of a 3D visual obstacle is (x, y, w, h, t, c), (x, y) is the center point of the obstacle in the vehicle coordinates
  • the location of the tie, w and h are the width and length of the rectangular border of the obstacle
  • t is the type of obstacle (such as cars, pedestrians, trees, etc.)
  • c is a floating point number from 0 to 1, describing the obstacle
  • visual obstacles can also contain more information, such as
  • Step A2 Use the point cloud data of each point cloud obstacle detected by the radar sensor and the image data of each visual obstacle to perform obstacle matching to obtain each first obstacle pair, wherein the first obstacle pair includes A point cloud obstacle and a visual obstacle belonging to the same obstacle.
  • each point cloud obstacle detected by this type of radar sensor can be matched with each visual obstacle detected by the camera, that is, for each type of radar sensor, the The similarity calculation is performed on the point cloud obstacles and visual obstacles detected by the type radar sensor to realize the association and matching between point cloud obstacles and visual obstacles.
  • each point cloud obstacle detected by the lidar sensor can be matched with each visual obstacle detected by the camera.
  • the point cloud data of each point cloud obstacle detected by the lidar sensor can be used Perform obstacle matching with the image data of each visual obstacle detected by the camera, so as to detect a point cloud obstacle and a visual obstacle belonging to the same obstacle, and form these two obstacles into an obstacle pair, defined here is the first obstacle pair, so that the lidar sensor can correspond to one or more first obstacle pairs, and each first obstacle pair includes a point cloud obstacle detected by a lidar sensor and a camera detected a visual impediment.
  • This matching process corresponds to the module 27 shown in FIG. 2 .
  • each obstacle detected by the millimeter-wave radar sensor can be matched with each obstacle detected by the camera.
  • the point cloud data of each point cloud obstacle detected by the millimeter-wave radar sensor and the camera can be used.
  • the image data of each detected visual obstacle is used for obstacle matching, so as to detect a point cloud obstacle and a visual obstacle belonging to the same obstacle, and these two obstacles form an obstacle pair, which is defined as the first A pair of obstacles, so that the millimeter-wave radar sensor can correspond to one or more first obstacle pairs, and each first obstacle pair includes a point cloud obstacle detected by a millimeter-wave radar sensor and a point cloud detected by a camera. a visual impediment.
  • This matching process corresponds to the module 28 shown in FIG. 2 .
  • step A2 "using the point cloud data of each point cloud obstacle detected by the radar sensor and the image data of each visual obstacle to perform obstacle matching" can be Including: using the point cloud data of each point cloud obstacle detected by the radar sensor and the image data of each visual obstacle to calculate the correlation score between each point cloud obstacle and each visual obstacle; based on the correlation score As a result, each point cloud obstacle and each visual obstacle are matched.
  • the point cloud data of each point cloud obstacle detected by the radar sensor, and each visual obstacle can be used image data, construct an obstacle correlation matrix, that is, for each point cloud obstacle and each visual obstacle, calculate the correlation score between the two, the higher the score, the two represent the same actual obstacle The more likely it is, and vice versa, where the calculation of the association score can be considered in terms of motion model, shape model, etc.
  • the bipartite graph matching algorithm can be applied to match point cloud obstacles and visual obstacles.
  • the optional matching algorithm includes but is not limited to the Hungarian matching algorithm.
  • Step A3 For each first obstacle pair, integrate the image data corresponding to the visual obstacle in the first obstacle pair into the point cloud data corresponding to the point cloud obstacle in the first obstacle pair.
  • the lidar sensor can correspond to one or more first obstacle pairs, each first obstacle pair includes a laser point cloud obstacle and a visual obstacle, for each first For the obstacle pair, the image data corresponding to the visual obstacle in the first obstacle pair is integrated into the point cloud data corresponding to the laser point cloud obstacle in the first obstacle pair, that is, for the laser point Cloud obstacles add visual information. If the visual information also includes the motion state information of the visual obstacle, it actually incorporates the motion state. In this way, each laser point cloud obstacle fused with visual information is an initial fusion obstacle thing.
  • the millimeter-wave radar sensor can correspond to one or more first obstacle pairs, and each first obstacle pair includes a laser point cloud obstacle and a visual obstacle,
  • the image data corresponding to the visual obstacle in the first obstacle pair is integrated into the point cloud data corresponding to the millimeter wave point cloud obstacle in the first obstacle pair , that is, to add visual information to millimeter-wave point cloud obstacles, if the visual information also includes the motion state information of visual obstacles, it actually incorporates the motion state, so that every millimeter-wave point cloud fused with visual information
  • the obstacle is an initial fusion obstacle.
  • the embodiment of the present application may also include: for each point cloud obstacle, if the point cloud obstacle is not matched with a visual obstacle, confirming whether there is a historical visual obstacle fused with the point cloud obstacle Information, if it exists, integrate the historical visual obstacle information into the point cloud data corresponding to the point cloud obstacle. Specifically, for a point cloud obstacle that cannot match a visual obstacle, it can be judged whether the previously fused visual information of the point cloud obstacle is expired or not. If not, continue to use the previous historical visual information for fusion, otherwise The historical visual information is eliminated to realize the maintenance of historical data.
  • S104 Using the description data of each initially fused obstacle and the generated image data, perform fusion processing of obstacle data to obtain description data of each target fused obstacle.
  • the description information includes the position, shape, motion state, category, and life of the corresponding target fusion obstacle. Periodic and other information, which are accurate and generated in real time.
  • the final fusion process corresponds to the module 29 shown in FIG. 2 .
  • "perform fusion processing of obstacle data by using description data and generated image data of each initially fused obstacle" in S104 may include the following steps B1-B3:
  • Step B1 When there are two types of radar sensors, each initial fusion obstacle corresponding to one radar sensor is defined as each first fusion obstacle, and each initial fusion obstacle corresponding to the other radar sensor is defined as each first fusion obstacle Two fusion obstacles.
  • each initial fusion obstacle corresponding to the lidar sensor is defined as each first fusion obstacle Objects
  • each initial fusion obstacle corresponding to the millimeter wave radar sensor is defined as each second fusion obstacle.
  • Step B2 Use the description data of each first fusion obstacle and the description data of each second fusion obstacle to perform obstacle matching to obtain each second obstacle pair, wherein the second obstacle pair includes The first fusion obstacle and the second fusion obstacle of the object.
  • the description data of the two can be used for obstacle matching, that is, the similarity calculation is performed between each first fusion obstacle and each second fusion obstacle to realize Association and matching between each first fusion obstacle and each second fusion obstacle.
  • each first fusion obstacle corresponding to the lidar sensor can be matched with each second fusion obstacle corresponding to the millimeter-wave radar sensor, and when performing obstacle matching, the first fusion obstacle can be used
  • the description data and the description data of each second fusion obstacle are used for obstacle matching, so as to detect a first fusion obstacle and a second fusion obstacle belonging to the same obstacle, and combine these two obstacles into one obstacle Yes, defined here as the second obstacle pair. In this manner, one or more second obstacle pairs can be obtained.
  • "perform obstacle matching by using the description data of each first fusion obstacle and the description data of each second fusion obstacle" in step B2 may include: using each first fusion obstacle A fusion of description data of obstacles and description data of each second fusion obstacle, calculating an association score between each first fusion obstacle and each second fusion obstacle; based on the association score result, for each first fusion Obstacles are matched with respective second fusion obstacles.
  • the description data of each first fusion obstacle and the description data of each second fusion obstacle can be used to construct an obstacle correlation matrix, that is, for each first fusion obstacle and each second Fusion of obstacles, calculate the correlation score between the two, the higher the score, the greater the possibility of characterizing the two as the same actual obstacle, and vice versa, wherein the calculation of the correlation score can be obtained from the motion model, shape model, Color texture, category and other considerations.
  • a bipartite graph matching algorithm can be applied to match the first fusion obstacle and the second fusion obstacle.
  • the optional matching algorithm includes but is not limited to the Hungarian matching algorithm.
  • Step B3 Perform fusion processing on the description data of each second obstacle pair and the generated image data.
  • the final target fusion obstacle can include obstacles that are only perceived by a single sensor (such as lidar sensor, millimeter wave radar sensor, camera).
  • the telephoto camera can see, but the lidar sensor and A distant vehicle that cannot be detected by the millimeter-wave radar sensor is a target fusion obstacle.
  • the embodiment of the present application can calculate accurate and stable obstacle position, shape, category, motion state and other information in the vehicle driving environment in real time, and the perception effect is more accurate and feature information richer than that of any single sensor.
  • the device includes: a point cloud data generation unit 310, which is used to collect the surrounding environment of the target vehicle by using at least two types of radar sensors information, and based on the collected surrounding environment information, generate point cloud data for describing each obstacle in the environment; the image data generation unit 320 is used to use the camera to collect the surrounding image information of the target vehicle, and based on the collected surrounding image information, Generate image data for describing each obstacle in the image; the first fusion processing unit 330 is used to use the generated point cloud data and image data to perform fusion processing of obstacle data to obtain description data of each initial fusion obstacle; The second fusion processing unit 340 is configured to use the description data of each initially fused obstacle and the generated image data to perform fusion processing on the obstacle data to obtain the description data of each target fused obstacle.
  • a point cloud data generation unit 310 which is used to collect the surrounding environment of the target vehicle by using at least two types of radar sensors information, and based on the collected surrounding environment information, generate point cloud data for describing each obstacle in the environment
  • the at least two types of radar sensors include: a lidar sensor and a millimeter wave radar sensor.
  • the first fusion processing unit 330 includes: a first definition subunit, configured to, for each type of radar sensor, define each obstacle correspondingly detected by the radar sensor as each point cloud obstacles, and define each obstacle detected by the camera as each visual obstacle; the first matching subunit is used to use the point cloud data of each point cloud obstacle detected by the radar sensor, and each visual obstacle image data of the object, and perform obstacle matching to obtain each first obstacle pair, wherein the first obstacle pair includes a point cloud obstacle and a visual obstacle belonging to the same obstacle; the first fusion subunit, For each first obstacle pair, the image data corresponding to the visual obstacle in the first obstacle pair is integrated into the point cloud data corresponding to the point cloud obstacle in the first obstacle pair.
  • the first matching subunit is specifically configured to: use the point cloud data of each point cloud obstacle detected by the radar sensor and the image data of each visual obstacle to calculate each The association score between each point cloud obstacle and each visual obstacle; based on the association score result, each point cloud obstacle is matched with each visual obstacle.
  • the first fusion processing unit 330 further includes a second fusion subunit; the second fusion subunit is used for each point cloud obstacle, if the point cloud obstacle is not matched to If there is a visual obstacle, confirm whether there is historical visual obstacle information fused with the point cloud obstacle, and if so, integrate the historical visual obstacle information into the point cloud corresponding to the point cloud obstacle data.
  • the second fusion processing unit 340 includes: a second definition subunit, configured to, when there are two types of radar sensors, each initial fusion obstacle corresponding to one of the radar sensors Objects are defined as each first fusion obstacle, and each initial fusion obstacle corresponding to another radar sensor is defined as each second fusion obstacle; the second matching subunit is used to use the description data of each first fusion obstacle and Perform obstacle matching on the description data of each second fusion obstacle to obtain each second obstacle pair, wherein the second obstacle pair includes a first fusion obstacle and a second fusion obstacle belonging to the same obstacle;
  • the third fusion subunit is configured to perform fusion processing on the description data of each second obstacle pair and the generated image data.
  • the second matching subunit is specifically configured to: use the description data of each first fusion obstacle and the description data of each second fusion obstacle to calculate the The association score between the object and each second fusion obstacle; based on the association score result, each first fusion obstacle is matched with each second fusion obstacle.
  • the device embodiment since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment.
  • the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. It can be understood and implemented by those skilled in the art without creative effort.
  • the embodiment of the present application also provides an electronic device.
  • the structure diagram of the electronic device is shown in FIG. Electrically connected; the memory 4002 is configured to store at least one computer-executable instruction, and the processor 4001 is configured to execute the at least one computer-executable instruction, thereby performing any one of the embodiments or any optional one in the present application. Steps of any method for generating obstacle information provided in the embodiments.
  • the processor 4001 can be FPGA (Field-Programmable Gate Array, Field Programmable Gate Array) or other devices with logic processing capabilities, such as MCU (Microcontroller Unit, micro control unit), CPU (Central Process Unit, central processing unit ).
  • MCU Microcontroller Unit, micro control unit
  • CPU Central Process Unit, central processing unit
  • the embodiment of the present application also provides another computer-readable storage medium, which stores a computer program, and the computer program is used to realize any of the functions provided by any embodiment or any optional implementation mode in the present application when executed by a processor. Steps of a method for generating obstacle information.
  • the computer-readable storage medium includes but is not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory, read-only memory), RAM ( Random Access Memory, Random Access Memory), EPROM (Erasable Programmable Read-Only Memory, Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory, Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or ray card. That is, a readable storage medium includes any medium that stores or transmits information in a form readable by a device (eg, a computer).
  • a device eg, a computer

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Traffic Control Systems (AREA)
  • Radar Systems Or Details Thereof (AREA)

Abstract

一种障碍物信息生成方法、装置、电子设备及计算机可读存储介质，该方法包括：利用至少两种类型的雷达传感器分别采集目标车辆的周围环境信息，生成用于描述环境中各个障碍物的点云数据（S101）；利用相机采集目标车辆的周围图像信息，生成用于描述图像中各个障碍物的图像数据（S102）；利用生成的点云数据和图像数据，进行障碍物数据的融合处理，得到各个初始融合障碍物的描述数据（S103）；利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理，得到各个目标融合障碍物的描述数据（S104）。该方法能够实时生成车辆行驶环境中准确的、稳定的障碍物信息。

Description

障碍物信息生成 技术领域
本申请涉及计算机技术领域，特别涉及用于障碍物信息生成的方法、装置、设备及计算机可读存储介质。
背景技术
在智能驾驶系统中，感知模块负责接收各传感器的原始信息，为智能驾驶车辆构建实时的车身环境模型，利用该车身环境模型提供障碍物的位置、形状、类别、运动状态等信息，以为下游模块的运作和决策提供依据。
目前，车辆现有的主流传感器有激光雷达、毫米波雷达、相机等，不同的传感器在环境建模中具备不同的优势，但任何单一传感器都存在一定的缺陷。
具体地，由激光雷达数据识别出的障碍物信息有3D立方体边框；由毫米波雷达数据识别出的障碍物信息有充斥着假阳的2D位置信息和速度信息；由相机数据识别出的障碍物信息只有平面上的2D矩形边框和类别信息，或者基于深度学习算法识别出的障碍物信息有精度较低的3D立方体边框和类别信息。
如果采用过于简单的特征来描述障碍物，会存在以下缺陷：首先，在复杂的交通场景中，仅借助边框信息不足以区分相邻的几个障碍物，造成障碍物关联错误，最终导致识别出错误的运动状态、类别；其次，把障碍物简化成3D立方体边框表示的方法中，障碍物对点云分割非常敏感，运动物体的遮挡会造成点云分割不稳定，最终导致计算出错误的运动状态；此外，激光雷达和毫米波雷达偶尔出现错误识别(即假阳)，如果没有别的传感器验证，会影响车辆正常行驶。因此，采用现有技术方案，无法实时计算出车辆行驶环境里准确的、稳定的障碍物位置、形状、类别、运动状态等信息。
发明内容
有鉴于此，本申请提供了一种障碍物信息生成方法、装置、设备及计算机可读存储介质，能够实时生成车辆行驶环境中准确的、稳定的障碍物信息。
具体地，本申请是通过如下技术方案实现的：一种障碍物信息生成方法，包括：利用至少两种类型的雷达传感器分别采集目标车辆的周围环境信息，并基于采集的周围环境信息，生成用于描述环境中各个障碍物的点云数据；利用相机采集目标车辆的周围图像信息，并基于采集的周围图像信息，生成用于描述图像中各个障碍物的图像数据；利用生成的点云数据和图像数据，进行障碍物数据的融合处理，得到各个初始融合障碍物 的描述数据；利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理，得到各个目标融合障碍物的描述数据。
一种障碍物信息生成装置，包括：点云数据生成单元，用于利用至少两种类型的雷达传感器分别采集目标车辆的周围环境信息，并基于采集的周围环境信息，生成用于描述环境中各个障碍物的点云数据；图像数据生成单元，用于利用相机采集目标车辆的周围图像信息，并基于采集的周围图像信息，生成用于描述图像中各个障碍物的图像数据；第一融合处理单元，用于利用生成的点云数据和图像数据，进行障碍物数据的融合处理，得到各个初始融合障碍物的描述数据；第二融合处理单元，用于利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理，得到各个目标融合障碍物的描述数据。
一种电子设备，包括：处理器、存储器；所述存储器，用于存储计算机程序；所述处理器，用于通过调用所述计算机程序，执行上述障碍物信息生成方法。
一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现上述障碍物信息生成方法。
在以上本申请提供的技术方案中，利用至少两种类型的雷达传感器以及相机，对目标车辆的车身环境进行感知，从而得到每一雷达传感器对应检测的各个障碍物的点云数据、以及相机对应检测到的各个障碍物的图像数据，然后，通过对检测得到的障碍物数据进行二次融合处理，可以得到各个目标融合障碍物的描述数据。可见，本申请能够实时计算出车辆行驶环境中准确的、稳定的障碍物位置、形状、类别、运动状态等信息，感知效果比任何单一传感器的感知效果更准确、特征信息更丰富。
附图说明
图1为本申请示出的一种障碍物信息生成方法的流程示意图；
图2为本申请示出的生成障碍物信息的模块组成示意图；
图3为本申请示出的一种障碍物信息生成装置的组成示意图；
图4为本申请示出的一种电子设备的结构示意图。
具体实施方式
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本申请相一致的所有实施方式。相反，它们仅是与如 所附权利要求书中所详述的、本申请的一些方面相一致的装置和方法的例子。
在本申请使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本申请。在本申请和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本文中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。
应当理解，尽管在本申请可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如，在不脱离本申请范围的情况下，第一信息也可以被称为第二信息，类似地，第二信息也可以被称为第一信息。取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。
需要说明的是，车辆现有的主流传感器有激光雷达、毫米波雷达、相机等，不同的传感器在环境建模中具备不同的优势和缺点。具体来讲，激光雷达能够提供以传感器为圆心的三维点云信息，精度非常高，擅长测距和描述障碍物的轮廓，能够为主车辆提供最直接反映客观事物的真实形态特征，除此以外，该传感器在不同天气下都有较好的鲁棒性，缺点是无法得知障碍物的颜色、纹理；毫米波雷达能够感知较远的距离，对金属物体(比如车辆)尤其敏感，观测距离比激光雷达远，缺点是检测出的噪声多，容易受天气影响；相机拥有高分辨率，获取的图像数据能够提供丰富的颜色、纹理信息，但易受恶劣天气条件影响，距离度量精度差，不适用于物体运动状体估计(包括位置、速度等信息)。
可见，不同的传感器在环境建模中具备不同的优势，由于任何单一传感器都存在一定的缺陷，因此，可以融合多传感器信息进行环境感知。为此，本申请实施例提供了一种障碍物信息生成方法，具体是一种基于多传感器数据的车身环境感知方法，该方法能够最大限度地利用不同传感器的优势，实时计算出车辆行驶环境中准确的、稳定的障碍物位置、形状、类别、运动状态等信息。
参见图1，为本申请实施例提供的一种障碍物信息生成方法的流程示意图，该方法包括以下步骤S101-S104：
S101：利用至少两种类型的雷达传感器分别采集目标车辆的周围环境信息，并基于采集的周围环境信息，生成用于描述环境中各个障碍物的点云数据。
在本申请实施例中，目标车辆可以是一种智能驾驶车辆，可以在目标车辆上安装至 少两种类型的雷达传感器，本申请实施例不对雷达传感器的类型进行限定，每一类型的雷达传感器可以对目标车辆周围一定范围内的障碍物进行检测即可。
在本申请实施例的一种实现方式中，S101中的“至少两种类型的雷达传感器”可以包括：激光雷达传感器和毫米波雷达传感器。
若目标车辆上安装有激光雷达传感器，则可以通过该激光雷达传感器采集目标车辆的周围环境信息，从而依据该周围环境信息，生成三维的激光点云数据，该激光点云数据可以描述车辆周围环境中的各个障碍物。其中，该激光点云数据的产生设备包括但不限于单线激光雷达、多线激光雷达以及能产生点云数据的双目立体摄像头等。
若目标车辆上安装有毫米波雷达传感器，则可以通过该毫米波雷达传感器采集目标车辆的周围环境信息，从而依据该周围环境信息，生成三维的毫米波点云数据，该毫米波点云数据可以描述车辆周围环境中的各个障碍物。此外，部分毫米波设备也能借助供应商提供的算法直接产生障碍物位置、类别、运动状态等信息，智能驾驶软件在必要的时候可以借助这些信息。
参见图2所示的生成障碍物信息的模块组成示意图，在目标车辆的智能驾驶系统中，模块21和模块24可以利用激光雷达传感器对车辆周围障碍物进行检测和跟踪，产生各个激光雷达障碍物的激光点云数据，类似的，模块23和模块26可以利用毫米波雷达传感器对车辆周围障碍物进行检测和跟踪，产生周围环境中各个毫米波雷达障碍物的毫米波点云数据。
可以理解的是，由于激光雷达传感器和毫米波雷达传感器对车辆周围的检测范围可能不同，因此，激光雷达传感器与毫米波雷达传感器检测到的各个障碍物对象，可能全部相同、也可能部分相同。
S102：利用相机采集目标车辆的周围图像信息，并基于采集的周围图像信息，生成用于描述图像中各个障碍物的图像数据。
在本申请实施例中，还可以在目标车辆上安装相机，该相机设备包括但不限于单目相机、双目相机、深度相机或更先进的图像采集设备等。
以目标车辆上安装有单目相机为例，则可以利用该单目相机采集目标车辆的周围图像信息，并使用基于深度学习的检测模型对该单目相机采集的图像进行图像特征提取，生成二维或二维以上的图像数据，该图像数据可以描述车辆周围环境中的各个障碍物。
如图2所示，在目标车辆的智能驾驶系统中，模块22和模块25可以利用相机对车 辆周围障碍物进行检测和跟踪，产生周围环境中各个视觉障碍物的图像数据。
可以理解的是，由于相机与上述雷达传感器对车辆周围的检测范围可能不同，因此，相机与上述雷达传感器检测到的各个障碍物对象，可能全部相同、也可能部分相同。
S103：利用生成的点云数据和图像数据，进行障碍物数据的融合处理，得到各个初始融合障碍物的描述数据。
在本申请实施例中，当利用至少两种类型的雷达传感器对障碍物进行检测，得到不同雷达传感器检测到的各个障碍物的点云数据，以及，利用相机对障碍物进行检测，得到各个障碍物的图像数据之后，便可以基于这些数据进行初始的数据融合处理，具体可以将相同障碍物的点云数据和图像数据进行融合，这里，为便于区分，将融合处理后的每一障碍物定义为初始融合障碍物，通过初步的数据融合处理后，便可以得到各个初始融合障碍物的描述数据。
在本申请实施例的一种实现方式中，S103中的“利用生成的点云数据和图像数据，进行障碍物数据的融合处理”，可以包括以下步骤A1-A3：步骤A1：对于每种类型的雷达传感器，将该雷达传感器对应检测的各个障碍物定义为各个点云障碍物，并将相机对应检测的各个障碍物定义为各个视觉障碍物。
以目标车辆上安装有“激光雷达传感器”和“毫米波雷达传感器”这两种类型的雷达传感器为例，由于激光雷达传感器和毫米波雷达传感器产生的是三维的点云数据，因此，为便于区分，可以将激光雷达传感器和毫米波雷达传感器对应检测的障碍物定义为点云障碍物。其中，点云障碍物的描述形式可以是(p，v)，p表示该点云障碍物包含的三维/二维点云集合，v表示该点云障碍物在自车坐标系下的速度。
此外，由于上述单目相机的原始输出是二维图像，因此，为便于区分，可以将该单目相机对应检测的障碍物定义为视觉障碍物。其中，该二维图像可以经基于深度学习的物体检测算法得到视觉障碍物，从运行速度和检测精度考虑，可选用的检测模型包括但不限于YOLO、SSD和RetinaNet等；另外，视觉障碍物可以是2D障碍物或是3D障碍物，一种2D视觉障碍物的描述形式是(u，v，w，h，t，c)，其中，(u，v)是障碍物中心点在图像坐标系上的位置，w和h分别是2D边框的宽和长，t是障碍物的类别(比如车、行人、树木等等)，c是一个0到1的浮点数，描述该障碍物的置信度，置信度过小的障碍物可以过滤掉；一种3D视觉障碍物的描述形式是(x，y，w，h，t，c)，(x，y)是障碍物中心点在自车坐标系上的位置，w和h分别是障碍物矩形边框的宽和长，t是障碍物的类别(比如车、 行人、树木等等)，c是一个0到1的浮点数，描述该障碍物的置信度；当然，根据不同视觉检测模型，视觉障碍物还能包含更多信息，比如颜色、纹理等。
步骤A2：利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，进行障碍物匹配，得到各个第一障碍物对，其中，第一障碍物对包括属于同一障碍物的一个点云障碍物和一个视觉障碍物。
对于每种类型的雷达传感器来讲，可以将该类型的雷达传感器检测的各个点云障碍物与相机检测的各个视觉障碍物进行匹配，即，对于每种类型的雷达传感器来讲，可以将该类型的雷达传感器检测的点云障碍物和视觉障碍物进行相似性计算，实现点云障碍物和视觉障碍物之间的关联和匹配。
下面以目标车辆上安装有“激光雷达传感器”和“毫米波雷达传感器”这两种类型的雷达传感器为例进行介绍。
具体来讲，可以将激光雷达传感器检测的各个点云障碍物与相机检测的各个视觉障碍物进行匹配，在进行障碍物匹配时，可以利用激光雷达传感器检测的各个点云障碍物的点云数据和相机检测的各个视觉障碍物的图像数据，进行障碍物匹配，以便检测出属于同一障碍物的一个点云障碍物和一个视觉障碍物，将这两个障碍物组成一个障碍物对，这里定义为第一障碍物对，这样，激光雷达传感器便可以对应一个或多个第一障碍物对，每一第一障碍物对中均包括一个激光雷达传感器检测的一个点云障碍物和相机检测的一个视觉障碍物。该匹配流程对应图2所示的模块27。
同理，可以将毫米波雷达传感器检测的各个障碍物与相机检测的各个障碍物进行匹配，在进行障碍物匹配时，可以利用毫米波雷达传感器检测的各个点云障碍物的点云数据和相机检测的各个视觉障碍物的图像数据，进行障碍物匹配，以便检测出属于同一障碍物的一个点云障碍物和一个视觉障碍物，将这两个障碍物组成一个障碍物对，这里定义为第一障碍物对，这样，毫米波雷达传感器便可以对应一个或多个第一障碍物对，每一第一障碍物对中均包括一个毫米波雷达传感器检测的一个点云障碍物和相机检测的一个视觉障碍物。该匹配流程对应图2所示的模块28。
在本申请实施例的一种实现方式中，步骤A2中的“利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，进行障碍物匹配”，可以包括：利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，计算每个点云障碍物和每个视觉障碍物之间的关联得分；基于关联得分结 果，对各个点云障碍物和各个视觉障碍物进行匹配。
在本实现方式中，对于每种类型的雷达传感器(比如激光雷达传感器和毫米波雷达传感器)来讲，可以利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，构建一个障碍物关联矩阵，即，对每个点云障碍物和每个视觉障碍物，计算二者之间的关联得分，得分越高，表征二者为同一个实际障碍物的可能性越大，反之亦然，其中，关联得分的计算可以从运动模型、形状模型等方面考虑。然后，可以根据构建的关联矩阵，应用二分图匹配算法来匹配点云障碍物与视觉障碍物，根据实际需要，可选用的匹配算法包括但不限于匈牙利匹配算法。
步骤A3：对于每一第一障碍物对，将该第一障碍物对中的视觉障碍物对应的图像数据，融入到该第一障碍物对中的点云障碍物对应的点云数据中。
以激光雷达传感器为例，该激光雷达传感器可以对应一个或多个第一障碍物对，每一第一障碍物对中均包括一个激光点云障碍物和一个视觉障碍物，对于每一第一障碍物对来讲，将该第一障碍物对中的视觉障碍物对应的图像数据，融入到该第一障碍物对中的激光点云障碍物对应的点云数据中，即，给激光点云障碍物加入视觉信息，如果视觉信息中还包括视觉障碍物的运动状态信息，实际上还融入了运动状态，这样，每一融合了视觉信息后的激光点云障碍物即为一个初始融合障碍物。
同理，以毫米波雷达传感器为例，该毫米波雷达传感器可以对应一个或多个第一障碍物对，每一第一障碍物对中均包括一个激光点云障碍物和一个视觉障碍物，对于每一第一障碍物对来讲，将该第一障碍物对中的视觉障碍物对应的图像数据，融入到该第一障碍物对中的毫米波点云障碍物对应的点云数据中，即，给毫米波点云障碍物加入视觉信息，如果视觉信息中还包括视觉障碍物的运动状态信息，实际上还融入了运动状态，这样，每一融合了视觉信息后的毫米波点云障碍物即为一个初始融合障碍物。
进一步地，本申请实施例还可以包括：对于每一点云障碍物，若该点云障碍物未匹配到一个视觉障碍物，则确认是否存在与该点云障碍物融合过的历史的视觉障碍物信息，若存在，则将历史的视觉障碍物信息，融入到该点云障碍物对应的点云数据中。具体来讲，对于匹配不上视觉障碍物的点云障碍物，可以判断该点云障碍物以前融合过的视觉信息过期与否，如果没过期，则继续使用以前的历史视觉信息进行融合，否则将该历史视觉信息进行消除，实现对历史数据的维护。
需要说明的是，本申请实施例把视觉信息融合进点云障碍物，其目的是提升障碍物 的准确性和信息量。
S104：利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理，得到各个目标融合障碍物的描述数据。
在本申请实施例中，当通过步骤S103得到每种类型的雷达传感器对应的各个初始融合障碍物(即融合了视觉信息的点云障碍物)之后，可以利用这些初始融合障碍物以及相机检测的视觉障碍物的相关数据，进行最终的数据融合处理，从而得到一个或对个目标融合障碍物对的描述信息，该描述信息中包括对应目标融合障碍物的位置、形状、运动状态、类别、生命周期等信息，这些信息是准确的、且实时生成的。其中，该最终融合流程对应图2所示的模块29。
在本申请实施例的一种实现方式中，S104中的“利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理”，可以包括以下步骤B1-B3：
步骤B1：当存在两种类型的雷达传感器时，将其中一个雷达传感器对应的各个初始融合障碍物定义为各个第一融合障碍物，将另一个雷达传感器对应的各个初始融合障碍物定义为各个第二融合障碍物。
以目标车辆上安装有“激光雷达传感器”和“毫米波雷达传感器”这两种类型的雷达传感器为例，为便于区分，将激光雷达传感器对应的各个初始融合障碍物定义为各个第一融合障碍物，将毫米波雷达传感器对应的各个初始融合障碍物定义为各个第二融合障碍物。
步骤B2：利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，进行障碍物匹配，得到各个第二障碍物对，其中，所述第二障碍物对包括属于同一障碍物的第一融合障碍物和第二融合障碍物。
对于各个第一融合障碍物以及各个第二融合障碍物，可以利用二者的描述数据进行障碍物匹配，即，将各个第一融合障碍物与和各个第二融合障碍物进行相似性计算，实现各个第一融合障碍物与各个第二融合障碍物之间的关联和匹配。
下面以目标车辆上安装有“激光雷达传感器”和“毫米波雷达传感器”这两种类型的雷达传感器为例进行介绍。
具体来讲，可以将激光雷达传感器对应的各个第一融合障碍物、与毫米波雷达传感器对应的各个第二融合障碍物进行匹配，在进行障碍物匹配时，可以利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，进行障碍物匹配，以便检测出属 于同一障碍物的一个第一融合障碍物和一个第二融合障碍物，将这两个障碍物组成一个障碍物对，这里定义为第二障碍物对。按照方式，可以得到一个或多个的第二障碍物对。
在本申请实施例的一种实现方式中，步骤B2中的“利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，进行障碍物匹配”，可以包括：利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，计算每个第一融合障碍物和每个第二融合障碍物之间的关联得分；基于关联得分结果，对各个第一融合障碍物和各个第二融合障碍物进行匹配。
在本实现方式中，可以利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，构建一个障碍物关联矩阵，即，对每个第一融合障碍物和每个第二融合障碍物，计算二者之间的关联得分，得分越高，表征二者为同一个实际障碍物的可能性越大，反之亦然，其中，关联得分的计算可以从运动模型、形状模型、颜色纹理、类别等方面考虑。然后，可以根据构建的关联矩阵，应用二分图匹配算法来匹配第一融合障碍物与第二融合障碍物物，根据实际需要，可选用的匹配算法包括但不限于匈牙利匹配算法。
步骤B3：将各个第二障碍物对的描述数据与生成的图像数据，进行融合处理。
当通过步骤B2进行障碍物匹配得到各个第二障碍物对后，可以基于该匹配结果、以及上述步骤S102得到的各个视觉障碍物的图像数据，推演出目标车辆周围环境的一个或多个目标融合障碍物。需要说明的是，最终的目标融合障碍物可以包含只由单一传感器(比如激光雷达传感器、毫米波雷达传感器、相机)感知到的障碍物，比如，长焦相机能看到、但激光雷达传感器和毫米波雷达传感器检测不到的远处车辆，该远处车辆即为目标融合障碍物。
在以上本申请实施例提供的障碍物信息生成方法中，利用至少两种类型的雷达传感器以及相机，对目标车辆的车身环境进行感知，从而得到每一雷达传感器对应检测的各个障碍物的点云数据、以及相机对应检测到的各个障碍物的图像数据，然后，通过对检测得到的障碍物数据进行二次融合处理，可以得到各个目标融合障碍物的描述数据。可见，本申请实施例能够实时计算出车辆行驶环境中准确的、稳定的障碍物位置、形状、类别、运动状态等信息，感知效果比任何单一传感器的感知效果更准确、特征信息更丰富。
参见图3，为本申请实施例提供的一种障碍物信息生成装置的组成示意图，该装置 包括：点云数据生成单元310，用于利用至少两种类型的雷达传感器分别采集目标车辆的周围环境信息，并基于采集的周围环境信息，生成用于描述环境中各个障碍物的点云数据；图像数据生成单元320，用于利用相机采集目标车辆的周围图像信息，并基于采集的周围图像信息，生成用于描述图像中各个障碍物的图像数据；第一融合处理单元330，用于利用生成的点云数据和图像数据，进行障碍物数据的融合处理，得到各个初始融合障碍物的描述数据；第二融合处理单元340，用于利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理，得到各个目标融合障碍物的描述数据。
在本申请实施例的一种实现方式中，所述至少两种类型的雷达传感器包括：激光雷达传感器和毫米波雷达传感器。
在本申请实施例的一种实现方式中，第一融合处理单元330，包括：第一定义子单元，用于对于每种类型的雷达传感器，将该雷达传感器对应检测的各个障碍物定义为各个点云障碍物，并将相机对应检测的各个障碍物定义为各个视觉障碍物；第一匹配子单元，用于利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，进行障碍物匹配，得到各个第一障碍物对，其中，所述第一障碍物对包括属于同一障碍物的一个点云障碍物和一个视觉障碍物；第一融合子单元，用于对于每一第一障碍物对，将该第一障碍物对中的视觉障碍物对应的图像数据，融入到该第一障碍物对中的点云障碍物对应的点云数据中。
在本申请实施例的一种实现方式中，第一匹配子单元，具体用于：利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，计算每个点云障碍物和每个视觉障碍物之间的关联得分；基于关联得分结果，对各个点云障碍物和各个视觉障碍物进行匹配。
在本申请实施例的一种实现方式中，第一融合处理单元330还包括第二融合子单元；第二融合子单元，用于对于每一点云障碍物，若该点云障碍物未匹配到一个视觉障碍物，则确认是否存在与该点云障碍物融合过的历史的视觉障碍物信息，若存在，则将所述历史的视觉障碍物信息，融入到该点云障碍物对应的点云数据中。
在本申请实施例的一种实现方式中，第二融合处理单元340，包括：第二定义子单元，用于当存在两种类型的雷达传感器时，将其中一个雷达传感器对应的各个初始融合障碍物定义为各个第一融合障碍物，将另一个雷达传感器对应的各个初始融合障碍物定义为各个第二融合障碍物；第二匹配子单元，用于利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，进行障碍物匹配，得到各个第二障碍物对，其中， 所述第二障碍物对包括属于同一障碍物的第一融合障碍物和第二融合障碍物；第三融合子单元，用于将各个第二障碍物对的描述数据与生成的图像数据，进行融合处理。
在本申请实施例的一种实现方式中，第二匹配子单元，具体用于：利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，计算每个第一融合障碍物和每个第二融合障碍物之间的关联得分；基于关联得分结果，对各个第一融合障碍物和各个第二融合障碍物进行匹配。
上述装置中各个单元的功能和作用的实现过程具体详见上述方法中对应步骤的实现过程，在此不再赘述。
对于装置实施例而言，由于其基本对应于方法实施例，所以相关之处参见方法实施例的部分说明即可。以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本申请方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
本申请实施例还提供了一种电子设备，该电子设备的结构示意图如图4所示，该电子设备4000包括至少一个处理器4001、存储器4002和总线4003，至少一个处理器4001均与存储器4002电连接；存储器4002被配置用于存储有至少一个计算机可执行指令，处理器4001被配置用于执行该至少一个计算机可执行指令，从而执行如本申请中任意一个实施例或任意一种可选实施方式提供的任意一种障碍物信息生成方法的步骤。
进一步，处理器4001可以是FPGA(Field－Programmable Gate Array，现场可编程门阵列)或者其它具有逻辑处理能力的器件，如MCU(Microcontroller Unit，微控制单元)、CPU(Central Process Unit，中央处理器)。
应用本申请实施例，能够实时计算出车辆行驶环境中准确的、稳定的障碍物位置、形状、类别、运动状态等信息，感知效果比任何单一传感器的感知效果更准确、特征信息更丰富。
本申请实施例还提供了另一种计算机可读存储介质，存储有计算机程序，该计算机程序用于被处理器执行时实现本申请中任意一个实施例或任意一种可选实施方式提供的任意一种障碍物信息生成方法的步骤。
本申请实施例提供的计算机可读存储介质包括但不限于任何类型的盘(包括软盘、 硬盘、光盘、CD-ROM、和磁光盘)、ROM(Read-Only Memory，只读存储器)、RAM(Random Access Memory，随即存储器)、EPROM(Erasable Programmable Read-Only Memory，可擦写可编程只读存储器)、EEPROM(Electrically Erasable Programmable Read-Only Memory，电可擦可编程只读存储器)、闪存、磁性卡片或光线卡片。也就是，可读存储介质包括由设备(例如，计算机)以能够读的形式存储或传输信息的任何介质。
应用本申请实施例，能够实时计算出车辆行驶环境中准确的、稳定的障碍物位置、形状、类别、运动状态等信息，感知效果比任何单一传感器的感知效果更准确、特征信息更丰富。
以上所述仅为本申请的较佳实施例而已，并不用以限制本申请，凡在本申请的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本申请保护的范围之内。

Claims (10)

  1. 一种障碍物信息生成方法，其特征在于，包括：
    利用至少两种类型的雷达传感器分别采集目标车辆的周围环境信息，并基于采集的周围环境信息，生成用于描述环境中各个障碍物的点云数据；
    利用相机采集目标车辆的周围图像信息，并基于采集的周围图像信息，生成用于描述图像中各个障碍物的图像数据；
    利用生成的点云数据和图像数据，进行障碍物数据的融合处理，得到各个初始融合障碍物的描述数据；
    利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理，得到各个目标融合障碍物的描述数据。
  2. 根据权利要求1所述的方法，其特征在于，所述至少两种类型的雷达传感器包括：激光雷达传感器和毫米波雷达传感器。
  3. 根据权利要求1或2所述的方法，其特征在于，所述利用生成的点云数据和图像数据，进行障碍物数据的融合处理，包括：
    对于每种类型的雷达传感器，将该雷达传感器对应检测的各个障碍物定义为各个点云障碍物，并将相机对应检测的各个障碍物定义为各个视觉障碍物；
    利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，进行障碍物匹配，得到各个第一障碍物对，其中，所述第一障碍物对包括属于同一障碍物的一个点云障碍物和一个视觉障碍物；
    对于每一第一障碍物对，将该第一障碍物对中的视觉障碍物对应的图像数据，融入到该第一障碍物对中的点云障碍物对应的点云数据中。
  4. 根据权利要求3所述的方法，其特征在于，所述利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，进行障碍物匹配，包括：
    利用该雷达传感器对应检测的各个点云障碍物的点云数据、以及各个视觉障碍物的图像数据，计算每个点云障碍物和每个视觉障碍物之间的关联得分；
    基于关联得分结果，对各个点云障碍物和各个视觉障碍物进行匹配。
  5. 根据权利要求3所述的方法，其特征在于，所述方法还包括：
    对于每一点云障碍物，若该点云障碍物未匹配到一个视觉障碍物，则确认是否存在与该点云障碍物融合过的历史的视觉障碍物信息；
    若存在，则将所述历史的视觉障碍物信息，融入到该点云障碍物对应的点云数据中。
  6. 根据权利要求1或2所述的方法，其特征在于，所述利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理，包括：
    当存在两种类型的雷达传感器时，将其中一个雷达传感器对应的各个初始融合障碍物定义为各个第一融合障碍物，将另一个雷达传感器对应的各个初始融合障碍物定义为各个第二融合障碍物；
    利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，进行障碍物匹配，得到各个第二障碍物对，其中，所述第二障碍物对包括属于同一障碍物的第一融合障碍物和第二融合障碍物；
    将各个第二障碍物对的描述数据与生成的图像数据，进行融合处理。
  7. 根据权利要求6所述的方法，其特征在于，所述利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，进行障碍物匹配，包括：
    利用各个第一融合障碍物的描述数据和各个第二融合障碍物的描述数据，计算每个第一融合障碍物和每个第二融合障碍物之间的关联得分；
    基于关联得分结果，对各个第一融合障碍物和各个第二融合障碍物进行匹配。
  8. 一种障碍物信息生成装置，其特征在于，包括：
    点云数据生成单元，用于利用至少两种类型的雷达传感器分别采集目标车辆的周围环境信息，并基于采集的周围环境信息，生成用于描述环境中各个障碍物的点云数据；
    图像数据生成单元，用于利用相机采集目标车辆的周围图像信息，并基于采集的周围图像信息，生成用于描述图像中各个障碍物的图像数据；
    第一融合处理单元，用于利用生成的点云数据和图像数据，进行障碍物数据的融合处理，得到各个初始融合障碍物的描述数据；
    第二融合处理单元，用于利用各个初始融合障碍物的描述数据和生成的图像数据，进行障碍物数据的融合处理，得到各个目标融合障碍物的描述数据。
  9. 一种电子设备，包括：
    存储器，用于存储计算机程序；
    处理器，用于通过调用所述计算机程序，执行如权利要求1-7中任一项所述的障碍物信息生成方法。
  10. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时实现权利要求1-7任一项所述的障碍物信息生成方法。
PCT/CN2022/070563 2021-05-12 2022-01-06 障碍物信息生成 WO2022237210A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110517510.6 2021-05-12
CN202110517510.6A CN115236672A (zh) 2021-05-12 2021-05-12 障碍物信息生成方法、装置、设备及计算机可读存储介质

Publications (1)

Publication Number Publication Date
WO2022237210A1 true WO2022237210A1 (zh) 2022-11-17

Family

ID=83666055

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/070563 WO2022237210A1 (zh) 2021-05-12 2022-01-06 障碍物信息生成

Country Status (2)

Country Link
CN (1) CN115236672A (zh)
WO (1) WO2022237210A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117492452B (zh) * 2024-01-03 2024-04-05 安徽中科星驰自动驾驶技术有限公司 一种用于自动驾驶3d障碍物的多模态融合方法

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013117475A (ja) * 2011-12-05 2013-06-13 Toyota Motor Corp 障害物検出装置
CN108646739A (zh) * 2018-05-14 2018-10-12 北京智行者科技有限公司 一种传感信息融合方法
CN109814112A (zh) * 2019-01-15 2019-05-28 北京百度网讯科技有限公司 一种超声波雷达与激光雷达信息融合方法和系统
CN110794406A (zh) * 2019-11-12 2020-02-14 北京经纬恒润科技有限公司 多源传感器数据融合系统和方法
CN110866544A (zh) * 2019-10-28 2020-03-06 杭州飞步科技有限公司 传感器数据的融合方法、装置及存储介质
WO2020103533A1 (zh) * 2018-11-20 2020-05-28 中车株洲电力机车有限公司 一种轨道和道路障碍物检测方法
CN111352112A (zh) * 2020-05-08 2020-06-30 泉州装备制造研究所 基于视觉、激光雷达和毫米波雷达的目标检测方法

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013117475A (ja) * 2011-12-05 2013-06-13 Toyota Motor Corp 障害物検出装置
CN108646739A (zh) * 2018-05-14 2018-10-12 北京智行者科技有限公司 一种传感信息融合方法
WO2020103533A1 (zh) * 2018-11-20 2020-05-28 中车株洲电力机车有限公司 一种轨道和道路障碍物检测方法
CN109814112A (zh) * 2019-01-15 2019-05-28 北京百度网讯科技有限公司 一种超声波雷达与激光雷达信息融合方法和系统
CN110866544A (zh) * 2019-10-28 2020-03-06 杭州飞步科技有限公司 传感器数据的融合方法、装置及存储介质
CN110794406A (zh) * 2019-11-12 2020-02-14 北京经纬恒润科技有限公司 多源传感器数据融合系统和方法
CN111352112A (zh) * 2020-05-08 2020-06-30 泉州装备制造研究所 基于视觉、激光雷达和毫米波雷达的目标检测方法

Also Published As

Publication number Publication date
CN115236672A (zh) 2022-10-25

Similar Documents

Publication Publication Date Title
US11593950B2 (en) System and method for movement detection
Zhe et al. Inter-vehicle distance estimation method based on monocular vision using 3D detection
US10863166B2 (en) Method and apparatus for generating three-dimensional (3D) road model
WO2021072696A1 (zh) 目标检测与跟踪方法、系统、可移动平台、相机及介质
WO2022016311A1 (zh) 基于点云的三维重建方法、装置和计算机设备
US11935250B2 (en) Method, device and computer-readable storage medium with instructions for processing sensor data
CN110582778A (zh) 具有对传感器数据的机器学习分类的嵌入式机动车辆感知
CN110738121A (zh) 一种前方车辆检测方法及检测系统
CN109214348A (zh) 一种障碍物检测方法、装置、设备及存储介质
JP2015148601A (ja) マッピング、位置特定、及び姿勢補正のためのシステム及び方法
CN111192295A (zh) 目标检测与跟踪方法、相关设备和计算机可读存储介质
US20220058818A1 (en) Object-centric three-dimensional auto labeling of point cloud data
Danescu et al. Particle grid tracking system stereovision based obstacle perception in driving environments
CN112446227A (zh) 物体检测方法、装置及设备
WO2024012212A1 (zh) 环境感知方法、域控制器、存储介质及车辆
CN113255779B (zh) 多源感知数据融合识别方法、系统及计算机可读存储介质
Schreier et al. From grid maps to parametric free space maps—A highly compact, generic environment representation for ADAS
CN114495045A (zh) 感知方法、感知装置、感知系统及相关设备
WO2022237210A1 (zh) 障碍物信息生成
CN116817891A (zh) 一种实时多模态感知的高精地图构建方法
CN112147635A (zh) 一种检测系统、方法及装置
Aranjuelo et al. Multimodal deep learning for advanced driving systems
CN116433715A (zh) 基于多传感器前融合结果的时序追踪方法、装置及介质
CN113988197A (zh) 基于多相机、多激光雷达的联合标定及目标融合检测方法
Chen et al. Exploring the effect of 3D object removal using deep learning for LiDAR-based mapping and long-term vehicular localization

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22806188

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22806188

Country of ref document: EP

Kind code of ref document: A1