CN113655494A - Target detection method, device and medium with road side camera and 4D millimeter wave integrated - Google Patents
Target detection method, device and medium with road side camera and 4D millimeter wave integrated Download PDFInfo
- Publication number
- CN113655494A CN113655494A CN202110853786.1A CN202110853786A CN113655494A CN 113655494 A CN113655494 A CN 113655494A CN 202110853786 A CN202110853786 A CN 202110853786A CN 113655494 A CN113655494 A CN 113655494A
- Authority
- CN
- China
- Prior art keywords
- image
- camera
- target
- millimeter wave
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 22
- 238000000034 method Methods 0.000 claims abstract description 14
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 6
- 238000013507 mapping Methods 0.000 claims abstract description 4
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 230000008901 benefit Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Abstract
The invention relates to a target detection method, equipment and a medium for fusing a road side camera and 4D millimeter waves, wherein the target detection method comprises the following steps: acquiring a first image of a road side camera and a first point cloud of a 4D millimeter wave radar; projecting the first point cloud onto the first image, adding distance information on the first image, and acquiring a second image; taking the second image as the input of a depth completion convolution network to obtain a third image, wherein the third image is a dense depth image; converting the third image into a second point cloud, and encoding to generate a fourth image; inputting the fourth image into a convolutional neural network for target detection to obtain a target frame; and mapping the target frame into a 3D frame under a camera coordinate system, and adding speed information on the 3D frame based on the first point cloud. Compared with the prior art, the method has the advantages of long sensing distance, accurate target positioning and the like.
Description
Technical Field
The invention relates to the technical field of intelligent vehicle networking, in particular to a method, equipment and medium for detecting a target by fusing a road side camera and 4D millimeter waves.
Background
Currently, automatic driving and roadside sensing are dominated by lidar-based 3D detection algorithms. According to the characteristics of the laser radar, the common target detection methods can be divided into four types, namely BEV-based, camera view-based, point-wise feature-based and fusion feature-based. The laser radar has the characteristic of no influence of illumination, and in addition, the technology can directly obtain accurate three-dimensional information.
However, the laser radar has the following disadvantages:
(1) the laser radar has high cost, and the large-scale application of the market is hindered to a certain extent.
(2) Lidar is susceptible to weather. Laser light generally attenuates less in a clear weather and has a longer propagation distance. In bad weather such as heavy rain, heavy smoke, heavy fog and the like, attenuation is increased rapidly, propagation distance is affected greatly, and laser beams are distorted and shaken by atmospheric circulation, so that the measurement accuracy of the laser radar is directly affected. Therefore, the original data of the laser radar can be interfered in rainy days, foggy days, snowy days, large dust emission and other conditions, and meanwhile, the detection algorithm is influenced, so that the target object cannot be accurately positioned.
Therefore, it is necessary to develop a new detection method.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provide a method, equipment and medium for detecting a target by fusing a roadside camera and 4D millimeter waves, which have the advantages of long sensing distance and accurate target positioning.
The purpose of the invention can be realized by the following technical scheme:
in a first aspect, the invention provides a target detection method with a road side camera fused with 4D millimeter waves, which includes the following steps:
acquiring a first image of a road side camera and a first point cloud of a 4D millimeter wave radar;
projecting the first point cloud onto the first image, adding distance information on the first image, and acquiring a second image;
taking the second image as the input of a depth completion convolution network to obtain a third image, wherein the third image is a dense depth image;
converting the third image into a second point cloud, and encoding to generate a fourth image;
inputting the fourth image into a convolutional neural network for target detection to obtain a target frame;
and mapping the target frame into a 3D frame under a camera coordinate system, and adding speed information on the 3D frame based on the first point cloud.
Furthermore, the road side camera and the 4D millimeter wave radar are calibrated in advance, and camera radar calibration information is stored.
Further, projecting the first point cloud onto the first image based on the camera radar calibration information.
Further, camera calibration parameters are obtained based on the camera radar calibration information, and the third image is converted into a second point cloud through the camera calibration parameters.
Further, the second point cloud generates a fourth image by BEV encoding.
Further, the speed information is obtained based on a plurality of acquisition speed fits of the corresponding target acquired by the 4D millimeter wave radar.
Further, the velocity information is obtained by neural network fitting based on the plurality of acquisition velocities.
Further, the depth-complementing convolutional network is a multi-view-fused depth-complementing convolutional network.
In a second aspect, the present invention provides an electronic device comprising:
one or more processors;
a memory; and
one or more programs stored in memory, the one or more programs including instructions for performing a method of object detection with a roadside camera fused with 4D millimeter waves as described above.
In a third aspect, the present invention provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for performing the roadside camera-4D millimeter wave fused object detection method as described above.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention adopts the 4D millimeter wave radar, has the wave band characteristic, and can provide effective roadside 3D target object perception in severe weather such as rain, fog, snow, hail and the like;
2. because the millimeter wave radar can directly acquire the speed of the target, the target speed can be directly fitted through the neural network, and the target speed is conveniently acquired;
3. the computing force requirement can be reduced, and the process of obtaining the removal speed through target tracking is omitted;
4. because the millimeter waves can directly acquire the speed, the target loss rate can be reduced aiming at the target tracking part of post-processing;
5. the obtained dimension information is more: the laser radar only has coordinate values, and the millimeter waves have speed in addition to the coordinate values;
6. the sensing distance is far: the effective distance of the 4D millimeter wave data is 300 meters, 250 meters of effective sensing can be provided by matching with a camera, and the laser is only 150 meters;
7. the target positioning is accurate: the original data quantity of the 4D millimeter waves is far larger than that of the traditional millimeter waves (about 11 times), the ranging precision is slightly inferior to that of a laser radar, and the engineering requirements are completely met; due to the adoption of the depth completion technology, the camera can be equivalently used as a laser radar (720p image is equivalently used as 720 line laser), and the denser the camera, the more obvious the positioning advantage is;
8. hardware cost reduction: the millimeter wave radar has low cost and is much lower than laser.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a schematic diagram of a deep-fill convolution network structure adopted in this embodiment;
fig. 3 is a convolutional neural network employed in the present embodiment.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation manner and a specific operation process are given, but the scope of the present invention is not limited to the following embodiments.
As shown in fig. 1, the present embodiment provides a target detection method with a roadside camera fused with 4D millimeter waves, including the following steps:
and S1, calibrating the road side camera and the 4D millimeter wave radar in advance, and storing camera radar calibration information.
S2, acquiring image data collected by the road side camera, namely a first image, and point cloud data collected by the 4D millimeter wave radar, namely a first point cloud.
S3, projecting the first point cloud onto the first image based on the camera radar calibration information, adding distance information to the first image, and acquiring a second image, specifically:
and (3) space-time synchronization is carried out on the camera and the 4D millimeter wave, pixel level fusion is carried out, and the depth map and the candidate frame are marked by utilizing a data analysis algorithm.
The added distance information specifically includes: an rbgd matrix is initialized and distances of projected points are added in the d-channel.
And S4, taking the second image as the input of the depth completion convolution network to obtain a third image, wherein the third image is a dense depth map.
The depth-complementing convolutional network may be a multi-view-fused depth-complementing convolutional network, and the structure of the depth-complementing convolutional network adopted in this embodiment is shown in fig. 2.
S5, obtaining camera calibration parameters based on the camera radar calibration information, converting the third image into second point cloud through the camera calibration parameters, generating a fourth image, namely a depth BEV image, through BEV coding, and inputting the fourth image into a convolutional neural network for target detection to obtain a target frame.
The convolutional neural network adopted in this embodiment is shown in fig. 3, and the input fourth image sequentially passes through the encoder and the decoder to obtain a heat map and a corresponding target detection result.
S6, mapping the target frame to be a 3D frame under a camera coordinate system, and adding speed information on the 3D frame based on the first point cloud, wherein the speed information is obtained based on a plurality of acquisition speeds of corresponding targets acquired by a 4D millimeter wave radar in a fitting mode. The present embodiment obtains the velocity information by neural network fitting.
The method can realize accurate target detection at lower cost.
The above functions, if implemented in the form of software functional units and sold or used as a separate product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.
Claims (10)
1. A road side camera and 4D millimeter wave fused target detection method is characterized by comprising the following steps:
acquiring a first image of a road side camera and a first point cloud of a 4D millimeter wave radar;
projecting the first point cloud onto the first image, adding distance information on the first image, and acquiring a second image;
taking the second image as the input of a depth completion convolution network to obtain a third image, wherein the third image is a dense depth image;
converting the third image into a second point cloud, and encoding to generate a fourth image;
inputting the fourth image into a convolutional neural network for target detection to obtain a target frame;
and mapping the target frame into a 3D frame under a camera coordinate system, and adding speed information on the 3D frame based on the first point cloud.
2. The method for detecting the target fusing the roadside camera and the 4D millimeter wave according to claim 1, wherein the roadside camera and the 4D millimeter wave radar are calibrated in advance, and camera radar calibration information is stored.
3. The method for detecting the target fusing the roadside camera and the 4D millimeter waves according to claim 2, wherein the first point cloud is projected onto the first image based on the camera radar calibration information.
4. The method for detecting the target fusing the roadside camera and the 4D millimeter waves according to claim 2, wherein camera calibration parameters are obtained based on the camera radar calibration information, and the third image is converted into a second point cloud by the camera calibration parameters.
5. The roadside camera and 4D millimeter wave fused target detection method of claim 1, wherein the second point cloud generates a fourth image by BEV encoding.
6. The method for detecting the target fusing the roadside camera and the 4D millimeter wave according to claim 1, wherein the speed information is obtained based on a plurality of acquisition speeds of the corresponding target obtained by the 4D millimeter wave radar.
7. The method for detecting a target fusing a roadside camera and 4D millimeter waves according to claim 6, wherein the speed information is obtained by neural network fitting based on the plurality of acquisition speeds.
8. The method of claim 1, wherein the depth-padded convolutional network is a multi-view fused depth-padded convolutional network.
9. An electronic device, comprising:
one or more processors;
a memory; and
one or more programs stored in memory, the one or more programs including instructions for performing the roadside camera-4D millimeter wave fused object detection method of any of claims 1-8.
10. A computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for performing the roadside camera-4D millimeter wave fused object detection method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110853786.1A CN113655494A (en) | 2021-07-27 | 2021-07-27 | Target detection method, device and medium with road side camera and 4D millimeter wave integrated |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110853786.1A CN113655494A (en) | 2021-07-27 | 2021-07-27 | Target detection method, device and medium with road side camera and 4D millimeter wave integrated |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113655494A true CN113655494A (en) | 2021-11-16 |
Family
ID=78490730
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110853786.1A Pending CN113655494A (en) | 2021-07-27 | 2021-07-27 | Target detection method, device and medium with road side camera and 4D millimeter wave integrated |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113655494A (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001099930A (en) * | 1999-09-29 | 2001-04-13 | Fujitsu Ten Ltd | Sensor for monitoring periphery |
| US20020027503A1 (en) * | 2000-09-04 | 2002-03-07 | Takashi Higuchi | Periphery monitoring system |
| EP3222972A1 (en) * | 2016-03-25 | 2017-09-27 | LG Electronics Inc. | Display apparatus and vehicle having the same |
| US20180157920A1 (en) * | 2016-12-01 | 2018-06-07 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for recognizing obstacle of vehicle |
| EP3525131A1 (en) * | 2018-02-09 | 2019-08-14 | Bayerische Motoren Werke Aktiengesellschaft | Methods and apparatuses for object detection in a scene represented by depth data of a range detection sensor and image data of a camera |
| CN110414418A (en) * | 2019-07-25 | 2019-11-05 | 电子科技大学 | A kind of Approach for road detection of image-lidar image data Multiscale Fusion |
| JP2019194821A (en) * | 2018-05-06 | 2019-11-07 | 英俊 古川 | Target recognition device, target recognition method, and program |
| CN110532896A (en) * | 2019-08-06 | 2019-12-03 | 北京航空航天大学 | A kind of road vehicle detection method merged based on trackside millimetre-wave radar and machine vision |
| CN110794397A (en) * | 2019-10-18 | 2020-02-14 | 北京全路通信信号研究设计院集团有限公司 | Target detection method and system based on camera and radar |
| KR102132303B1 (en) * | 2019-11-13 | 2020-07-10 | 주식회사 동부아이씨티 | Traffic signal control system with automatic sensing of vehicle waiting for turn left signal using heat image camera and radar |
| CN111554088A (en) * | 2020-04-13 | 2020-08-18 | 重庆邮电大学 | Multifunctional V2X intelligent roadside base station system |
| US20200271473A1 (en) * | 2018-02-14 | 2020-08-27 | Tusimple, Inc. | Lane Marking Localization |
| US20200409372A1 (en) * | 2018-03-20 | 2020-12-31 | Huawei Technologies Co., Ltd. | Data fusion method and related device |
-
2021
- 2021-07-27 CN CN202110853786.1A patent/CN113655494A/en active Pending
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001099930A (en) * | 1999-09-29 | 2001-04-13 | Fujitsu Ten Ltd | Sensor for monitoring periphery |
| US20020027503A1 (en) * | 2000-09-04 | 2002-03-07 | Takashi Higuchi | Periphery monitoring system |
| EP3222972A1 (en) * | 2016-03-25 | 2017-09-27 | LG Electronics Inc. | Display apparatus and vehicle having the same |
| US20180157920A1 (en) * | 2016-12-01 | 2018-06-07 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for recognizing obstacle of vehicle |
| EP3525131A1 (en) * | 2018-02-09 | 2019-08-14 | Bayerische Motoren Werke Aktiengesellschaft | Methods and apparatuses for object detection in a scene represented by depth data of a range detection sensor and image data of a camera |
| US20200271473A1 (en) * | 2018-02-14 | 2020-08-27 | Tusimple, Inc. | Lane Marking Localization |
| US20200409372A1 (en) * | 2018-03-20 | 2020-12-31 | Huawei Technologies Co., Ltd. | Data fusion method and related device |
| JP2019194821A (en) * | 2018-05-06 | 2019-11-07 | 英俊 古川 | Target recognition device, target recognition method, and program |
| CN110414418A (en) * | 2019-07-25 | 2019-11-05 | 电子科技大学 | A kind of Approach for road detection of image-lidar image data Multiscale Fusion |
| CN110532896A (en) * | 2019-08-06 | 2019-12-03 | 北京航空航天大学 | A kind of road vehicle detection method merged based on trackside millimetre-wave radar and machine vision |
| CN110794397A (en) * | 2019-10-18 | 2020-02-14 | 北京全路通信信号研究设计院集团有限公司 | Target detection method and system based on camera and radar |
| KR102132303B1 (en) * | 2019-11-13 | 2020-07-10 | 주식회사 동부아이씨티 | Traffic signal control system with automatic sensing of vehicle waiting for turn left signal using heat image camera and radar |
| CN111554088A (en) * | 2020-04-13 | 2020-08-18 | 重庆邮电大学 | Multifunctional V2X intelligent roadside base station system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022022694A1 (en) | Method and system for sensing automated driving environment | |
| CN109143207B (en) | Laser radar internal reference precision verification method, device, equipment and medium | |
| CN110738121A (en) | front vehicle detection method and detection system | |
| CN107632308B (en) | Method for detecting contour of obstacle in front of vehicle based on recursive superposition algorithm | |
| CN111192295B (en) | Target detection and tracking method, apparatus, and computer-readable storage medium | |
| CN108109139B (en) | Airborne LIDAR three-dimensional building detection method based on gray voxel model | |
| CN113761999B (en) | Target detection method and device, electronic equipment and storage medium | |
| CN113340334B (en) | Sensor calibration method and device for unmanned vehicle and electronic equipment | |
| CN110378919B (en) | Narrow-road passing obstacle detection method based on SLAM | |
| WO2022142628A1 (en) | Point cloud data processing method and device | |
| CN110850394B (en) | Automatic driving laser radar intensity calibration method | |
| CN103679167A (en) | Method for processing CCD images | |
| CN114119850B (en) | Virtual and actual laser radar point cloud fusion method | |
| CN111080784A (en) | Ground three-dimensional reconstruction method and device based on ground image texture | |
| CN110927762A (en) | Positioning correction method, device and system | |
| US20230206500A1 (en) | Method and apparatus for calibrating extrinsic parameter of a camera | |
| CN108074232B (en) | Voxel segmentation-based airborne LIDAR building detection method | |
| CN116109601A (en) | Real-time target detection method based on three-dimensional laser radar point cloud | |
| CN113327296B (en) | Laser radar and camera online combined calibration method based on depth weighting | |
| CN117274749B (en) | Fused 3D target detection method based on 4D millimeter wave radar and image | |
| CN113988197A (en) | Multi-camera and multi-laser radar based combined calibration and target fusion detection method | |
| CN116467848B (en) | Millimeter wave radar point cloud simulation method and device | |
| CN111353481A (en) | Road obstacle identification method based on laser point cloud and video image | |
| CN113655494A (en) | Target detection method, device and medium with road side camera and 4D millimeter wave integrated | |
| CN116681932A (en) | Object identification method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |