CN116699602A - Target detection system and method based on millimeter wave radar and camera fusion - Google Patents
Target detection system and method based on millimeter wave radar and camera fusion Download PDFInfo
- Publication number
- CN116699602A CN116699602A CN202310619119.6A CN202310619119A CN116699602A CN 116699602 A CN116699602 A CN 116699602A CN 202310619119 A CN202310619119 A CN 202310619119A CN 116699602 A CN116699602 A CN 116699602A
- Authority
- CN
- China
- Prior art keywords
- millimeter wave
- wave radar
- target
- camera
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 87
- 230000004927 fusion Effects 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 20
- 238000012216 screening Methods 0.000 claims abstract description 10
- 238000006243 chemical reaction Methods 0.000 claims abstract description 6
- 238000012935 Averaging Methods 0.000 claims description 12
- 238000012360 testing method Methods 0.000 claims description 8
- 230000003068 static effect Effects 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 6
- 230000035559 beat frequency Effects 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 5
- 230000003287 optical effect Effects 0.000 claims description 3
- 238000000605 extraction Methods 0.000 abstract description 2
- 238000012544 monitoring process Methods 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 238000001914 filtration Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 235000016796 Euonymus japonicus Nutrition 0.000 description 1
- 240000006570 Euonymus japonicus Species 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000011897 real-time detection Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
- G01S13/867—Combination of radar systems with cameras
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/88—Radar or analogous systems specially adapted for specific applications
- G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
- G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
- G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
- G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
- G01S7/415—Identification of targets based on measurements of movement associated with the target
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/08—Detecting or categorising vehicles
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Theoretical Computer Science (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Electromagnetism (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
The invention provides a target detection system and method based on millimeter wave radar and camera fusion, which are characterized in that the millimeter wave radar and the camera are used for respectively detecting targets, after time matching alignment, the millimeter wave radar raw data acquired by the millimeter wave radar are subjected to algorithm processing such as distance FFT, doppler FFT, CFAR constant false alarm detection, DOA estimation and the like, 3D point cloud data of the targets of interest of the millimeter wave radar in a detection scene are obtained initially, then the 3D point cloud data of the targets of interest of the millimeter wave radar are subjected to algorithm processing such as preliminary screening, cluster fusion, effective target extraction and the like through a rule formulated by a real road condition, and then the camera data from the targets detected by the YOLOv4 algorithm at the same time are subjected to information fusion under the unified conversion of a space coordinate system, so that the advantage complementation of the detection performance of two sensors is realized, and the situations of incomplete, false detection and missed detection of single-sensor targets in an automatic driving system and an intelligent monitoring system are effectively solved.
Description
Technical Field
The invention relates to the technical field of radar signal processing, in particular to a target detection system and method based on millimeter wave radar and camera fusion.
Background
The multi-sensor information fusion technology mainly refers to an information processing process which is performed by automatically analyzing and integrating multi-sensor or multi-source information and data under a certain criterion by utilizing a computer technology so as to complete required decision and estimation. In the application of intelligent traffic systems, sensors for environmental sensing mainly include laser radar, millimeter wave radar, cameras, etc. Each sensor has its own advantages and disadvantages. For a vision sensor camera, the method has the advantages of low cost, rich information and easy perception classification, and the network structure with continuously improved detection rate and calculation efficiency comprises networks such as YOLO, fast-RCNN and the like, which proves that the target detection in the camera image using deep learning is definitely successful. However, the adaptability of the camera to the illumination environment is poor, three-dimensional information of the target is difficult to obtain, and the accuracy is low, so that the performance of the camera on target detection is greatly reduced in relatively bad environments such as rainy and foggy days or in environments with shielding.
The millimeter wave radar which is one of the most widely used automobile sensors has longer detection distance, higher measurement accuracy of the longitudinal distance and speed of the target, is not influenced by illumination conditions, can work in all weather, but cannot accurately identify the type of the target. Since the millimeter wave radar and the camera have extremely complementary characteristics, the fusion of the millimeter wave radar and the camera is most widely used in the research of a multi-sensor fusion technology. As a current research hotspot, a target detection technology based on millimeter wave radar and camera fusion is used, in the research of the technology, shigeki of Toyota automobile company (Toyota Motor Corporation) proposes a millimeter wave radar and camera combined calibration method based on homography transformation, and an occupied grid method is used for clustering targets; the method of foreign scholars Masaki is that a target point cloud point detected by millimeter wave radar is projected to an image pixel coordinate system to generate a target hypothesis region, and then target category identification is carried out through traditional visual detection; similarly, the university of Qinghua university Luo Xiao utilizes the same fusion concept, but in the detection of vehicles, the shadow of the bottom of the vehicle and the millimeter wave radar point are adopted to simultaneously extract the target hypothesis region, so that the condition of missed detection of the millimeter wave radar can be reduced. And the scholars Xinyu obtain target hypothesis areas by utilizing millimeter wave radar point projection and directly input the target hypothesis areas into the RPN network for target classification.
Based on the above analysis, in the current target detection of millimeter wave radar and camera information fusion, the following problems and disadvantages are often caused: 1) Information of millimeter wave radar is not fully utilized; 2) The data volume of the fusion system is too high, so that post-processing is difficult 3), and the processing speed is not as high as one stage due to the use of a two-stage visual target detection algorithm. Therefore, the novel target detection system and method based on the millimeter wave radar and camera fusion have great theoretical significance and practical significance.
Disclosure of Invention
In view of the above, the invention aims to provide a target detection system and method based on millimeter wave radar and camera fusion, aiming at solving the problems that the existing target detection based on a single sensor is incomplete in information, poor in robustness, seriously influenced by external environment, and easy to miss detection and miss detection. The millimeter wave radar target detection processing algorithm in the method mainly utilizes the distance FFT, doppler FFT, CFAR constant false alarm detection, DOA estimation and other algorithm processing, and then combines a series of rules set in the actual road condition to screen and reject data, thereby improving the detection precision of the target and reducing the data processing capacity in the later period. The camera video target detection mainly uses a YOLOv4 network with detection precision and detection speed, and a data set used for training the YOLOv4 network is obtained by making a vehicle picture acquired under a real road condition. The sampling value of the millimeter wave radar and the video acquired by the camera are used as input, the millimeter wave radar interesting target and the camera interesting target are obtained through the radar algorithm and the YOLOv4 algorithm under the time matching, then the target matching and the target fusion are carried out under the transformation of a space coordinate system, the information of the two sensors is effectively integrated, the information of the target is more comprehensive, the defect of a single sensor is overcome, and the conditions of false detection and missing detection are reduced.
In order to achieve the above object, a first aspect of the present invention provides a target detection method based on fusion of millimeter wave radar and a camera, the method comprising the steps of:
s1, detecting a detection scene by using a millimeter wave radar and a camera at the same time, so as to realize time matching of original data of the millimeter wave radar and acquired data of the camera;
s2, processing millimeter wave radar original data obtained through time matching through a preset radar algorithm to obtain distance, speed and angle information of a detection scene millimeter wave radar target of interest, and thus obtaining 3D point cloud data of the detection scene millimeter wave radar target of interest;
s3, screening out the static targets, the invalid targets and the empty targets in the 3D point cloud data obtained in the step S2 through transverse distance and longitudinal distance limitation of the actual lane condition and a preset rule to obtain 3D point cloud data with higher reliability;
s4, obtaining a target cluster after DBSCAN clustering fusion of the 3D point cloud data with higher reliability obtained in the step S3, obtaining a new cluster by screening points in the cluster, and obtaining a 3D coordinate point capable of representing a target position by distance averaging;
s5, performing target detection and ranging model algorithm detection on camera acquisition data obtained through time matching by using a YOLOv4 network to obtain a camera interest target and a camera detection distance of the target;
s6, establishing a camera coordinate system by taking an optical center of the camera as a center, establishing a millimeter wave radar coordinate system by taking the center of a millimeter wave radar acquisition board, and calculating to obtain conversion relations among the millimeter wave radar coordinate system, the camera coordinate system, the world coordinate system, the image coordinate system and the pixel coordinate system;
and S7, converting the 3D coordinate points capable of representing the target positions into pixel coordinate systems according to the conversion relations among the coordinate systems in the step S6, obtaining 2D pixel points under the pixel coordinate systems, and carrying out target matching and target fusion according to the two groups of relations, namely whether the 2D pixel points are in the target recognition frame obtained in the step S5 or not, and the difference value of the longitudinal distance of the target measured by the millimeter wave radar and the longitudinal distance of the target measured by the camera.
Further, in step S1, time matching between the millimeter wave radar original data and the camera acquired data is achieved, specifically: and taking the millimeter wave radar original data acquisition time with relatively low frequency as a reference, and performing backward compatibility on the camera acquired data with relatively high frequency.
Further, the step S2 specifically includes the following steps:
s21, processing the original data acquired by the millimeter wave radar by using a distance FFT and Doppler FFT algorithm to obtain distance and speed information of an interest target of the millimeter wave radar in a detection scene;
s22, setting an optimal CFAR constant false alarm detection threshold according to the intensity analysis of the received signals to obtain the number of millimeter wave radar interest targets;
s23, according to the number of the millimeter wave radar interest targets, angle information of the millimeter wave radar interest targets is obtained through processing by using a DOA estimation algorithm, and a 3d point cloud image of the millimeter wave radar interest targets is drawn according to the information obtained in the steps S21, S22 and S23.
Further, the step S21 specifically includes the following steps:
s211, performing 3-dimensional FFT on the original data acquired by the millimeter wave radar, and then detecting the peak position through peak search to obtain beat frequency and Doppler frequency;
s212, subtracting the Doppler frequency from the beat frequency to obtain an intermediate frequency signal, and obtaining the distance and the speed of the millimeter wave radar interest target according to the relation between the frequency and the actual distance and the speed.
Further, the step S3 specifically includes the following steps:
s31, setting the transverse distance as three standard lane distances according to the actual lane distance, and eliminating the interference of trees at two ends of the lane, garbage cans, guardrails and targets of non-adjacent lanes;
s32, according to the maximum detection distance of the millimeter wave radar test, setting the longitudinal distance to be 2m-200m, and eliminating the interference of false targets caused by distant non-interested target vehicles and short-distance ground clutter reflection;
and S33, eliminating point cloud data with the target parameter of 0 according to the condition of the 3D point cloud data, and eliminating interference of empty targets and invalid targets.
Further, the step S4 specifically includes the following steps:
s41, performing comparison analysis according to the length and width of an actual vehicle and a test result to obtain an appointed radius value and a minimum point minPts of an optimal DBSCAN cluster fusion algorithm;
s42, performing DBSCAN cluster fusion on the obtained high-reliability point cloud data to form clusters;
s43, screening the points in each cluster to obtain a new cluster, and carrying out distance averaging on the points in the new cluster to obtain a 3D coordinate point capable of representing the target position.
Further, the step S43 specifically includes the following steps:
s431, selecting the mode of the speed values of all points for each formed cluster to be v ref The speeds of other points are v respectively i (i.gtoreq.3), setting a threshold v of speed thold 0.05m/s;
s432, subtracting the speed of each point except the reference point from the speed of the reference point in each formed cluster;
s433, if the point after subtraction operation meets the formula 1, the point is reserved, if the point does not meet the formula 1, the point is rejected, each formed cluster is rejected to form a new cluster, and the formula 1 is as follows:
|v ref -v i |<v thold
and S434, respectively averaging the distances of the points in the new cluster in the x direction, the y direction and the z direction to obtain a 3d coordinate point which can represent the target position.
Further, the step S7 specifically includes the following steps:
s71, averaging the point distances in each new cluster to obtain a 3D coordinate point capable of representing the target position, and converting the 3D coordinate point capable of representing the target position into a pixel point under a pixel coordinate system through the coordinate relation established in the step S6;
s72, if the converted pixel points are satisfied in the visual recognition frame, and the longitudinal distance d of the target measured by the camera c Target longitudinal distance d measured by millimeter wave radar r Satisfying equation 2 is regarded as successful matching, and equation 2 is as follows:
|d c -d r |<d thold
wherein d is thold If one of the two conditions is not satisfied, the matching is unsuccessful;
s73, if the matching is unsuccessful, carrying out specific judgment according to the condition that the condition is not satisfied, including: when the camera is successful in detection and the millimeter wave radar is unsuccessful in detection, judging according to the confidence coefficient obtained by visually detecting the target, and considering that the target exists if the confidence coefficient is larger than 0.75, and preliminarily adopting a target distance value measured by the camera to match the next frame of data with the millimeter wave radar; when the millimeter wave radar is successfully detected and the camera is not successfully detected, processing and tracking the acquired data of the millimeter wave radar for multiple frames, if the target is not lost, the target exists, the data detected by the millimeter wave radar is directly used, and if the target is lost, the interference target is discarded.
A second aspect of the present invention provides an object detection system based on a millimeter wave radar and camera fusion, the system comprising a millimeter wave radar, a camera and a computer, the millimeter wave radar and the camera being respectively connected to the computer, the computer being configured to implement the method of the first aspect.
Compared with the prior art, the invention has the beneficial effects that:
according to the target detection system and method based on millimeter wave radar and camera fusion, targets are detected by utilizing a system built by the millimeter wave radar and a camera, then on the premise of time matching of two paths of data, after the millimeter wave radar data are processed by using a radar correlation algorithm, the obtained millimeter wave radar data are subjected to data screening and removing, clustering fusion, effective target extraction and other processes to obtain high-reliability targets, camera video data are subjected to target detection by utilizing a vehicle target picture of real road running as a training set to train an obtained YOLOv4 network to obtain high-reliability camera targets of interest, and then the millimeter wave radar targets of interest are converted into an image coordinate system to be matched with the camera targets of interest based on the position relation between the camera and the millimeter wave radar in an established hardware platform and the conversion relation of the coordinate system, and then are fused. The invention can well fuse the information of the two sensors and has better robustness. Meanwhile, compared with some existing multi-sensor fusion methods, the method provided by the invention has the advantages that the millimeter wave radar invalid data is screened and filtered in advance, and targets can be effectively extracted by using a clustering DBSCAN algorithm commonly used in engineering; in the aspect of camera target detection, a one-stage YOLOv4 algorithm is used, so that the method has higher detection precision and higher speed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only preferred embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic overall flow chart of a target detection method based on millimeter wave radar and camera fusion provided by an embodiment of the invention.
Fig. 2 is a schematic diagram of sensor time matching according to an embodiment of the present invention.
Fig. 3 is a block diagram of a millimeter wave radar test provided by an embodiment of the present invention.
Fig. 4 is a schematic diagram of millimeter wave radar DOA estimation provided by an embodiment of the present invention.
Fig. 5 is a point cloud diagram of a detection scene of an object provided by an embodiment of the present invention.
Fig. 6 is a diagram of targets provided by an embodiment of the present invention after screening by proposed rules.
Fig. 7 is a graph of actual distance and velocity of a target provided by an embodiment of the present invention.
FIG. 8 is a graph of the effect of 3d coordinate points that can represent a target after the processing of the DBSCAN cluster fusion algorithm and the distance averaging according to the embodiment of the invention.
Fig. 9 is a diagram of an effect of detecting an object based on fusion of millimeter wave radar and a camera according to an embodiment of the present invention.
Detailed Description
The principles and features of the present invention are described below with reference to the drawings, the illustrated embodiments are provided for the purpose of illustrating the invention and are not to be construed as limiting the scope of the invention.
Referring to fig. 1, the present embodiment provides a target detection method based on millimeter wave radar and camera fusion, including the steps of:
s1, acquiring data of a detection scene by adopting a millimeter wave radar and a camera which are arranged on the same tripod, and performing time matching alignment on acquired millimeter wave radar original data and camera video data. The frame rate of the millimeter wave radar is 25FPS, and the frame rate of the camera is 30FPS; the corresponding frequencies are 40ms and 33ms, respectively. In order to ensure the accuracy of fusion, the most appropriate time matching mode is to select data at the same moment, namely, select the common multiple of 40 and 33, but the data can be lost too much, so that the effect of real-time detection of the target can not be achieved. For the acquired millimeter wave radar original data and camera video data of a detection scene, the data measurement time of the millimeter wave radar with lower frequency is used as a reference, so that backward compatibility of the camera data with higher frequency is realized, as shown in fig. 2, the time error of the data is not more than 33ms at maximum, and according to the highest time speed of 70km/h specified by urban roads in China, the pixel offset caused by converting the movement of an object into a pixel coordinate system is calculated, so that the data measurement time of the millimeter wave radar with lower frequency is used as a reference, and the backward compatibility of the camera data with higher frequency is realized, thereby realizing the time matching of two sensor data.
S2, as shown in FIG. 2, the millimeter wave radar raw data of the 0ms, 40ms, 80ms, 120ms and the like obtained from the time matching in the step S1 are processed by using a related radar detection algorithm to obtain the distance and speed information of the related targets of the detection scene, so that the obtained 3D point cloud data of the targets of interest of the millimeter wave radar of the detection scene is obtained. The specific steps for processing and acquiring the information of the related targets by using the related radar detection algorithm are as follows:
s211, performing 3-dimensional FFT on data acquired by millimeter wave radar, detecting peak position (x, y, z) by peak search, and obtaining beat frequency f e ：
Doppler frequency f d The method comprises the following steps:
wherein f s Is the sampling frequency, N dFFT And N vFFT FFT points in the distance and velocity dimensions, respectively.
S212, in actual condition, doppler shift is generated due to movement of the target, so as to obtain the true intermediate frequency signal frequency f IF By beat frequency f e Subtracting Doppler frequency f d Obtained finally according to the relation between the frequency and the actual distance and speedThe distance d from the target is:
the speed v of the target is:
where c is the speed of light and k is the rate of change of the frequency modulated continuous wave signal over time.
S22, setting an optimal CFAR constant false alarm detection threshold according to the intensity analysis of the received signals to obtain the number of the interest targets, and effectively reducing the false alarm and missed detection problems.
S23, estimating the angle of an object by the millimeter wave radar angle, wherein at least two receiving antennas are needed, as shown in fig. 4, a chirp is sent through the transmitting antenna, after the object is transmitted, the signal is respectively received by the first antenna and the second antenna, the receiving distance of the first antenna is d, the receiving distance of the second antenna is d+Δd, and the additional distance can generate a phase difference, so that the frequency omega can be obtained as follows:
assuming that the signals received by the two antennas are parallel, as shown in fig. 4, there is Δd=dsin θ, and the frequency ω can be converted into:
the angle θ of the target is available as it is.
S3, the 3D point cloud data of the detection scene millimeter wave radar interest target obtained in the step S2 not only comprises the point cloud data of the target, but also comprises the point cloud data of invalid targets and some empty targets generated by interference of other clutter, such as interference targets like garbage cans, billboards, guardrails, trees and the like on the roadside. In order to reduce the calculation amount of post-processing and improve the processing speed, the specific steps of filtering the static target, the invalid target and the empty target by limiting the transverse distance and the longitudinal distance of the actual lane condition and setting related rules to obtain the point cloud data with high reliability are as follows:
s31, as the targets of the non-adjacent lanes are not main detection objects, the proper transverse distance is set as the standard distance (-5.4 m-5.4 m) of the own lane and the three lanes including the two adjacent lanes according to the actual distance of the lanes, and the interference of trees, garbage cans, guardrails and the targets of the non-adjacent lanes at the two ends of the lanes can be eliminated.
In the actual test, a false target caused by short-distance ground clutter reflection occurs. And then, according to the maximum detection distance of the millimeter wave radar test, the longitudinal distance is set to be 2-200 m, so that the interference of false targets caused by distant non-interested target vehicles and short-distance ground clutter reflection can be effectively eliminated.
S33, analyzing a large amount of measured data to obtain that some empty targets with parameters of 0 exist in the point cloud data, and eliminating the empty targets according to whether the target parameters are 0 or not.
Further, in order to verify the validity of the algorithm for filtering the static target, the invalid target and the empty target by setting the relevant rule provided by the invention to obtain the point cloud data with high reliability, the static target data and the dynamic target data under different detections are adopted to verify, and the result is shown in fig. 5, and as can be seen from fig. 5, the algorithm for filtering the static target, the invalid target and the empty target based on the setting the relevant rule to obtain the point cloud data with high reliability can effectively filter the point cloud data of the static target, the invalid target and the empty target, and effectively reduce the data amount processed in the later stage and the fused interference.
S4, one target often corresponds to a plurality of point cloud data points, after the point cloud data with high reliability obtained in the step S3 is subjected to DBSCAN cluster fusion, the point cloud data corresponding to the same target is found to form a target cluster, the point cloud in the formed cluster is screened to form a new cluster, and then the specific steps of obtaining the 3D coordinate point capable of representing the target position through distance averaging are as follows:
s41, comparing and analyzing according to the length and width of the actual vehicle and a large number of experimental results to obtain the optimal designated radius epsilon value of the DBSCAN cluster fusion algorithm as 2 and the minimum point minPts as 4.
S42, performing DBSCAN cluster fusion on the obtained high-reliability point cloud data to form clusters.
After forming the target cluster, the point cloud data of the non-own target possibly exist in the target cluster, the point cloud data need to be removed, and then the distances in the x direction, the y direction and the z direction are respectively averaged for the points in each formed new cluster, so that the specific steps for obtaining the points capable of representing the target position are as follows:
s431, selecting the mode of the speed values of all points for each formed cluster to be v ref The speeds of other points are v respectively i (i.gtoreq.3), setting a threshold v of speed thold 0.05m/s;
s432, subtracting the speed of each point except the reference point from the speed of the reference point in each formed cluster to obtain a speed difference value with the reference point;
s433, if the point after subtraction operation meets the formula 1, the point is reserved, if the point does not meet the formula 1, the point is rejected, each formed cluster is rejected to form a new cluster, and the formula 1 is as follows:
|v ref -v i |<v thold
and S434, respectively averaging the distances of the points in the new cluster in the x direction, the y direction and the z direction to obtain a 3d coordinate point which can represent the target position.
Further, in order to verify the effectiveness of the 3D coordinate point algorithm capable of representing the target position based on the distance average after DBSCAN cluster fusion, DBSCAN cluster fusion is performed on the 3D point cloud data obtained through primary screening of the detection scene, the designated radius epsilon value of the cluster fusion is set to be 2, the minimum point minPts is set to be 4 according to actual conditions, and as shown in the result of FIG. 6, compared with the target position of FIG. 7, the DBSCAN cluster fusion algorithm can effectively fuse the point cloud points of the target into clusters. Meanwhile, as can be seen from comparison of the target positions in fig. 8 and fig. 7, the 3D coordinate points which can effectively represent the target positions can be obtained after the point distances of the new clusters obtained after the points in the clusters are screened and removed are averaged.
S5, 18000 vehicle images in an actual road environment are used as a training set, 2000 vehicle images are used as a test set, and the YOLOv4 network is trained with Batch Size equal to 100 and 180 epochs. And (3) performing target detection on the camera video data obtained by the time matching in the step (S1) by the obtained YOLOv4 network to obtain a video interest target, drawing a recognition frame and confidence level, and measuring a target distance by the camera.
S6, a camera coordinate system is established by taking an optical center of the camera as a center, a millimeter wave radar coordinate system is established by taking a millimeter wave radar acquisition board center, the camera coordinate system is selected as a world coordinate system, the camera is calibrated by a Zhang Zhengyou calibration method to obtain an internal reference matrix, an external reference matrix and a distortion function of the camera, and the camera and the millimeter wave radar are combined with the installation position, so that the relations of the millimeter wave radar coordinate system, the camera coordinate system, the world coordinate system, the picture coordinate system and the pixel coordinate system are obtained.
And S7, converting the 3D point cloud data which can represent the target position and is obtained in the step S4 into a pixel coordinate system through the relation between the coordinate systems in the step S6, and obtaining the pixel point of the millimeter wave radar target point in the pixel coordinate system. And (3) performing target matching and target fusion according to the distance between the pixel point and the center point of the target recognition frame obtained in the step (S5) and the two groups of relations of the target longitudinal distance measured by the radar and the target longitudinal distance measured by the camera, wherein the specific steps are as follows:
and S71, averaging the point distances in each new cluster to obtain point cloud data points which can represent the target, and converting the point cloud data points which can represent the target into pixel points in a pixel coordinate system through the coordinate relation established in the step S6.
S72, if the converted pixel point is fullThe foot is in the visual recognition frame and the longitudinal distance d of the target measured by the camera c Target longitudinal distance d measured by millimeter wave radar r Satisfying equation 2 is regarded as successful matching, and equation 2 is as follows:
|d c -d r |<d thold
wherein d is thold If one of the two conditions is not satisfied, the matching is unsuccessful.
S73, if the matching is unsuccessful, carrying out specific judgment according to the condition that the condition is not satisfied, including: when the camera is successful in detection and the millimeter wave radar is unsuccessful in detection, judging according to the confidence coefficient obtained by visually detecting the target, and considering that the target exists if the confidence coefficient is larger than 0.75, and preliminarily adopting a target distance value measured by the camera to match the next frame of data with the millimeter wave radar; when the millimeter wave radar is successfully detected and the camera is not successfully detected, processing and tracking the acquired data of the millimeter wave radar for multiple frames, if the target is not lost, the target exists, the data detected by the millimeter wave radar is directly used, and if the target is lost, the interference target is discarded.
The invention further provides a target detection system based on the fusion of the millimeter wave radar and the camera, the system comprises the millimeter wave radar, the camera and a computer, the millimeter wave radar and the camera are respectively connected with the computer, and the computer is used for realizing the method in the embodiment of the method. The working principle and the beneficial effects of the system can refer to the foregoing method embodiments, and are not described herein.
In order to verify the effect of the target detection system and method based on the millimeter wave radar and camera fusion, the established millimeter wave radar and camera acquisition system is adopted to be placed at a road intersection for acquisition data verification, and the acquired data processing result shows that as shown in fig. 9, the target detection system and method based on the millimeter wave radar and camera fusion can effectively fuse information from the millimeter wave radar and the camera, effectively integrate the advantages of the two sensors, solve the problem of incomplete target information caused by single sensor detection, and reduce the situations of false detection and missed detection to a certain extent.
The foregoing description of the preferred embodiments of the invention is not intended to limit the invention to the precise form disclosed, and any such modifications, equivalents, and alternatives falling within the spirit and scope of the invention are intended to be included within the scope of the invention.
Claims (9)
1. The target detection method based on the fusion of the millimeter wave radar and the camera is characterized by comprising the following steps:
s1, detecting a detection scene by using a millimeter wave radar and a camera at the same time, so as to realize time matching of original data of the millimeter wave radar and acquired data of the camera;
s2, processing millimeter wave radar original data obtained through time matching through a preset radar algorithm to obtain distance, speed and angle information of a detection scene millimeter wave radar target of interest, and thus obtaining 3D point cloud data of the detection scene millimeter wave radar target of interest;
s3, screening out the static targets, the invalid targets and the empty targets in the 3D point cloud data obtained in the step S2 through transverse distance and longitudinal distance limitation of the actual lane condition and a preset rule to obtain 3D point cloud data with higher reliability;
s4, obtaining a target cluster after DBSCAN clustering fusion of the 3D point cloud data with higher reliability obtained in the step S3, obtaining a new cluster by screening points in the cluster, and obtaining a 3D coordinate point capable of representing a target position by distance averaging;
s5, performing target detection and ranging model algorithm detection on camera acquisition data obtained through time matching by using a YOLOv4 network to obtain a camera interest target and a camera detection distance of the target;
s6, establishing a camera coordinate system by taking an optical center of the camera as a center, establishing a millimeter wave radar coordinate system by taking the center of a millimeter wave radar acquisition board, and calculating to obtain conversion relations among the millimeter wave radar coordinate system, the camera coordinate system, the world coordinate system, the image coordinate system and the pixel coordinate system;
and S7, converting the 3D coordinate points capable of representing the target positions into pixel coordinate systems according to the conversion relations among the coordinate systems in the step S6, obtaining 2D pixel points under the pixel coordinate systems, and carrying out target matching and target fusion according to the two groups of relations, namely whether the 2D pixel points are in the target recognition frame obtained in the step S5 or not, and the difference value of the longitudinal distance of the target measured by the millimeter wave radar and the longitudinal distance of the target measured by the camera.
2. The method for detecting the target based on the fusion of the millimeter wave radar and the camera according to claim 1, wherein the time matching of the original data of the millimeter wave radar and the acquired data of the camera is realized in the step S1, specifically: and taking the millimeter wave radar original data acquisition time with relatively low frequency as a reference, and performing backward compatibility on the camera acquired data with relatively high frequency.
3. The method for detecting a target based on the fusion of millimeter wave radar and camera according to claim 1, wherein the step S2 specifically comprises the steps of:
s21, processing the original data acquired by the millimeter wave radar by using a distance FFT and Doppler FFT algorithm to obtain distance and speed information of an interest target of the millimeter wave radar in a detection scene;
s22, setting an optimal CFAR constant false alarm detection threshold according to the intensity analysis of the received signals to obtain the number of millimeter wave radar interest targets;
s23, according to the number of the millimeter wave radar interest targets, angle information of the millimeter wave radar interest targets is obtained through processing by using a DOA estimation algorithm, and a 3d point cloud image of the millimeter wave radar interest targets is drawn according to the information obtained in the steps S21, S22 and S23.
4. The method for detecting an object based on the fusion of millimeter wave radar and camera according to claim 1, wherein the step S21 specifically comprises the steps of:
s211, performing 3-dimensional FFT on the original data acquired by the millimeter wave radar, and then detecting the peak position through peak search to obtain beat frequency and Doppler frequency;
s212, subtracting the Doppler frequency from the beat frequency to obtain an intermediate frequency signal, and obtaining the distance and the speed of the millimeter wave radar interest target according to the relation between the frequency and the actual distance and the speed.
5. The method for detecting a target based on the fusion of millimeter wave radar and camera according to claim 4, wherein the step S3 specifically comprises the steps of:
s31, setting the transverse distance as three standard lane distances according to the actual lane distance, and eliminating the interference of trees at two ends of the lane, garbage cans, guardrails and targets of non-adjacent lanes;
s32, according to the maximum detection distance of the millimeter wave radar test, setting the longitudinal distance to be 2m-200m, and eliminating the interference of false targets caused by distant non-interested target vehicles and short-distance ground clutter reflection;
and S33, eliminating point cloud data with the target parameter of 0 according to the condition of the 3D point cloud data, and eliminating interference of empty targets and invalid targets.
6. The method for detecting a target based on the fusion of millimeter wave radar and camera according to claim 1, wherein the step S4 specifically comprises the steps of:
s41, performing comparison analysis according to the length and width of an actual vehicle and a test result to obtain an appointed radius epsilon value and a minimum point minPts of an optimal DBSCAN cluster fusion algorithm;
s42, performing DBSCAN cluster fusion on the obtained high-reliability point cloud data to form clusters;
s43, screening the points in each cluster to obtain a new cluster, and carrying out distance averaging on the points in the new cluster to obtain a 3D coordinate point capable of representing the target position.
7. The method for detecting an object based on the fusion of millimeter wave radar and camera according to claim 6, wherein step S43 specifically comprises the steps of:
s431, for eachThe clusters formed all first select the mode of the velocity values of all points as v ref The speeds of other points are v respectively i (i.gtoreq.3), setting a threshold v of speed thold 0.05m/s;
s432, subtracting the speed of each point except the reference point from the speed of the reference point in each formed cluster;
s433, if the point after subtraction operation meets the formula 1, the point is reserved, if the point does not meet the formula 1, the point is rejected, each formed cluster is rejected to form a new cluster, and the formula 1 is as follows:
|v ref -v i |<v thold
and S434, respectively averaging the distances of the points in the new cluster in the x direction, the y direction and the z direction to obtain a 3d coordinate point which can represent the target position.
8. The method for detecting a target based on the fusion of millimeter wave radar and camera according to claim 1, wherein the step S7 specifically comprises the steps of:
s71, averaging the point distances in each new cluster to obtain a 3D coordinate point capable of representing the target position, and converting the 3D coordinate point capable of representing the target position into a pixel point under a pixel coordinate system through the coordinate relation established in the step S6;
s72, if the converted pixel points are satisfied in the visual recognition frame, and the longitudinal distance d of the target measured by the camera c Target longitudinal distance d measured by millimeter wave radar r Satisfying equation 2 is regarded as successful matching, and equation 2 is as follows:
|d c -d r |<d thold
wherein d is thold If one of the two conditions is not satisfied, the matching is unsuccessful;
s73, if the matching is unsuccessful, carrying out specific judgment according to the condition that the condition is not satisfied, including: when the camera is successful in detection and the millimeter wave radar is unsuccessful in detection, judging according to the confidence coefficient obtained by visually detecting the target, and considering that the target exists if the confidence coefficient is larger than 0.75, and preliminarily adopting a target distance value measured by the camera to match the next frame of data with the millimeter wave radar; when the millimeter wave radar is successfully detected and the camera is not successfully detected, processing and tracking the acquired data of the millimeter wave radar for multiple frames, if the target is not lost, the target exists, the data detected by the millimeter wave radar is directly used, and if the target is lost, the interference target is discarded.
9. An object detection system based on a combination of a millimeter wave radar and a camera, wherein the system comprises the millimeter wave radar, the camera and a computer, the millimeter wave radar and the camera being respectively connected to the computer, the computer being adapted to implement the method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310619119.6A CN116699602A (en) | 2023-05-29 | 2023-05-29 | Target detection system and method based on millimeter wave radar and camera fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310619119.6A CN116699602A (en) | 2023-05-29 | 2023-05-29 | Target detection system and method based on millimeter wave radar and camera fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116699602A true CN116699602A (en) | 2023-09-05 |
Family
ID=87825007
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310619119.6A Pending CN116699602A (en) | 2023-05-29 | 2023-05-29 | Target detection system and method based on millimeter wave radar and camera fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116699602A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117093872A (en) * | 2023-10-19 | 2023-11-21 | 四川数字交通科技股份有限公司 | Self-training method and system for radar target classification model |
| CN117949942A (en) * | 2024-03-26 | 2024-04-30 | 北京市计量检测科学研究院 | Target tracking method and system based on fusion of radar data and video data |
-
2023
- 2023-05-29 CN CN202310619119.6A patent/CN116699602A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117093872A (en) * | 2023-10-19 | 2023-11-21 | 四川数字交通科技股份有限公司 | Self-training method and system for radar target classification model |
| CN117093872B (en) * | 2023-10-19 | 2024-01-02 | 四川数字交通科技股份有限公司 | Self-training method and system for radar target classification model |
| CN117949942A (en) * | 2024-03-26 | 2024-04-30 | 北京市计量检测科学研究院 | Target tracking method and system based on fusion of radar data and video data |
| CN117949942B (en) * | 2024-03-26 | 2024-06-07 | 北京市计量检测科学研究院 | Target tracking method and system based on fusion of radar data and video data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111368706B (en) | Data fusion dynamic vehicle detection method based on millimeter wave radar and machine vision | |
| CN109444911B (en) | Unmanned ship water surface target detection, identification and positioning method based on monocular camera and laser radar information fusion | |
| CN111369541B (en) | Vehicle detection method for intelligent automobile under severe weather condition | |
| CN103176185B (en) | Method and system for detecting road barrier | |
| WO2022141914A1 (en) | Multi-target vehicle detection and re-identification method based on radar and video fusion | |
| CN112946628A (en) | Road running state detection method and system based on radar and video fusion | |
| CN112215306B (en) | Target detection method based on fusion of monocular vision and millimeter wave radar | |
| CN116699602A (en) | Target detection system and method based on millimeter wave radar and camera fusion | |
| CN114565900A (en) | Target detection method based on improved YOLOv5 and binocular stereo vision | |
| CN105225482A (en) | Based on vehicle detecting system and the method for binocular stereo vision | |
| CN113850102B (en) | Vehicle-mounted vision detection method and system based on millimeter wave radar assistance | |
| CN105160649A (en) | Multi-target tracking method and system based on kernel function unsupervised clustering | |
| CN111476762B (en) | Obstacle detection method and device of inspection equipment and inspection equipment | |
| CN111123262B (en) | Automatic driving 3D modeling method, device and system | |
| CN109541601A (en) | Differentiating obstacle and its detection method based on millimeter wave | |
| CN112906777A (en) | Target detection method and device, electronic equipment and storage medium | |
| CN113627373A (en) | Vehicle identification method based on radar-vision fusion detection | |
| CN115690713A (en) | Binocular camera-based radar-vision fusion event detection method | |
| CN112784679A (en) | Vehicle obstacle avoidance method and device | |
| CN116978009A (en) | Dynamic object filtering method based on 4D millimeter wave radar | |
| CN117111085A (en) | Automatic driving automobile road cloud fusion sensing method | |
| CN115690746A (en) | Non-blind area sensing method and system based on vehicle-road cooperation | |
| CN114092778A (en) | Radar camera data fusion system and method based on characterization learning | |
| CN117197019A (en) | Vehicle three-dimensional point cloud image fusion method and system | |
| CN110865367A (en) | Intelligent fusion method for radar video data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |