CN114818819A - Road obstacle detection method based on millimeter wave radar and visual signal - Google Patents
Road obstacle detection method based on millimeter wave radar and visual signal Download PDFInfo
- Publication number
- CN114818819A CN114818819A CN202210493558.2A CN202210493558A CN114818819A CN 114818819 A CN114818819 A CN 114818819A CN 202210493558 A CN202210493558 A CN 202210493558A CN 114818819 A CN114818819 A CN 114818819A
- Authority
- CN
- China
- Prior art keywords
- millimeter wave
- wave radar
- visual
- target
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 87
- 230000000007 visual effect Effects 0.000 title claims abstract description 84
- 238000013507 mapping Methods 0.000 claims abstract description 16
- 238000000034 method Methods 0.000 claims abstract description 16
- 230000004927 fusion Effects 0.000 claims abstract description 12
- 238000000605 extraction Methods 0.000 claims description 11
- 238000001914 filtration Methods 0.000 claims description 10
- 230000001360 synchronised effect Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 5
- 238000013527 convolutional neural network Methods 0.000 claims description 5
- 230000003287 optical effect Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 4
- 238000003860 storage Methods 0.000 claims description 4
- 238000013528 artificial neural network Methods 0.000 claims description 3
- 230000001629 suppression Effects 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 description 12
- 238000006243 chemical reaction Methods 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008447 perception Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 241000282472 Canis lupus familiaris Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/02—Preprocessing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Signal Processing (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
A road obstacle detection method based on millimeter wave radar and visual signals detects obstacles in front of a road through a millimeter wave radar and a visual sensor, detects the type of a target object by fusing two signals of the millimeter wave radar and the visual image, firstly preprocesses the millimeter wave radar signal, converts a millimeter wave radar message into a form of a target point position, and performs time synchronization with the visual image signal; mapping a millimeter wave radar target point on the visual image signal according to the coordinate matching relation to realize the fusion of the two modes; and performing target detection on the millimeter wave radar information mapped to the visual image signal, and finally performing scale estimation on a detection result to refine the obstacle information. The method realizes multi-mode fine-grained obstacle detection, has the advantages of accuracy and operation efficiency compared with the traditional single-mode method, and has high practical value.
Description
Technical Field
The invention belongs to the technical field of artificial intelligence and machine learning, relates to a convolutional neural network and knowledge base technology, and particularly relates to a road obstacle detection method based on a millimeter wave radar and a visual signal.
Background
With the development of artificial intelligence technology, the automatic driving technology is getting into more and more vision of people. The automatic driving integrates a plurality of technologies such as automatic control, intelligent environment perception, optimization and the like, and is a highly developed product of artificial intelligence. The intelligent environment perception is particularly important in the automatic driving technology, and the perception of obstacles provides an important basis for vehicle control planning. Only by real-time and accurate obstacle sensing and detection, the automobile which automatically runs can effectively avoid obstacles, safety accidents such as collision and the like are avoided, and the safety performance of automatic driving is ensured.
Obstacle sensing at the present stage is mainly based on optical sensors, vision sensors or millimeter wave radar sensors.
The optical sensor is used for installing an optical distance measuring instrument on the vehicle body and detecting the distance of the peripheral direction of the vehicle body. The disadvantage is that distance information can be acquired as much as possible without specific category information. And the distance measuring device is a fixing device, can only sense the fixed direction and has lower reliability. In addition, the instrument is exposed outside the vehicle and is easy to damage.
The radar detection is to extract the characteristics capable of reflecting the target attribute information from the radar signal echo characteristics of the target, and the machine makes the type or model judgment on the target according to a certain judgment criterion. The millimeter wave short-range detection radar has the advantages of small volume, light weight, easy high integration, wide frequency band, high resolution, strong anti-interference performance, better all-weather working capability and wide application scene. However, the robustness of target feature extraction is seriously reduced by factors such as dense distribution, large quantity, various types, various and changeable motion states, complex clutter interference and the like of road scene targets for automatic driving application; the millimeter wave radar target characteristic data is incomplete, the reliability is low, the problems of complex environment of manually designed identification features, poor adaptability and the like further limit the engineering popularization capability and performance improvement of the target identification algorithm. The complexity of the clutter environment, the technical bottleneck of target identification and the like, so that the millimeter wave radar detection is applied to an automatic driving scene independently and has great difficulty.
Another common approach is to perform object detection around the vehicle based on a visual approach. The research of the vision-based target detection method mainly focuses on the aspects of optimization, performance improvement and the like of a detection algorithm so as to identify a target more accurately and efficiently. However, in practical application scenarios, it is difficult for the visual target detection method to obtain information such as the accurate position of the target, and the visual target detection method is susceptible to interference factors affecting the visual image, such as illumination and weather.
Disclosure of Invention
The invention aims to solve the problems that: in the prior art, a single signal source is mainly used for detecting an environment sensing obstacle, so that the problems that the recognition accuracy is low, the target cannot be effectively monitored and the like due to the limitation of a sensor signal cannot be avoided, and the monitoring requirement of an actual complex road scene cannot be met.
The technical scheme of the invention is as follows: a road obstacle detection method based on millimeter wave radar and visual signals detects obstacles in front of a road through a millimeter wave radar and a visual sensor, detects the type of a target object by fusing two signals of the millimeter wave radar and the visual image, firstly preprocesses the millimeter wave radar signal, converts a millimeter wave radar message into a form of a target point position, and performs time synchronization with the visual image signal; mapping a millimeter wave radar target point on the visual image signal according to the coordinate matching relation to realize the fusion of the two modes; and performing target detection on the millimeter wave radar information mapped to the visual image signal, and finally performing scale estimation on a detection result to refine the obstacle information.
Further, analog detection is carried out on millimeter wave radar information mapped on the visual image signal, clutter is judged to be clutter when the millimeter wave radar information is lower than an analog threshold, next detection is not carried out, and target detection is carried out on structures higher than the analog threshold through a feature extraction network.
Further, the invention specifically comprises the following steps:
1) acquiring a message signal stream and a visual image signal stream of the millimeter wave radar sensor, and synchronously matching the two signal streams by adopting multi-thread processing according to a message timestamp;
2) taking millimeter wave radar signals and visual image signals at the same time, converting the synchronous millimeter wave radar signals to map the synchronous millimeter wave radar signals to the visual image signals to obtain a mapped point set S, wherein the mapped point set S comprises a group of pixel coordinate points on an image and is used for mapping a millimeter wave detection object target at the target position of the image;
3) respectively taking each point in the S from the point set S obtained in the step 2), and carrying out Edge box local similarity estimation on the position image;
4) according to the similarity estimation result of the Edge box, clutter filtering is carried out on the millimeter wave radar signal, a millimeter wave detection point with the similarity higher than a threshold value of the Edge box is reserved, a detection point with the similarity lower than the threshold value is regarded as an irrelevant clutter, and the part of clutter is filtered;
5) performing local feature extraction on the target position of the millimeter wave radar signal after filtering by adopting a convolutional neural network;
6) performing characteristic pyramid characteristic fusion on the multi-scale characteristics output in the step 5);
7) classifying and performing regression prediction on the fused features to obtain a detection result;
8) integrating detection results of mapping points of each millimeter wave radar and carrying out non-maximum value suppression;
9) and performing standard formatting on information such as the prediction type, the confidence coefficient, the position result and the like, outputting the information in a descending order according to the confidence coefficient, and displaying the obstacle detection result.
The invention also provides a road obstacle detection device based on the millimeter wave radar and the visual signal, wherein the device is provided with a computer readable storage medium, and a computer program is configured in the computer readable storage medium, and when the computer program is executed, the road obstacle detection method is realized.
The method accurately and efficiently detects the road obstacle target by using the millimeter wave radar sensor information and the vision sensor information, and comprises a heterogeneous signal synchronous matching technology, a millimeter wave radar signal mapping technology, an analog estimation technology, an image feature extraction technology, a feature pyramid feature fusion technology, an image feature target detection technology and a scale estimation technology.
The invention has the beneficial effects that: an efficient and accurate obstacle detection and identification technology is provided. Compared with the prior art, the method has the following advantages.
(1) The invention contains the computer vision target detection function based on deep learning, extracts the semantic information of sensor data and effectively improves the accuracy and efficiency of target detection.
(2) The invention also combines the signals of the millimeter wave radar sensor, improves the stability of the system when the visual image effect is not good in abnormal environments such as rainy and snowy weather, and more comprehensively covers the long-distance targets which are easy to ignore in the visual detection.
(3) The structure of the invention can be expanded to other types of detection, such as road driving detection, personnel flow detection and the like, and has extremely strong generalization capability and wide market application prospect.
Drawings
FIG. 1 is an overall flow chart of the present invention.
Fig. 2 is a flowchart of millimeter wave radar clutter filtering.
Detailed Description
The invention provides a road obstacle detection method based on a millimeter wave radar and a visual signal, which combines two modal signals of the millimeter wave radar and the visual signal and is used for identifying a road obstacle. The general flow diagram of the process of the present invention is shown in FIG. 1. The technical scheme of the invention specifically comprises the following steps:
(1) and inputting a message signal stream and a visual image signal stream of the millimeter wave radar sensor, and synchronously matching two inputs according to message timestamps of the two different input streams.
Respectively establishing a millimeter wave radar message reading thread and a visual image reading thread, storing a video frame read by the visual image reading thread in a data structure dictionary by taking a timestamp as a key, simultaneously reading a latest millimeter wave radar message by the millimeter wave radar message reading thread, using the read millimeter wave radar message timestamp to index the visual image dictionary, matching the group of messages with the visual image for subsequent processing if the indexing is successful, and comparing the timestamp with the earliest and latest timestamps in the dictionary if the indexing is not successful. If the millimeter wave radar message timestamp is earlier than the earliest visual data, enabling the visual image reading thread to sleep until the millimeter wave radar message is read to the same time as the image data; similarly, if the millimeter wave radar message is later than the latest visual data, the millimeter wave radar message reading thread is made to sleep until the visual image data is read to the same moment.
The image reading thread can store the visual signals for at most 2 minutes, and the oldest data can be cleared in time sequence along with the continuous reading of new data. The reason for setting to 2 minutes is that since the multithreading execution has uncertainty, a buffer window is established, and the problem that data cannot be matched due to thread scheduling problems is avoided.
(2) A set of millimeter wave radar signals and visual image signals are taken. The millimeter wave radar signal is a millimeter wave radar detection result at a certain moment, and the visual image signal is a monitoring picture of the vehicle-mounted image acquisition device at the same moment. And converting the synchronous millimeter wave radar signal to map the synchronous millimeter wave radar signal to a visual image signal to obtain a mapped point set S, wherein the mapped point set S comprises a group of pixel coordinate points on the image.
The mapping of the millimeter wave radar signal and the visual image signal is essentially the conversion of a real coordinate system and a pixel coordinate system, and the real coordinate system needs to be converted into the image coordinate system first, and then the image coordinate system needs to be converted into the pixel coordinate system. The pixel coordinate system takes the upper left corner of the image of the visual image signal as an origin, the u axis is rightward, and the v axis is downward; the image coordinate system refers to the x-axis to the right and the y-axis to the down, with the center of the image as the origin. The conversion relation between the pixel coordinate system and the image coordinate system is as follows:
wherein u is 0 ,v 0 The coordinates of the origin of the image coordinate system in the pixel coordinate system, u, v the coordinates of the target point in the pixel coordinate system, and x, y the coordinates of the target point in the image coordinate system. d x Represents the actual physical size of each pixel along the x-axis; d is a radical of y Representing the actual physical size of each pixel along the y-axis.
The above formula is expressed in homogeneous coordinates as:
the correspondence of the image coordinate system and the real coordinate system has the following conversion relationship:
wherein f is the focal length of the camera, X c ,Y c ,Z c The x, y and z axis coordinates of the target position in the real coordinate system are shown.
In combination with the above equation, the conversion relationship between the pixel coordinate system and the real coordinate system is as follows:
wherein the content of the first and second substances,
is the internal reference matrix of the camera, determined by the camera itself, with M C Then the above formula can be expressed as:
therefore, the coordinate transformation formula of the real coordinate system and the pixel coordinate system can be obtained. Signals provided by the millimeter wave radar sensor are real coordinates of a target, and millimeter wave radar detection signals are converted into pixel coordinate mapping and visual image images according to the formula.
(3) And (3) mapping the millimeter wave radar detection result obtained in the step (2) to a point set S, respectively taking each point in the S, and carrying out Edge box local similarity estimation on the position image.
The Edge box method is to calculate the number of contours contained in a frame and overlapped with the frame Edge based on contour information generated by the target Edge in an image, if the more contour information contained in the target frame, the higher the possibility that the frame contains the target is, and then to rank all potential target frames to realize rough estimation of the target area. In the existing similarity estimation method, Edge Boxes can well detect objects in various shapes, have certain scale adaptivity and have high operation speed.
And performing clutter filtering on the millimeter wave radar signal according to the similarity estimation result of the Edge box. And keeping millimeter wave radar detection points with higher Edge box similarity, and regarding the detection points with lower similarity as irrelevant clutter, and filtering the part of clutter.
(4) And local feature extraction is carried out on the target position of the millimeter wave radar signal after filtering.
And performing local feature extraction, performing inverse push according to the size of the target to obtain the pixel size in the visual signal, and intercepting the partial local image signal to perform feature extraction.
The pixel scale estimation of the obstacle is jointly obtained according to the internal and external parameters of the camera, the target distance and the direction measured by the millimeter wave radar and the actual size of an expected target. Firstly, each pixel value of the visual image picture is obtained by the illumination intensity of the corresponding position on the camera visual sensor, and the width and the height of the visual sensor are respectively W s ,H s The width and height of the pixel of the target in the picture are w p ,h p Resolution of picture W p ,H p . Then there is
Wherein w s ,h s The width and height of the object imaged on the vision sensor. Then, the distance and the direction of the target measured by the millimeter wave radar can obtain the through lens of the camera from the targetThe vertical distance d between a point and the perpendicular line of the optical axis can be obtained from the focal length f according to the perspective principle
Wherein w t ,h t The actual width and height of the target. According to sampling statistics, the length and the width of the target to be detected are intensively distributed within 2 meters. According to the above formula t ＝2,h t The pixel scale of the target on the visual signal can be found by the distance detected by the millimeter wave radar as 2. Centered at the detection point, w s ,h s And intercepting the local image for the length and the width, and inputting the local image into a subsequent target detection neural network.
The feature extraction adopts CSP-Darknet, and the network mainly comprises a CBL module and a CSP module. The CBL module is the most commonly used convolution layer plus batch normalization layer plus activation function layer in the convolutional neural network structure, where the activation function uses leakage Relu. The network input is a local image of a region where a millimeter wave radar signal target position is located, and feature output of different scales is intercepted at different depths of the network through calculation of a plurality of convolution layers.
(5) And carrying out feature pyramid feature fusion on the three features output in the step.
The feature fusion is to fuse the feature outputs of different depths of the convolutional neural network, so that shallow features are fused into more deep semantic information, and deep features are fused into more shallow position information. The multi-scale feature fusion part adopts a feature pyramid structure, and divides the features extracted at the previous stage into 3 different scales through bilinear interpolation up-sampling and convolution neural network down-sampling. For the features with larger scale, the feature centers are divided finely for predicting smaller objects in the image, while for the features with small scale, the feature centers are sparse and the receptive field is large enough for predicting larger objects in the image. The side connection between the characteristic pyramid structures also ensures the fusion of deep semantic information and shallow position information, and improves the accuracy of the subsequent prediction branches on target classification prediction and position regression. In addition, a bottom-up feature pyramid is added behind the top-down feature pyramid. Which contains two PAN structures. The PAN structure uses convolutional layers for subsampling. By combining the operations, the feature pyramid transmits strong semantic features from top to bottom, the PAN transmits strong positioning features from bottom to top, and parameter aggregation is performed on different detection layers from different backbone layers.
(6) And classifying and performing regression prediction on the features to obtain a detection result.
And performing convolution dimensionality reduction on each pixel position of the multiple scale features after feature fusion, further abstracting the features, and respectively transmitting the features into a classification prediction branch and a detection frame position regression branch. The classification branch is responsible for predicting the possible object classes of the pixel position, in the invention, the confidence coefficient of each class is calculated according to a classification network trained in advance, and the class with the highest position confidence coefficient is output. Identifiable categories include a large number of object categories that may appear on the road, such as pedestrians, bicycles, cars, card pools, street lamps, dogs, light boards, and the like. The detection frame position regression prediction detection frame can calculate the pixel coordinates of the detection frame according to the length, the width and the deviation of the pixel position and the prediction result by combining the characteristic scale and the pixel position information.
(7) And integrating detection results of mapping points of the millimeter wave radars and carrying out non-maximum suppression.
The detection frame coordinates obtained in step 7 are based on the local image of each millimeter wave radar mapping point. Therefore, after the detection results of all the mapping points are obtained, the coordinates of the detection frame need to be converted into the pixel coordinates in the original image of the visual image. The conversion mode is that the x and y coordinates of the local pixel coordinate are added with the x and y pixel coordinates of the upper left corner of the local image in the original image respectively. After the conversion is completed, a series of detection frame pixel coordinates in the original image are obtained, wherein a large number of repeated detections are included. To remove duplicate detection, the results need to be non-maxima suppressed. The specific method is that sorting is carried out according to the classification confidence degree in a descending order, and if the coincidence proportion of a certain detection frame and another detection frame which is sorted before the certain detection frame is higher than a fixed threshold value, the certain detection frame and the other detection frame are removed from the detection result.
(8) And performing standard formatting on the information such as the prediction type, the confidence coefficient, the position, the scale estimation result and the like, and outputting in a descending order according to the confidence coefficient.
The method is realized by a computer program, and provides a road obstacle detection device based on a millimeter wave radar and a visual image signal. The signal preprocessing module is used for preprocessing millimeter wave radar signals, converting millimeter wave radar messages into a target point position form and carrying out time synchronization with visual signals; the mode mapping module maps the millimeter wave radar target point on the visual signal to realize the fusion of two modes; the clutter filtering module is used for judging similarity and filtering clutter generated by environmental noise in the millimeter wave radar signal; the multi-mode detection module is used for carrying out target detection on the millimeter wave radar information mapped to the image information acquired by the vision sensor; and the scale estimation module carries out scale estimation on the detection result and refines the obstacle information.
Claims (7)
1. A road obstacle detection method based on millimeter wave radar and visual signals is characterized in that a road front obstacle is detected through a millimeter wave radar and a visual sensor, the type of a target object is detected by fusing two signals of the millimeter wave radar and the visual image, firstly, the millimeter wave radar signal is preprocessed, a millimeter wave radar message is converted into a target point position form, and time synchronization is carried out on the millimeter wave radar message and the visual image signal; mapping a millimeter wave radar target point on the visual image signal according to the coordinate matching relation to realize the fusion of the two modes; and performing target detection on the millimeter wave radar information mapped to the visual image signal, and finally performing scale estimation on a detection result to refine the obstacle information.
2. The method as claimed in claim 1, wherein the millimeter wave radar and visual signal based road obstacle detection method comprises performing analog detection on the millimeter wave radar information mapped on the visual image signal, determining the millimeter wave radar information as clutter if the millimeter wave radar information is lower than an analog threshold, and performing no further detection, and performing target detection on the millimeter wave radar information if the millimeter wave radar information is higher than the analog threshold through the feature extraction network.
3. The method for detecting the road obstacle based on the millimeter wave radar and the visual signal as claimed in claim 2, which comprises the following steps:
1) acquiring a message signal stream and a visual image signal stream of the millimeter wave radar sensor, and synchronously matching the two signal streams by adopting multi-thread processing according to a message timestamp;
2) taking millimeter wave radar signals and visual image signals at the same time, converting the synchronous millimeter wave radar signals to map the synchronous millimeter wave radar signals to the visual image signals to obtain a mapped point set S, wherein the mapped point set S comprises a group of pixel coordinate points on an image and is used for mapping a millimeter wave detection object target at the target position of the image;
3) respectively taking each point in the S of the point set S obtained in the step 2), and carrying out Edge box local similarity estimation on the position image;
4) according to the similarity estimation result of the Edge box, clutter filtering is carried out on the millimeter wave radar signal, a millimeter wave detection point with the similarity higher than a threshold value of the Edge box is reserved, a detection point with the similarity lower than the threshold value is regarded as an irrelevant clutter, and the part of clutter is filtered;
5) performing local feature extraction on the target position of the millimeter wave radar signal after filtering by adopting a convolutional neural network;
6) performing characteristic pyramid characteristic fusion on the multi-scale characteristics output in the step 5);
7) classifying and performing regression prediction on the fused features to obtain a detection result;
8) integrating detection results of mapping points of each millimeter wave radar and carrying out non-maximum value suppression;
9) and performing standard formatting on information such as the prediction type, the confidence coefficient, the position result and the like, outputting the information in a descending order according to the confidence coefficient, and displaying the obstacle detection result.
4. The method for detecting the road obstacle based on the millimeter wave radar and the visual signal as claimed in claim 1 or 2, wherein when the millimeter wave radar signal and the visual image signal are time-synchronized, the method adopts multi-thread heterogeneous data synchronization matching, specifically: respectively establishing a millimeter wave radar message reading thread and a visual image reading thread, storing a video frame read by the visual image reading thread in a data structure dictionary by taking a timestamp as a key, simultaneously reading a latest millimeter wave radar message by the millimeter wave radar message reading thread, using the read millimeter wave radar message timestamp to index the visual image dictionary, matching the group of messages with the visual image for subsequent processing if the indexing is successful, and comparing the timestamp with the earliest and latest timestamps in the dictionary if the indexing is not successful. If the millimeter wave radar message timestamp is earlier than the earliest visual data, enabling the visual image reading thread to sleep until the millimeter wave radar message is read to the same time as the image data; similarly, if the millimeter wave radar message is later than the latest visual data, the millimeter wave radar message reading thread is made to sleep until the visual image data is read to the same moment.
5. A method for detecting a road obstacle based on a millimeter wave radar and a visual signal as claimed in claim 1 or 2, wherein the mapping of the millimeter wave radar target point to the visual image signal is specifically:
wherein X c ,Y c ,Z c The coordinate of the target point of the millimeter wave radar in the x, y and z axes of a real coordinate system, M C The reference matrix of the vision sensor is determined by the properties of the vision sensor, and u and v are coordinates of the target point in a pixel coordinate system.
6. The method for detecting the road obstacle based on the millimeter wave radar and the visual signal as claimed in claim 1 or 2, wherein the target detection is performed by adopting self-adaptive local visual signal extraction, and the specific method is as follows:
wherein w t ,h t Is the actual width and height of the target, w s ,h s The width and height of an image of a target on a visual sensor are set as d is the vertical distance between the target and one point of a lens of the visual sensor and a perpendicular line of an optical axis, f is the focal length of the visual sensor, the corresponding relation between the dimension in the image and the length and width of the real target is obtained according to the formula, the length and width of the detected target are intensively distributed within n meters, and then w is set as t ＝n,h t N, the pixel scale of the target on the visual signal is obtained by the distance detected by the millimeter wave radar, w is the center of the detected target point s ,h s And intercepting the local image for the length and the width, and inputting the local image into a subsequent target detection neural network for target detection.
7. A road obstacle detection apparatus based on millimeter-wave radar and visual signals, characterized in that a computer-readable storage medium is provided in the apparatus, a computer program is configured in the computer-readable storage medium, and the computer program, when executed, implements the road obstacle detection method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210493558.2A CN114818819A (en) | 2022-05-07 | 2022-05-07 | Road obstacle detection method based on millimeter wave radar and visual signal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210493558.2A CN114818819A (en) | 2022-05-07 | 2022-05-07 | Road obstacle detection method based on millimeter wave radar and visual signal |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114818819A true CN114818819A (en) | 2022-07-29 |
Family
ID=82510809
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210493558.2A Pending CN114818819A (en) | 2022-05-07 | 2022-05-07 | Road obstacle detection method based on millimeter wave radar and visual signal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114818819A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116148801A (en) * | 2023-04-18 | 2023-05-23 | 深圳市佰誉达科技有限公司 | Millimeter wave radar-based target detection method and system |
| KR102617591B1 (en) * | 2023-07-10 | 2023-12-27 | 메타빌드 주식회사 | Smart object information matching system and method using artificial intelligence based image sensor and radar sensor |
-
2022
- 2022-05-07 CN CN202210493558.2A patent/CN114818819A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116148801A (en) * | 2023-04-18 | 2023-05-23 | 深圳市佰誉达科技有限公司 | Millimeter wave radar-based target detection method and system |
| KR102617591B1 (en) * | 2023-07-10 | 2023-12-27 | 메타빌드 주식회사 | Smart object information matching system and method using artificial intelligence based image sensor and radar sensor |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107609522B (en) | Information fusion vehicle detection system based on laser radar and machine vision | |
| CN110942449B (en) | Vehicle detection method based on laser and vision fusion | |
| CN102765365B (en) | Pedestrian detection method based on machine vision and pedestrian anti-collision warning system based on machine vision | |
| CN111369541B (en) | Vehicle detection method for intelligent automobile under severe weather condition | |
| CN102867417B (en) | Taxi anti-forgery system and taxi anti-forgery method | |
| CN112396650A (en) | Target ranging system and method based on fusion of image and laser radar | |
| KR102195164B1 (en) | System and method for multiple object detection using multi-LiDAR | |
| CN114818819A (en) | Road obstacle detection method based on millimeter wave radar and visual signal | |
| Chiu et al. | Automatic Traffic Surveillance System for Vision-Based Vehicle Recognition and Tracking. | |
| CN114359181B (en) | Intelligent traffic target fusion detection method and system based on image and point cloud | |
| CN112487919A (en) | 3D target detection and tracking method based on camera and laser radar | |
| CN115943439A (en) | Multi-target vehicle detection and re-identification method based on radar vision fusion | |
| Fu et al. | A camera–radar fusion method based on edge computing | |
| CN112951000A (en) | Large-scale vehicle blind area bidirectional early warning system | |
| CN113850102B (en) | Vehicle-mounted vision detection method and system based on millimeter wave radar assistance | |
| CN103034843A (en) | Method for detecting vehicle at night based on monocular vision | |
| CN114898296A (en) | Bus lane occupation detection method based on millimeter wave radar and vision fusion | |
| CN114495064A (en) | Monocular depth estimation-based vehicle surrounding obstacle early warning method | |
| CN113568002A (en) | Rail transit active obstacle detection device based on laser and image data fusion | |
| CN112699267B (en) | Vehicle type recognition method | |
| Jiang et al. | Target detection algorithm based on MMW radar and camera fusion | |
| Bi et al. | A new method of target detection based on autonomous radar and camera data fusion | |
| CN115876198A (en) | Target detection and early warning method, device, system and medium based on data fusion | |
| CN115690713A (en) | Binocular camera-based radar-vision fusion event detection method | |
| CN116699602A (en) | Target detection system and method based on millimeter wave radar and camera fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |