CN114002669A - Road target detection system based on radar and video fusion perception - Google Patents
Road target detection system based on radar and video fusion perception Download PDFInfo
- Publication number
- CN114002669A CN114002669A CN202111230013.4A CN202111230013A CN114002669A CN 114002669 A CN114002669 A CN 114002669A CN 202111230013 A CN202111230013 A CN 202111230013A CN 114002669 A CN114002669 A CN 114002669A
- Authority
- CN
- China
- Prior art keywords
- road
- target
- camera
- control unit
- target detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
- G01S13/867—Combination of radar systems with cameras
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/017—Detecting movement of traffic to be counted or controlled identifying vehicles
- G08G1/0175—Detecting movement of traffic to be counted or controlled identifying vehicles by photographing vehicles, e.g. when violating traffic rules
Landscapes
- Engineering & Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention provides a road target detection system based on radar and video fusion perception, which comprises a radar sensor, a target detection camera, an intelligent camera, a calculation control unit, a time service unit and a communication module, wherein the radar sensor, the target detection camera, the intelligent camera, the calculation control unit, the time service unit and the communication module are respectively connected with the calculation control unit, the radar sensor is used for collecting road target data, the target detection camera and the intelligent camera are respectively used for collecting road target data, and the calculation control unit is used for receiving and processing the collected data transmitted by the radar sensor, the target detection camera and the intelligent camera. The invention adopts an integrated design, has the advantages of compact structure, convenient installation, convenient maintenance and the like, and can realize bidirectional coverage through the configuration of the front camera and the rear camera.
Description
Technical Field
The application relates to the technical field of road target detection, in particular to a road target detection system based on radar and video fusion perception.
Background
The intelligent traffic equipment comprises equipment and systems such as signal lamps, signal machines, bayonet cameras, panoramic video monitoring, roadside perception systems, driving guidance systems, road environment monitoring equipment, road lighting systems and intelligent bus posters in urban roads, expressways and rural villages. Wherein, on road pole, intelligent equipment such as bayonet socket camera, traffic radar, flash light, panorama ball machine have mainly been erect, and ground induction coil or earth magnetism car detector on cooperation ground to and functions such as bayonet socket flow statistics, each equipment structure is independent, and the part is through communication interface interconnection. Along with the development of intellectuality, the intelligent equipment of erectting is more and more, often needs to occupy a large amount of spaces on the road pole.
How to realize the system structure integrated design and the related information interconnection and intercommunication of the road intelligent equipment and realize the redundant data processing at the front end is the development requirement of the road intelligent equipment. On one hand, the installation is cheap, the maintenance is facilitated, the space is saved, and the management is convenient; on the other hand, the processing of redundant data is realized at the perception front end, so that the information transmission bandwidth can be effectively reduced, and the communication resources are saved.
Disclosure of Invention
In view of this, the present invention provides a road target detection system based on radar and video fusion sensing, so as to solve the technical problems mentioned in the above background art.
The technical scheme adopted by the invention for solving the technical problems is as follows:
a road target detection system based on radar and video fusion perception comprises a radar sensor, a target detection camera, an intelligent camera, a calculation control unit, a time service unit and a communication module, wherein the radar sensor, the target detection camera, the intelligent camera, the calculation control unit, the time service unit and the communication module are respectively connected with the calculation control unit, the radar sensor is used for collecting road target data, the target detection camera and the intelligent camera are respectively used for collecting road target data, and the calculation control unit is used for receiving and processing the collected data transmitted by the radar sensor, the target detection camera and the intelligent camera.
Optionally, the smart cameras are respectively located at the front side and the rear side of the system and used for bidirectional coverage of a road target, and the radar sensor is connected with the calculation control unit through an ethernet interface or a CAN interface.
Optionally, the radar sensor is configured to acquire speed, acceleration, distance, angle, heading angle, and type information of the road target.
Optionally, the object detection camera and the smart camera are used for acquiring image data of a road object.
Optionally, the calculation control unit is configured to obtain the position information of the road target through a target detection algorithm by using the image data of the road target.
Optionally, the obtaining of the position information of the road target by using the target detection algorithm includes:
obtaining the category and the characteristics of the road target and the pixel position information of the road target in the image data of the road target through a target detection algorithm;
and obtaining the category and the characteristic of the road target and the position information under the unified coordinate system through the coordinate conversion matrix of the target detection camera, the intelligent camera and the unified coordinate system.
Optionally, the calculation control unit obtains a speed and a heading angle measured based on the image data of the road target by using an inter-frame relationship.
Optionally, the calculation control unit performs correlation calculation on the speed, the acceleration, the distance, the angle, the course angle, and the type information of the road target acquired by the radar sensor and the image data of the road target respectively acquired by the target detection camera and the intelligent camera through a data fusion algorithm to obtain an identity number of the road target after fusion, and superimposes the image data of the road target on the identity number of the target after fusion to finally obtain the fusion information of the road target.
Optionally, the communication module is connected to an external device through a network port, and the communication module uploads the fusion information of the road target to a communication information management center or broadcasts the fusion information to vehicles with receiving capability in an area through the external device.
Optionally, the system further includes a light supplement unit, where the light supplement unit is connected to the calculation control unit, and the light supplement unit is configured to supplement light to a monitored road area; the time service unit is used for unifying clocks in the system and ensuring time synchronization of the system and an external clock source.
Compared with the prior art, the invention has the following beneficial effects:
(1) the system structure is characterized in that the integrated equipment comprises the detection and identification functions of intelligent equipment such as a bayonet camera, an electric alarm camera, a vehicle-road cooperative target detection camera and the like in the traditional sense, and has the advantages of compact structure, convenience in installation, convenience in maintenance and the like;
(2) and integrating the radar sensor and the camera according to the difference of the data of the road target detected by the radar sensor and the camera, and realizing the full-range coverage of the road target. Meanwhile, bidirectional coverage is realized through the configuration of the front camera and the rear camera.
(3) For the intelligent camera which is arranged on the monitoring road pole in a two-way mode, due to the fact that no overlapping area exists, the accuracy of target matching is not high under general conditions, the detection range is reduced, the image definition is improved, the license plate of a main detection target can be effectively identified, the accuracy of target matching is improved, and the target identification number in the full coverage range is unique.
Drawings
FIG. 1 is a schematic diagram of a road target detection system architecture based on radar and video fusion perception;
fig. 2 is a cross-sectional view of the detection range of the radar and camera.
Detailed Description
The present application will be further described with reference to the accompanying drawings and examples, it being understood that the examples described are only a part of the examples and are not intended to limit the scope of the invention. All other embodiments available to those of ordinary skill in the art are intended to be within the scope of the embodiments of the present application.
Fig. 1 is a schematic diagram of a road target detection system based on radar and video fusion perception according to an embodiment of the present application, as shown in fig. 1. The system comprises a radar sensor, a target detection camera, an intelligent camera, a calculation control unit, a time service unit and a communication module, wherein the radar sensor, the target detection camera, the intelligent camera, the calculation control unit, the time service unit and the communication module are respectively connected with the calculation control unit, the radar sensor is used for collecting road target data, the target detection camera and the intelligent camera are respectively used for collecting road target data, and the calculation control unit is used for receiving and processing collected data transmitted by the radar sensor, the target detection camera and the intelligent camera.
In the embodiment, the system adopts an integrated design, not only comprises the detection and identification functions of intelligent equipment such as a bayonet camera, an electric police camera, a vehicle-road cooperative target detection camera and the like in the traditional sense, but also has the advantages of compact structure, convenience in installation, convenience in maintenance and the like;
in this embodiment, the target detection camera may be connected to the calculation control unit through an MIPI/USB3.0 interface, and the smart camera may be connected to the calculation control unit through an MIPI/USB3.0 interface.
The intelligent cameras are respectively positioned on the front side and the rear side of the system and used for bidirectional coverage of road targets, and the radar sensor is connected with the calculation control unit through an Ethernet interface or a CAN interface.
The radar sensor is used for acquiring speed, acceleration, distance, angle, course angle and type information of a road target.
The target detection camera and the intelligent camera are used for acquiring image data of the road target.
In the embodiment, the radar sensor can be a millimeter wave radar, and is currently 80GHz, the detection band is not limited, and the radar sensor is mainly used for multi-target detection; the detection direction of the intelligent camera 1 faces backwards and horizontally downwards (45 degrees), and the intelligent camera can identify the license plate, the speed, the position, the vehicle appearance data, the vehicle cab environment and the cab face identification; the intelligent camera 2 can be used for identifying the license plate, the speed, the position, the vehicle appearance data, the road traffic light and the traffic incident identification when the direction is forwards and horizontally downwards (10 degrees); the direction of the object detection camera 1 is forward and horizontally downward (5 degrees), and the object detection camera can identify the speed, the position and the appearance data of the vehicle and the category, the speed and the position information of moving objects such as pedestrians; the direction of the object detection cameras 2-N is forward and horizontally downward (2 degrees), and the object detection cameras can identify the speed, the position, the appearance of the vehicle, and the type, the speed and the position information of moving objects such as pedestrians; generally, the radar coverage is large, the effective detection range of the camera is small, and the system achieves basic consistency with the radar coverage by utilizing a plurality of cameras. Meanwhile, the intelligent cameras are arranged in the front and the back of the road pole, so that the front and the back of the road pole can be covered in a two-way mode.
And each camera is responsible for acquiring the image information of the road target to obtain two-dimensional image information. And the computing unit is connected with the computing unit through a high-speed data interface (such as USB3.0, MIPI and the like).
The calculation control unit is used for obtaining the position information of the road target by using the image data of the road target through a target detection algorithm.
The obtaining of the position information of the road target through the target detection algorithm includes:
obtaining the category and the characteristics of the road target and the pixel position information of the road target in the image data of the road target through a target detection algorithm;
and obtaining the category and the characteristic of the road target and the position information under the unified coordinate system through the coordinate conversion matrix of the target detection camera, the intelligent camera and the unified coordinate system.
The calculation control unit obtains the speed and the course angle measured based on the image data of the road target by utilizing the inter-frame relation.
The calculation control unit calculates the relevance of the road target according to the speed, the acceleration, the distance, the angle, the course angle and the type information of the road target acquired by the radar sensor and the image data of the road target acquired by the target detection camera and the intelligent camera respectively through a data fusion algorithm to obtain the identification number of the road target after fusion, and the image data of the road target is superposed to the identification number of the road target after fusion to finally obtain the fusion information of the road target.
In the embodiment, in the multi-camera information fusion process, the target fusion effect is better under the condition that the correlation algorithm has an overlapping area. The two intelligent cameras do not sense an overlapping area, and the target fusion effect is poor. For the target (motor vehicle) mainly responsible for monitoring by the system, clear images can be obtained by reducing the detection range, so that the license plate information of the target vehicle is obtained, the association of the two camera detection targets is realized, and the target ID unification can be well realized.
In the present embodiment, the calculation control means obtains the type (vehicle, pedestrian, obstacle, etc.), the feature (color, size, vehicle brand, etc.) and the pixel position information in the image of the object by the object detection algorithm using the state information of the road object and the image data of the road object, and obtains the type, feature and position information in the unified coordinate system by the coordinate conversion matrix of each camera and the unified coordinate system (world coordinate system WGS 84). Using the inter-frame relationship, the velocity (v) measured based on the image is obtainedx，vy) Course angle (β °). The target detection algorithm can be Yolov4, YOLOP, modified Yolov4 and the like.
I.e. the camera can measure: position (x, y), velocity (v)x，vy) Heading angle (beta), category, size, and information on license plate, color, vehicle brand, etc., which radar does not have.
And then, calculating the target relevance by using output items obtained by radar and image acquisition through a data fusion algorithm, such as Hungarian matching, nearest neighbor algorithm and the like, obtaining a unique ID of the fused target, and superposing all information on the target ID to obtain target information richer than that of a single sensor.
The communication module is connected with external equipment through a network port, and uploads the fusion information of the road target to a communication information management center or broadcasts the fusion information to vehicles with receiving capability in the region through the external equipment.
In this embodiment, the communication module of the road target detection system is configured to communicate target detection information and traffic statistics information with an external device, that is, upload data to a traffic information management center for traffic fine management by using a private traffic network or a public traffic network of a private route in a network access or light manner. In addition, the communication module is connected with an RSU (road side Unit) through a network port and completes data exchange, and target detection information and traffic statistical information acquired by the broadcasting system are broadcasted. The RSU adopts a communication protocol of LTE-V2X or 5G-V2X, is connected with a vehicle provided with an OBU (on Board Unit) and other entities with receiving capability, and completes data transmission and reception.
The system also comprises a light supplementing unit, wherein the light supplementing unit is connected with the calculation control unit and is used for supplementing light to the monitored road area; the time service unit is used for unifying clocks in the system and simultaneously ensuring the time synchronization of the system and an external clock source.
In this embodiment, generally, under a low-light condition, it is difficult to acquire a qualified image for various detection functions in each camera monitoring area due to insufficient illumination. The normal facula of the flash lamp is controlled to be basically consistent with the focusing direction of the camera by adjusting the installation angle of each flash lamp, and then a manual or automatic light supplementing mode is set through the calculation control unit. For example, a manual mode is adopted, for example, setting 18:30 at night to 6:30 in the morning of the next day, and turning on a light supplement lamp; the automatic mode, for example, when the brightness is less than or equal to 38Lux at night and less than or equal to 45Lux at day, the flash lamp is started.
The time service unit is used for unifying internal clocks of the system and ensuring that the system and an external clock source achieve time synchronization (for example, synchronization with a GPS clock).
In another embodiment of the present application, as shown in fig. 2, in this embodiment, monitoring ranges of the sensors are respectively marked by different line segments, the smart camera 1 covers the rear of the device, the smart camera 2 covers a short distance, the target detection camera 1 covers a medium distance, the target detection camera 2 covers a long distance, and the radar covers a medium-short distance. Specifically, assume that the objects (vehicles, pedestrians, and other moving objects) move from left to right in fig. 2:
the intelligent camera 1 adopts a short-focus lens (6-12mm), the pitch angle of the camera is horizontally deviated from 20-45 degrees, the monitoring range is small, clear images of targets can be shot, meanwhile, relatively accurate target position information can be obtained in an image ranging mode under the condition of short distance, and further, target speed information can be obtained through the interframe relation.
The intelligent camera 2 adopts a medium and short focal length lens (8-25mm), the camera pitch angle is about 10 degrees horizontally, the detection range is small, clear images of the target can be shot, and the intelligent camera 2 can also identify the speed, position and structural information of the target similar to the intelligent camera 1. The structured data refers to information such as vehicle number plate, color of the plate, type of the plate, color of the body, brand of the vehicle, annual payment of the vehicle and the like.
Similarly, the target detection camera 1 adopts a middle focal length lens of 12-25mm, and the pitching angle of the camera is about 5 degrees downwards horizontally; the target detection camera 2 adopts a medium-long focal length lens of 16-35 mm; the target detection camera n adopts a long-focus lens to cover the distance. Meanwhile, the radar monitoring range includes all or a part of the object detection cameras 1, 2, n of the smart camera 2.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (10)
1. A road target detection system based on radar and video fusion perception is characterized by comprising a radar sensor, a target detection camera, an intelligent camera, a calculation control unit, a time service unit and a communication module, wherein the radar sensor, the target detection camera, the intelligent camera, the calculation control unit, the time service unit and the communication module are respectively connected with the calculation control unit, the radar sensor is used for collecting road target data, the target detection camera and the intelligent camera are respectively used for collecting road target data, and the calculation control unit is used for receiving and processing the collected data transmitted by the radar sensor, the target detection camera and the intelligent camera.
2. The system of claim 1, wherein the smart cameras are respectively located at the front side and the rear side of the system and used for bidirectional coverage of road targets, and the radar sensor is connected with the calculation control unit through an Ethernet interface or a CAN interface.
3. The system of claim 1, wherein the radar sensor is configured to collect speed, acceleration, distance, angle, heading angle, type information of a road target.
4. The system of claim 3, wherein the object detection camera and the smart camera are configured to capture image data of a road object.
5. The system of claim 4, wherein the calculation control unit is configured to obtain the position information of the road object by an object detection algorithm using the image data of the road object.
6. The system of claim 5, wherein the obtaining the location information of the road object by the object detection algorithm comprises:
obtaining the category and the characteristics of the road target and the pixel position information of the road target in the image data of the road target through a target detection algorithm;
and obtaining the category and the characteristic of the road target and the position information under the unified coordinate system through the coordinate conversion matrix of the target detection camera, the intelligent camera and the unified coordinate system.
7. The system of claim 6, wherein the computational control unit uses an inter-frame relationship to derive the measured speed, heading angle based on the image data of the road object.
8. The system according to claim 7, wherein the calculation control unit performs correlation calculation of the road target on the speed, acceleration, distance, angle, course angle, and type information of the road target collected by the radar sensor and the image data of the road target respectively collected by the target detection camera and the smart camera through a data fusion algorithm to obtain the identity number of the road target after fusion, and superimposes the image data of the road target on the identity number of the target after fusion to finally obtain the fusion information of the road target.
9. The system of claim 1, wherein the communication module is connected to an external device through a network port, and the communication module uploads the fusion information of the road object to a communication information management center or broadcasts the fusion information to vehicles with receiving capability in an area through the external device.
10. The system according to claim 1, further comprising a light supplement unit, wherein the light supplement unit is connected to the calculation control unit, and is configured to supplement light to the monitored road area; the time service unit is used for unifying clocks in the system and ensuring time synchronization of the system and an external clock source.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111230013.4A CN114002669A (en) | 2021-10-21 | 2021-10-21 | Road target detection system based on radar and video fusion perception |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111230013.4A CN114002669A (en) | 2021-10-21 | 2021-10-21 | Road target detection system based on radar and video fusion perception |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114002669A true CN114002669A (en) | 2022-02-01 |
Family
ID=79923590
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111230013.4A Pending CN114002669A (en) | 2021-10-21 | 2021-10-21 | Road target detection system based on radar and video fusion perception |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114002669A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114937367A (en) * | 2022-05-20 | 2022-08-23 | 苏州天准科技股份有限公司 | Intelligent camera system for cooperative monitoring of vehicle and road and control method |
| CN114944066A (en) * | 2022-05-20 | 2022-08-26 | 苏州天准科技股份有限公司 | Intelligent camera system for vehicle and road cooperative monitoring |
| CN115775384A (en) * | 2023-02-13 | 2023-03-10 | 北京理工大学前沿技术研究院 | Road target sensing method and system based on radar and video fusion |
| CN116168508A (en) * | 2022-05-20 | 2023-05-26 | 海南大学 | Driving fatigue detection and early warning control method and device for man-machine co-driving |
-
2021
- 2021-10-21 CN CN202111230013.4A patent/CN114002669A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114937367A (en) * | 2022-05-20 | 2022-08-23 | 苏州天准科技股份有限公司 | Intelligent camera system for cooperative monitoring of vehicle and road and control method |
| CN114944066A (en) * | 2022-05-20 | 2022-08-26 | 苏州天准科技股份有限公司 | Intelligent camera system for vehicle and road cooperative monitoring |
| CN116168508A (en) * | 2022-05-20 | 2023-05-26 | 海南大学 | Driving fatigue detection and early warning control method and device for man-machine co-driving |
| CN115775384A (en) * | 2023-02-13 | 2023-03-10 | 北京理工大学前沿技术研究院 | Road target sensing method and system based on radar and video fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114002669A (en) | Road target detection system based on radar and video fusion perception | |
| CN108833833B (en) | Intelligent networking-oriented automobile scene image data sensing and cooperative processing system | |
| CN112509333A (en) | Roadside parking vehicle track identification method and system based on multi-sensor sensing | |
| CN112099040A (en) | Whole-course continuous track vehicle tracking system and method based on laser radar network | |
| CN101373517A (en) | Method and system for recognizing license plate | |
| CN113074714B (en) | Multi-state potential sensing sensor based on multi-data fusion and processing method thereof | |
| CN102426801A (en) | Automatic parking lot berth management method and system based on visual analysis | |
| CN202686194U (en) | Vehicular tripod head inspection device | |
| CN111369796A (en) | Roadside sensing system | |
| WO2019132526A1 (en) | System and method for providing overhead camera-based precision localization for intelligent vehicles | |
| Wang et al. | Ips300+: a challenging multi-modal data sets for intersection perception system | |
| CN102752502A (en) | Intelligent camera and intelligent camera shooting method | |
| CN216873319U (en) | Thunder look fuses road monitored control system | |
| CN102306274A (en) | Device for identifying parking space state and method | |
| CN201134166Y (en) | Number plate of vehicle recognition system | |
| Zimmer et al. | Tumtraf intersection dataset: All you need for urban 3d camera-lidar roadside perception | |
| Zimmer et al. | A9 intersection dataset: All you need for urban 3d camera-lidar roadside perception | |
| CN219574949U (en) | Intelligent traffic perception integrated device based on binocular depth fusion of thunder and sound | |
| CN212992427U (en) | Image acquisition module | |
| TW201809670A (en) | Travelling vehicle measuring system and related method thereof measuring the travelling speed of a plurality of traffic vehicles on a road provided with at least one lane | |
| CN217426263U (en) | Holographic road network road monitoring system | |
| CN105303825A (en) | Violating inclined side parking evidence obtaining device and method | |
| CN104916136A (en) | Intelligent electronic police system for bus-only lane | |
| CN113345118B (en) | Parking charge management method, system and storage medium | |
| CN112765423A (en) | City perception base station |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |