WO2020134512A1

WO2020134512A1 - Traffic detection system based on millimeter wave radar and video

Info

Publication number: WO2020134512A1
Application number: PCT/CN2019/113964
Authority: WO
Inventors: 陈焰中; 章庆; 陈俊德
Original assignee: 南京慧尔视智能科技有限公司
Priority date: 2018-12-29
Filing date: 2019-10-29
Publication date: 2020-07-02
Also published as: CN109615870A

Abstract

A traffic detection system based on a millimeter wave radar and a video. According to the system, target information is acquired by a millimeter wave radar sensor and a video sensor (step 1); world coordinates are converted into image plane coordinates, and the position of a target projection acquired and detected by the millimeter wave radar sensor on an image plane is determined (step 2); a target type is identified by means of image machine learning; the data acquired by the video sensor and the data by the millimeter wave radar are fused (step 3), and the speed, coordinates and type of a target are output (step 4). According to system, the data detected by the video sensor and data detected by the millimeter wave radar are subjected to real-time data fusion, and the fused target information is sent to a system platform, thereby reducing the calculation amount of the system, accelerating the response time of the system, and improving detection precision.

Description

A traffic detection system based on millimeter wave radar and video

Technical field

The invention belongs to the technical field of data collection and processing in intelligent transportation, and particularly relates to a traffic detection system based on millimeter wave radar and video.

Background technique

Most of the existing traffic detection systems use video sensors. The accuracy of a single video sensor is not high in detection effect, the delay of image processing is large, and its accuracy is easily affected by rain, fog and other environments. Later, other types of sensors were gradually added to the traffic detection system, such as millimeter wave radar, geomagnetism, lidar and other sensors. The advantages of different sensors were used to perform multi-sensor data fusion, which greatly improved the detection performance of the system. The radar sensor can detect the position and speed information of the target in real time. The data processing can reach 20 times per second. The environment is highly adaptable. It can work around the clock and around the clock. The disadvantage is that it cannot be visualized. The video sensor can process the image information. If the image is analyzed and calculated, it takes a long time to transcode, it cannot respond in real time, and the environment is not adaptable.

Taking advantage of these two sensors, radar and video can be used as the detection front-end for data fusion processing. However, because these two sensors independently collect data and the output data format is inconsistent, the system eventually combines multiple information of the same target collected by multiple sensors, which has a huge amount of processing on the original data and aggravates the system platform. Data processing work. Moreover, in the current system, the radar and the camera have different installation positions, so the information fusion needs to correlate the coordinate systems of the two sensors, and the coordinate relationship between the two sensors is related to their relative positions. If one of the sensors has a slight difference To move, the coordinates need to be re-associated. Since the two sensors are often shaken during use, the installation position will deviate after a period of time, resulting in unstable distances and angles. Coordinate matching needs to be performed periodically, so installation and debugging are difficult and often require maintenance.

Summary of the invention

The present invention provides a traffic detection system based on millimeter wave radar and video in response to the problems of the existing technical institutions that are complicated and the work efficiency is low.

Specifically, the present invention is implemented using the following technical solution: the traffic detection system includes a data collection unit, a data processing unit, a data storage unit, a data communication unit, and a system platform;

The data acquisition unit includes a millimeter-wave radar sensor and a video sensor. The millimeter-wave radar sensor collects the target's coordinate position, target speed, and target length in real time and sends it to the data processing unit; the video sensor collects high-definition image data in real time and sends it to the data processing unit ;

The data processing unit includes a microprocessor and an embedded neural network processor, fuse the data information from the millimeter wave radar and video sensor detected by the same target, and finally output the complete information of the same target to the system platform in a unified format;

The implementation of the traffic detection system includes the following steps:

Step 1: The millimeter-wave radar sensor collects target information. The target information includes the identifier, speed, coordinates, and length of the target. The video sensor collects image information of the road surface. The millimeter-wave radar sensor and the video sensor collect The original data is sent to the data processing unit;

Step 2: Convert world coordinates to image plane coordinates; the millimeter wave radar sensor collects the target coordinates, and then corrects the internal and external parameters of the camera of the video sensor to obtain the conversion matrix between the target coordinates and the image plane coordinates. The coordinates are converted into image plane coordinates, and the position of the target detected and collected by the millimeter wave radar sensor on the image plane is determined;

Step 3: A neural network framework is used to identify the target type in the image captured by the video sensor; the neural network framework algorithm is run in the embedded neural network processor, and the unstructured data of the road surface image is converted into the target type through image machine learning And color structured data;

Step 4: Use the coordinate conversion of Step 2 and the image recognition of Step 3 to correspond to the targets detected by the millimeter wave radar sensor in the image captured by the camera, and obtain all the parameter information of each target in the image Fusion of superimposed target parameters, the parameter information includes the speed, coordinates and target type of each target;

Step 5: Send the fused target parameters to the system platform.

Further, the conversion of the world coordinates into image plane coordinates in step two, that is, the conversion of the target point in the world coordinate system to the point in the image coordinate system, is implemented in two steps:

(1) Transform the coordinates of the target detected by the millimeter wave radar sensor in the world coordinate system (X _w , Y _w , Z _w ) into the camera coordinate system (X _c , Y _c , Z _c ), the transformation formula is:

In the formula,

Is the rotation matrix from the world coordinate system to the camera coordinate system;

which is

Among them, the rotation around the X axis is α, the rotation around the Y axis is β, and the rotation around the Z axis is θ;

Is the translation matrix from the world coordinate system to the camera coordinate system. Since the distance between the millimeter wave radar sensor and the video sensor in this device is very close, the length of the translation is equivalent to 0, that is

Available:

The available target is transformed from the coordinates in the world coordinate system (X _w , Y _w , Z _w ) to the camera coordinate system (X _c , Y _c , Z _c )

X _C = cos(β)cos(θ)X _W +cos(β)sin(θ)Y _W -sin(β)Z _W

Y _C = (-cos(α)sin(θ)+sin(α)sin(θ)cos(θ)) X _W +(cos(α)cos(θ)+sin(α)sin(β)sin( θ))Y _W +sin(α)cos(β)Z _W

Z _C = (sin(α)sin(θ)+cos(α)sin(β)cos(θ)) X _W +(-sin(α)cos(θ)+cos(α)sin(β)sin( θ))Y _W +cos(α)cos(β)Z _W

(2) Transform the detected target from the camera coordinate system to the image coordinate system to determine the projection position (X _S , Y _S ) of the target in the picture taken by the video sensor to realize the detection by the millimeter wave radar sensor Target positioning on the image:

ω is the horizontal angle of view of the target in the camera,

Is the vertical viewing angle of the target in the camera, v is the horizontal dimension of the image, and h is the vertical dimension of the image.

Further, in step three, the data after the video sensor recognizes the target type, calculates the traffic flow statistical data in the data processing unit, counts and tracks the detected target, and calculates each cycle within a specified area Statistics of traffic flow for each model.

Further, the data processing unit determines whether the target has speeding, retrograde, or illegal traffic events based on the target speed and coordinates detected by the millimeter wave radar sensor; if a traffic event occurs, the data processing unit sends an instruction to the video sensor Take photos, take video forensics, and transfer event text information, image data, and video data to the system platform.

The beneficial effects of the present invention are as follows: The traffic detection system based on millimeter wave radar and video of the present invention reduces the data calculation difficulty of the system and simplifies the system process. The original image data collected by the video sensor is transmitted to the data processing unit without video encoding and decoding operations, so that the data is not delayed, and can be real-time data fusion with the data detected by the millimeter wave radar. In the data processing unit, the preliminary fusion processing of the two data is carried out, and the fused target information is sent to the system platform, which reduces the calculation amount of the system, speeds up the response time of the system, and improves the detection accuracy.

BRIEF DESCRIPTION

FIG. 1 is a system structure diagram of the present invention.

Fig. 2 is a system flowchart of the present invention.

FIG. 3 is a schematic diagram of camera calibration of the video sensor in the present invention.

4 is a schematic diagram of the projection relationship of the target from the camera coordinate system to the image coordinate system in the present invention.

detailed description

The present invention will be described in further detail below with reference to embodiments and with reference to the accompanying drawings.

Example 1:

An embodiment of the present invention is a traffic detection system based on millimeter wave radar and video. Referring to FIGS. 1, 2, 3, and 4, a traffic detection system based on millimeter wave radar and video includes a data collection unit and data. Processing unit, data storage unit, data communication unit and system platform.

Data acquisition unit: The data acquisition unit includes a millimeter wave radar sensor and a video sensor. The millimeter wave radar sensor collects target information in real time. The detected data content includes the target's coordinate position, target speed, target length, etc., and sends the collected target information to the data processing unit. The video sensor collects high-definition image data in real time, and sends the collected image information to the data processing unit.

Data processing unit: The data processing unit includes an ARM microprocessor and an embedded neural network processor (NPU). The embedded neural network processor (NPU) is responsible for the processing of image data, identifying the type of target (pedestrian, car, truck, etc.), and the data of the target information collected by the millimeter wave radar (target coordinate position, target speed, The length of the target, etc.) is superimposed on the video image. The ARM microprocessor is responsible for statistics of all traffic flow information and traffic incident recognition. The data processing unit converts the coordinate of the same target to the target information received in real time, merges the data information from the millimeter wave radar and video sensor detected by the same target, and finally outputs the complete information of the same target to the system platform in a unified format .

Data storage unit: The data storage unit includes an embedded multimedia memory card (EMMC) and an SD card. Including embedded multimedia memory card (EMMC) is used to store operating system, SD card is used to store traffic flow database and traffic event information.

Data communication unit: The communication unit includes a network (ETH) interface and an RS485 interface. The network (ETH) interface is used to transmit traffic flow information, traffic event information, and video image information to the system platform. The RS485 interface is used for docking the camera at the bayonet and the traffic control signal at the intersection.

As shown in Figure 2, the implementation of a traffic detection system based on millimeter wave radar and video includes the following steps:

Step 1: The millimeter-wave radar sensor and the video sensor are used as environment-aware front-ends to collect target and road information, respectively collect target information and road image information, and send the raw data collected by the millimeter-wave radar sensor and the video sensor to the data processing unit . Among them, the information collected by the millimeter wave radar sensor includes the identifier (ID), speed, coordinates, length, etc. of the target, and the video sensor collects the image information of the road surface.

Step 2: Convert world coordinates to image plane coordinates. The millimeter wave radar sensor collects the target coordinates, and then corrects the internal and external parameters of the camera of the video sensor to obtain a conversion matrix (that is, a rotation matrix and a translation matrix) between the target coordinates and the image plane coordinates, and converts the road surface world coordinates into Image plane coordinates, so that the position of the target detected by the millimeter wave radar sensor collection and projection on the image plane can be determined.

The conversion process from the point of the target in the world coordinate system to the point in the image coordinate system is implemented in two steps:

In the formula,

Is the rotation matrix from the world coordinate system to the camera coordinate system.

which is

As shown in Figure 3, where the rotation around the X axis is α, the rotation around the Y axis is β, and the rotation around the Z axis is θ.

Available:

X _C = cos(β)cos(θ)X _W +cos(β)sin(θ)Y _W -sin(β)Z _W

(2) Transform the detected target from the camera coordinate system to the image coordinate system, determine the projection position (X _S , Y _S ) of the target in the picture taken by the video sensor, and realize the target detected by the millimeter wave radar sensor Positioning on the image.

As shown in Figure 4, ω is the horizontal viewing angle of the target in the camera

Step 3: Use neural network framework YOLO to identify the target type in the image captured by the video sensor. The video detector only needs to identify the category of the target, without other complicated calculations, reducing the delay of transcoding, and speeding up the response time of the system. The YOLO algorithm is run in a neural processing unit (NPU). The neural processing unit (NPU) uses a "data-driven parallel computing" architecture. Through image machine learning, the unstructured data of the road surface image is converted into the structure of the target type and color. data.

Step 4: Using the coordinate conversion of Step 2 and the image recognition of Step 3, the targets detected by the millimeter wave radar sensor can be matched with the targets in the image captured by the camera, and all the parameter information of each target can be obtained. The parameters of the target are superimposed in the parameter. The parameters include the speed, coordinates, and type of each target (including vehicle type, color, etc.). Fusion calculation of the target data detected by the two sensors increases the types of target parameters that can be provided and improves the detection accuracy of the system.

Step 5: Send the fused target parameters (speed, coordinates, type, etc.) to the system platform. Perform preliminary statistical processing on the raw data detected by the two sensors without the need to pass the raw data detected by the front-end collection unit to the system platform, which greatly reduces the amount of calculation in the system.

The data after identifying the target type by the video sensor can be further calculated by the traffic flow statistical data in the data processing unit. Count and track the detected targets, and calculate the traffic flow, average speed, occupancy rate, and headway of each vehicle type (non-motor vehicle, car, bus, truck) in each cycle in a specified area Etc. traffic flow statistics.

The data processing unit can also determine whether the target has traffic events such as speeding, retrograde, or parking violation based on the target speed and coordinates detected by the millimeter wave radar sensor. If a traffic incident occurs, the data processing unit sends an instruction to the video sensor to take a picture, take video forensics, and transfer the text information, image data, and video data of the incident to the system platform.

Although the present invention has been disclosed as the above preferred embodiments, the embodiments are not intended to limit the present invention. Any equivalent changes or modifications made without departing from the spirit and scope of the present invention also belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the content defined in the claims of the present application.

Claims

A traffic detection system based on millimeter wave radar and video, characterized in that the traffic detection system includes a data acquisition unit, a data processing unit, a data storage unit, a data communication unit, and a system platform;

The data acquisition unit includes a millimeter-wave radar sensor and a video sensor. The millimeter-wave radar sensor collects the target's coordinate position, target speed, and target length in real time and sends it to the data processing unit; the video sensor collects high-definition image data in real time and sends it to the data processing unit ;

The data processing unit includes a microprocessor and an embedded neural network processor, fuse the data information from the millimeter wave radar and video sensor detected by the same target, and finally output the complete information of the same target to the system platform in a unified format;

The implementation of the traffic detection system includes the following steps:

Step 1: The millimeter-wave radar sensor collects target information. The target information includes the identifier, speed, coordinates, and length of the target. The video sensor collects image information of the road surface. The millimeter-wave radar sensor and the video sensor collect The original data is sent to the data processing unit;

Step 2: Convert world coordinates to image plane coordinates; the millimeter wave radar sensor collects the target coordinates, and then corrects the internal and external parameters of the camera of the video sensor to obtain the conversion matrix between the target coordinates and the image plane coordinates. The coordinates are converted into image plane coordinates, and the position of the target detected and collected by the millimeter wave radar sensor on the image plane is determined;

Step 3: A neural network framework is used to identify the target type in the image captured by the video sensor; the neural network framework algorithm is run in the embedded neural network processor, and the unstructured data of the road surface image is converted into the target type through image machine learning And color structured data;

Step 4: Use the coordinate conversion of Step 2 and the image recognition of Step 3 to correspond to the targets detected by the millimeter wave radar sensor in the image captured by the camera, and obtain all the parameter information of each target in the image Fusion of superimposed target parameters, the parameter information includes the speed, coordinates and target type of each target;

Step 5: Send the fused target parameters to the system platform.
The traffic detection system based on millimeter wave radar and video according to claim 1, characterized in that: in step two, the world coordinates are converted into image plane coordinates, that is, the point of the target in the world coordinate system to the point in the image coordinate system The conversion is divided into two steps:

(1) Transform the coordinates of the target detected by the millimeter wave radar sensor in the world coordinate system (X w , Y w , Z w ) into the camera coordinate system (X c , Y c , Z c ), the transformation formula is:

In the formula,
Is the rotation matrix from the world coordinate system to the camera coordinate system;

which is

Among them, the rotation around the X axis is α, the rotation around the Y axis is β, and the rotation around the Z axis is θ;

Is the translation matrix from the world coordinate system to the camera coordinate system. Since the distance between the millimeter wave radar sensor and the video sensor in this device is very close, the length of the translation is equivalent to 0, that is

Available:

The available target is transformed from the coordinates in the world coordinate system (X w , Y w , Z w ) to the camera coordinate system (X c , Y c , Z c )

X C = cos(β)cos(θ)X W +cos(β)sin(θ)Y W -sin(β)Z W

Y C = (-cos(α)sin(θ)+sin(α)sin(θ)cos(θ)) X W +(cos(α)cos(θ)+sin(α)sin(β)sin( θ))Y W

+sin(α)cos(β)Z W

Z C = (sin(α)sin(θ)+cos(α)sin(β)cos(θ)) X W +(-sin(α)cos(θ)+cos(α)sin(β)sin( θ))Y W

+cos(α)cos(β)Z W

(2) Transform the detected target from the camera coordinate system to the image coordinate system to determine the projection position (X S , Y S ) of the target in the picture taken by the video sensor to realize the detection by the millimeter wave radar sensor Target positioning on the image:

ω is the horizontal angle of view of the target in the camera,
Is the vertical viewing angle of the target in the camera, v is the horizontal dimension of the image, and h is the vertical dimension of the image.
The traffic detection system based on millimeter wave radar and video according to claim 1, characterized in that in step three, the data after the video sensor recognizes the target type, the traffic flow statistical data is calculated in the data processing unit Track the detected target and calculate the traffic flow statistics of each vehicle type in each cycle in a specified area.
The traffic detection system based on millimeter wave radar and video according to claim 1, characterized in that the data processing unit determines whether the target has overspeed, retrograde, illegal parking, etc. according to the target speed and coordinates detected by the millimeter wave radar sensor Traffic incident; if a traffic incident occurs, the data processing unit sends an instruction for the video sensor to take a picture, record video forensics, and transfer the event text information, image data, and video data to the system platform.