WO2022141914A1 - Multi-target vehicle detection and re-identification method based on radar and video fusion - Google Patents

Abstract

A multi-target vehicle detection and re-identification method based on radar and video fusion. A roadside sensing device mainly used in the method comprises a roadside video camera and a millimeter-wave radar. In the method, multi-target vehicle detection is performed on a travelling vehicle in a road by using two types of sensors, i.e. a video camera and a millimeter-wave radar, and data fusion is performed on video camera image data and millimeter-wave radar detection target data by using a multi-sensor data fusion method, such that the precision of target detection data is improved. In the method, two scenes, i.e. a single-video camera performing continuous tracking on a multi-target vehicle and a cross-video camera performing continuous tracking on the multi-target vehicle, are taken into consideration; on the basis of a multi-target vehicle image feature extracted on the basis of a video image, in combination with data detected by a roadside millimeter-wave radar, a geospatial feature of the multi-target vehicle is acquired, and a new dimension is added to a vehicle attribute; and the present invention is applied to a multi-target vehicle re-identification algorithm, such that the accuracy of multi-target vehicle re-identification can be effectively improved, thereby improving the vehicle supervision level in a management and control region.

Description

A multi-target vehicle detection and re-identification method based on Raivision fusion technical field

The invention belongs to the technical field of moving vehicle target detection, multi-sensor data fusion and vehicle re-identification, and relates to a method for detecting and re-identifying multi-target vehicles using video camera image data and millimeter-wave radar data fusion.

Background technique

In recent years, with the continuous development of my country's economy and the improvement of people's living standards, driving has become the preferred way for many people to travel, and commuting by car has also become a relatively common phenomenon. The rapid growth of motor vehicles puts more pressure on traffic regulators. As video surveillance plays an increasingly important role in the field of public safety, vehicle-related tasks such as vehicle object detection, vehicle classification, vehicle tracking, and driver behavior analysis have received increasing attention. How to judge whether the vehicle between different image frames of a single video camera is the same vehicle, and whether the vehicle in the video images between cross-video cameras is the same vehicle has become an important requirement for vehicle management and control. Carry out follow-up work such as vehicle trajectory tracking and driver behavior analysis.

Vehicle re-identification refers to judging whether the vehicle data captured by roadside sensors (video cameras, millimeter-wave radar, etc.) Vehicle identification problem. Solving this problem is of great significance to the precise management and control of vehicles in the region, regional security, and vehicle-road coordination.

At present, the existing vehicle re-identification methods are mainly divided into four categories:

Category 1: Sensor-Based Approaches

Using various sensors to detect vehicles and infer their identities is the most basic and earliest vehicle re-identification method. For each vehicle detected by the sensor, such methods usually use specific means to extract vehicle features, and determine the vehicle identity through feature matching. The earliest vehicle re-identification methods are based on various hardware detectors (such as infrared, ultrasonic, etc.) to extract the characteristic information of vehicles. After that, many vehicle recognition methods using other sensors or sensors have been proposed, such as the use of three-dimensional magnetic sensors to detect multi-dimensional features of vehicles and to obtain temporal information from the sensors for training Gaussian maximum likelihood classifiers. Induction coils are the most commonly used data acquisition tools in traffic scenarios, which can monitor various vehicle attributes (such as speed, volume, and vehicle footprint). Induction coils are generally deployed on urban arterial roads and highways. The real-time vehicle re-identification method based on the induction coil uses the data provided by the induction coil to extract the vehicle characteristics and estimate the travel time of the vehicle, so as to complete the vehicle re-identification task. With the advent of some emerging sensors or technologies such as Global Positioning System (GPS), Radio Frequency Identification (RFID), and cell phones, some approaches have explored beacon-based vehicle tracking and surveillance systems to accomplish vehicle re-identification tasks. For example, the vehicle re-identification algorithm based on radio frequency identification tags is suitable for various toll stations, and the GPS-based vehicle travel time estimation method is used to solve the problem of vehicle re-identification. Most of the sensor-based methods require the installation of a large number of hardware devices, and the experimental environment is harsh and difficult to reproduce. In addition, many methods are easily affected by the objective environment, such as weather conditions, signal strength, traffic congestion and vehicle speed, which will reduce the sensitivity of the sensor to varying degrees. There is also no unified performance evaluation standard in the same period, so this kind of method cannot be regarded as an ideal vehicle re-identification method.

The second category: based on vehicle license plate information.

With the development of computer vision technology, it is possible to identify vehicle identity information through image or video data, and the deployment of hardware facilities is greatly reduced, which can save a lot of hardware costs and labor installation costs. In the initial stage, the vehicle re-identification method based on computer vision technology mainly extracts the vehicle license plate information through the methods of license plate location, character segmentation and recognition. The positioning function is mainly realized by grayscale information, color and texture information, and the segmentation and recognition functions are mainly realized by template matching and neural network methods. The vehicle re-recognition method based on license plate has high accuracy, but its shortcomings are also very obvious: the traffic monitoring system has problems such as the change of shooting angle, the influence of weather, the change of illumination and the low image resolution. Once the license plate information of the target vehicle is lost, the method will fail. In the actual complex traffic environment, it is a common occurrence that the license plate information is difficult to obtain due to factors such as the occlusion of the license plate, the low resolution of the camera, the long shooting distance and the shooting angle, which will greatly reduce the license plate recognition. Accuracy. For the phenomenon of fake license plates, sets of license plates or even no license plates, it is powerless to do anything. Therefore, although license plate recognition is the simplest and most direct method to distinguish different vehicles, in many cases, only relying on license plate information cannot complete the re-recognition task.

The third category: based on vehicle image features.

It does not completely rely on the license plate, and integrates other non-license plate information to complete the vehicle re-identification work, thereby improving the stability and accuracy of the vehicle re-identification. The traditional vehicle re-recognition method without license plate information mainly performs the image feature matching process by extracting the HSV features, LBP features and HOG features of the vehicle image, and then realizes the vehicle re-identification process by classifying the vehicle color, model, and vehicle windshield. Identify work. The vehicle re-identification method based on no license plate information has strong interpretability, but is easily affected by the change of viewing angle and occlusion, and the re-identification accuracy is still low.

The fourth category: based on machine learning and deep neural networks.

In recent years, with the development of artificial intelligence algorithms and deep neural networks in the field of computer vision, vehicle re-identification has a new technical route. More and more vehicle re-identification methods based on machine learning and deep learning have been proposed, which greatly improve the accuracy of vehicle re-identification and gradually become the mainstream method for vehicle re-identification work.

The convolutional neural network takes the image as input, and does not need to extract complex artificial features in advance, and performs feature extraction through the process of continuous forward learning and backward feedback. Each layer of the convolutional neural network mainly includes feature extraction and feature mapping operations. In the feature extraction operation, the input of the neuron is the output of the previous layer, and the convolution kernel is used to perform the convolution operation on the input to obtain local features. Each layer can use multiple convolution kernels, which means that multiple features are extracted for the input. . Due to the weight sharing of the convolution kernel, the parameters of the network are greatly reduced. In the feature mapping operation, the sigmod or tanh function is used as the activation function of the convolutional network, so that the extracted features have displacement invariance. The convolutional neural network feature extraction operation automatically learns the training data, avoiding the use of artificially defined feature extraction methods to extract features fixedly, but implicitly learns from the training data autonomously, and because the convolution kernel weights are shared , so that the parallel learning can improve the computational efficiency.

Based on convolutional neural network and considering the influence of pose on recognition accuracy. The main method is to obtain multiple regional segmentation results of the target vehicle from the image to be recognized; use the convolutional neural network CNN to extract regional feature vectors from the multiple regional segmentation results, and fuse them with the global feature vectors to obtain the target vehicle's segmentation results. Appearance feature vector. Finally, the fused feature vector is used for vehicle re-identification and retrieval. Although this scheme considers the influence of posture on vehicle re-identification, the accuracy of the model is limited by the diversity of the dataset, which must include vehicle images from various angles. , and the scale is large enough, in real scenarios, it is difficult to collect all vehicle images from different angles and the number of datasets reaches hundreds of thousands. In addition, on the collected data set, it is necessary to mark the key points of the vehicle pictures from different angles. The angles of different pictures are different, so the number and position of the marked key points are different, resulting in a huge workload. Therefore, the method is more complicated in terms of feasibility and workload.

In addition, due to changes in camera perspective and illumination, the same vehicle has large intra-class differences under different perspectives or lighting conditions, or different vehicles have the same model to form high inter-class similarity. This severe challenge still limits the accuracy of vehicle re-identification. the main reason for the rate.

The above types of vehicle re-identification methods are based on images or video data captured by roadside cameras, and re-identify multi-target vehicles at the vehicle appearance level. The flow chart is shown in Figure 1, including vehicle image or video acquisition, vehicle Detection, feature extraction and expression, similarity metric calculation and display of detection results. However, it is not uncommon for different vehicles to be identical in appearance, except for the license plate. In this case, only re-identifying the target vehicle from the appearance level will greatly increase the misjudgment rate.

Millimeter wave radar is a radar that works in the millimeter wave band. Usually millimeter wave refers to the 30-300GHz frequency domain (wavelength is 1-10mm). The wavelength of millimeter wave is between microwave and centimeter wave, so millimeter wave radar has some advantages of microwave radar and photoelectric radar. Compared with the centimeter-wave seeker, the millimeter-wave seeker has the characteristics of small size, light weight and high spatial resolution. Compared with optical seekers such as infrared, laser, and TV, the millimeter-wave seeker has a strong ability to penetrate fog, smoke, and dust, and has the characteristics of all-weather (except heavy rain) all day. The propagation of light waves in the atmosphere is seriously attenuated, and the processing precision of the device is required to be high. Compared with light waves, millimeter waves use the atmospheric window (when millimeter waves and submillimeter waves propagate in the atmosphere, some attenuation due to the resonance absorption of gas molecules is a minimum frequency), the attenuation is small when they propagate, and they are affected by natural light. and thermal radiation sources have little impact. To this end, they are of great significance in communications, radar, guidance, remote sensing technology, radio astronomy and spectroscopy. Its advantages are mainly as follows:

(1) Small antenna aperture and narrow beam: high tracking and guidance accuracy; easy to perform low-elevation tracking, anti-ground multipath and clutter interference; high lateral resolution for near-air targets; high angular resolution for regional imaging and target monitoring high anti-jamming performance with narrow beam; high antenna gain; easy to detect small targets, etc.

(2) Large bandwidth: With high information rate, it is easy to use narrow pulse or wideband FM signal to obtain the detailed structural characteristics of the target; it has wide spectrum spreading ability, reduces multipath, clutter and enhances anti-jamming ability; adjacent frequency radar Or millimeter wave identifiers work, easy to overcome mutual interference; high distance resolution, easy to obtain accurate target tracking and identification capabilities.

(3) High Doppler frequency: good detection and identification capabilities of slow targets and vibrating targets; easy to use target Doppler frequency characteristics for target feature identification; penetration characteristics of dry air pollution, providing high performance in dust, smoke and dust Good detection capability in dry snow conditions.

Video cameras can extract the image features of multi-target vehicles, and can also use camera calibration technology and target detection algorithms to obtain positioning data of multi-target vehicles. Millimeter-wave radar can also detect the positioning data of multi-target vehicles, and the data fusion algorithm can obtain higher-precision positioning data of multi-target vehicles. In addition, millimeter-wave radar can accurately capture geospatial information such as vehicle speed and driving direction, which can add a new dimension to vehicle attributes. Combined with video image data, it can provide a solution to the above-mentioned vehicle re-identification problem.

current technology

Patent document CN108875754A

Patent document CN111582178A

Patent document CN111553205A

Patent document CN109508731A

Patent document CN111435421A

Description of related terms

1. Single video camera: a single video camera, multi-target vehicles can drive into and out of the field of view of the video camera.

2. Cross-video camera: a set of two or more video cameras with different fields of view. A multi-target vehicle can drive out of the field of view of one video camera and enter the field of view of another video camera.

3. Vehicle image features: The vehicle image HSV value, LBP and HOG are all vehicle image features. In addition, it also includes vehicle color, model, size, and license plate information.

4. Vehicle geospatial features: refers to the longitude and latitude coordinates, speed, and heading angle information of the vehicle.

5.HSV: HSV (Hue, Saturation, Value) is a color space created by A.R. Smith in 1978 based on the intuitive characteristics of color, also known as the hexagonal pyramid model. The parameters of the color in this model are: Hue (H), Saturation (S), Lightness (V).

6. LBP: The LBP (Local Binary Pattern) operator is an effective texture description operator, which has significant advantages such as rotation invariance and grayscale invariance. The basic idea is to use the gray value of the central pixel as a threshold and compare the binary code with its neighborhood to express local texture features.

7.HOG: Histogram of Oriented Gradient feature is a feature descriptor used for object detection in computer vision and image processing. It constructs features by calculating and counting the gradient direction histograms of local regions of the image. HOG features combined with SVM classifiers have been widely used in image recognition.

8. Large intra-class differences: Targets in the same category have large differences in feature reflection. In vehicle re-identification, it is reflected as follows: for the same vehicle, due to factors such as different shooting angles of video cameras and different light intensities, the image features of the same vehicle are quite different.

9. High similarity between classes: Targets of different classes have high similarity in feature reflection. In vehicle re-identification, it is reflected as follows: for different vehicles, because their models, colors, sizes are similar or belong to the same brand and the same model, they are reflected in the image features with high similarity.

10. Video camera data: video image data, after calibrating the camera and performing target detection on the image, the pixel coordinates of the image detection target and the world geographic coordinates can be obtained.

11. Video camera positioning data: the target positioning data output by the calibrated video camera, that is, the coordinates in the world geographic coordinate system converted from the pixel coordinate system.

12. Millimeter-wave radar data: the distance, azimuth, speed and vehicle heading angle of the vehicle relative to the millimeter-wave radar. The coordinates of the vehicle in the radar coordinate system can be calculated according to the above data, and the vehicle can be calculated after calibrating the millimeter-wave radar. The coordinates in the world coordinate system.

13. Millimeter-wave radar positioning data: target positioning coordinate data in the millimeter-wave radar data.

14. RTK: High-precision GPS measurement must use carrier phase observations. RTK equipment adopts differential positioning technology, that is, real-time dynamic positioning technology based on carrier phase observations, which can provide real-time three-dimensional positioning results of the station in the specified coordinate system. , and achieve centimeter-level accuracy. RTK can be divided into handheld type and vehicle type according to the application. Hand-held RTK can be used for hand-held single-point coordinate measurement. The vehicle type can be installed on the test vehicle for continuous coordinate measurement.

15. RTK positioning data: latitude and longitude coordinates obtained by handheld and vehicle-mounted RTK measurements.

16. Data set: The data set obtained by the data acquisition equipment, this patent refers to the positioning data of the experimental test vehicle collected by the video camera, the millimeter wave radar and the vehicle RTK equipment.

17. Central computing server: computer equipment used to receive, process and store data (including video camera image data and millimeter-wave radar positioning data) acquired by each data acquisition device.

18. Space-time matching: refers to the optimization of the matching method, so that the data sets obtained by different devices are synchronized in time for the positioning data collected by the same target, and are consistent in space, that is, the error is within an acceptable range (0.5 meters).

19. World geographic coordinate system: The world geographic coordinate system described in this patent is the WGS-84 coordinate system in the geographic coordinate system.

20. Pixel coordinate system: The pixel coordinate system represents the position of the image pixel in the image. Usually, the pixel in the upper left corner of the image is used as the origin, and the right direction is the positive direction of the x-axis, the downward direction is the positive direction of the y-axis, and the pixels are horizontal and vertical. The coordinates represent the number of pixels from the pixel to the y-axis and the x-axis, respectively.

21. Radar coordinate system: the millimeter-wave radar coordinate system, a three-dimensional space coordinate system with the millimeter-wave radar itself as the origin of the coordinate system.

22. Homography transformation matrix: The coordinate transformation matrix in different coordinate systems can be calculated by selecting the corresponding key points (at least 4 pairs of non-collinear points) in different coordinate systems.

23. Multiple times: at least 2 times.

24. Target vehicle: A vehicle that enters the field of view of the sensor (video camera and millimeter-wave radar).

25. Multi-target vehicle: A vehicle set containing at least 1 target vehicle.

26. Multi-target vehicle data: The positioning data of at least one vehicle is obtained by video cameras and millimeter-wave radars.

27. Deep learning target detection algorithm: The deep learning target detection algorithm used in this patent is the YOLO v5 target detection algorithm to detect targets on vehicles on the road.

28.bb: Bounding box, a rectangular box for target detection, which is returned by the calculation result of the target detection algorithm and is used to frame the detected target contour in the video image.

29. Multi-sensor data fusion algorithm: Synthesize the integrated multi-source data to form the best consistent estimate of the measured object and its properties. In this patent, Kalman filtering algorithm is used for data fusion.

30. Mahalanobis distance: Mahalanobis distance was proposed by Indian statistician P.C. Mahalanobis, which represents the distance between a point and a distribution. It is an efficient method to calculate the similarity between two unknown sample sets. Unlike Euclidean distance, it takes into account the connections between various properties (eg: a piece of information about height leads to an information about weight because the two are related) and is scale-independent ( scale-invariant), i.e. independent of the measurement scale.

31. Cosine distance: Cosine distance, also known as cosine similarity, or cosine similarity, evaluates the similarity of two vectors by calculating the cosine value of the angle between the two vectors.

32. Roadside: located in other positions that do not belong to the road surface itself, such as the side of the road, above the road (supported by poles or gantry), etc.

33. Lost vehicles: vehicles that have appeared within the collective field of view of the sensor devices (video cameras, millimeter-wave radars), but are currently not captured by a single device.

34. Sensor equipment: The sensors in this patent refer to instruments that can obtain data, including video cameras, millimeter-wave radars, and RTK positioning equipment (handheld and vehicle-mounted).

35. Vehicle re-identification system: a system framework for re-identification of lost vehicles.

36. Vehicle Loss Database: Record the last n frames before the vehicle is lost, (the value of n can be adjusted according to the effect) features, including vehicle image features and geospatial features.

37. ID re-assignment: When the vehicle re-identification system retrieves the lost vehicle, it assigns the vehicle's historical ID to the retrieved vehicle.

[Content of the invention]

In order to solve the problem that the current vehicle re-identification technology based on video images may cause large intra-class differences for the same vehicle under different viewing angles or lighting conditions due to changes in camera angle of view and illumination, or high inter-class similarity between different vehicles due to the same model. The problem that the recognition accuracy is not high, the technical solution idea adopted by the present invention is:

A multi-target vehicle detection and re-identification method based on Raivision fusion, which adds the dimension of geospatial information to vehicle features by adding millimeter wave radar detection data.

The specific plans are:

1) Calibrate the video camera S ₁ and the millimeter-wave radar S ₂ , and use the on-board RTK positioning data D ₀ as the relative true value (because the error level of the on-board RTK positioning data is centimeter-level, the video camera positioning data D ₁ and the millimeter-wave radar positioning _The error level _of data D2 is meter level, and the accuracy of vehicle RTK positioning data is much higher _than the accuracy of video camera and millimeter wave radar positioning data ₎ . The optimization algorithm performs space-time matching on the positioning data D ₁ and D ₂ to complete the sensor calibration.

2) Using the multi-target vehicle data obtained by the video camera and the millimeter-wave radar, the deep learning target detection algorithm is used to detect the multi-target vehicle in the video image of the video camera and extract the image features and geospatial features of the target vehicle respectively. Taking the vehicle RTK positioning data D ₀ as the relative true value, the multi-sensor data fusion algorithm is used to fuse the multi-target vehicle data obtained by the video camera and the millimeter-wave radar to obtain the best estimate of the multi-target vehicle positioning accuracy. Compared with D ₁ and D ₂ , data D ₃ has higher positioning accuracy;

Using the fused multi-target vehicle data, continuous tracking and re-identification judgment of multi-target vehicles between different image frames of a single video camera and multi-target vehicles across video camera images are performed.

The present invention specifically mainly relates to technical problems including the following aspects, which are respectively:

1. RTK differential positioning technology;

2. Camera calibration technology;

3. Image data multi-target vehicle detection technology;

4. Millimeter wave radar calibration technology;

5. Multi-sensor data spatiotemporal matching optimization technology;

6. Multi-sensor data fusion technology;

7. Continuous tracking and re-identification technology for multi-target vehicles in a single video camera image;

8. Continuous tracking and re-identification technology for multi-target vehicles across video camera images;

The specific flow of the technical solution adopted by the present invention to solve the technical problem is as follows:

1. Calibration:

1) Camera calibration: Use a handheld RTK device to accurately locate the key points in the camera image with world geographic coordinates, and select at least 4 non-collinear key points for positioning. Determines the pixel coordinates of the keypoint in the camera image pixel coordinate system. Calculate the homography transformation matrix between the pixel coordinate system and the world geographic coordinate system to complete the camera calibration.

2) Target detection frame coordinate calibration: install on-board RTK equipment for the experimental test vehicle to obtain real-time vehicle positioning data. Drive the experimental test vehicle into the field of view of the camera, and use the deep learning target detection algorithm to mark the detection frame for the experimental vehicle in the video image. Take the pixel coordinates of the midpoint of the bottom edge of the detection frame as the pixel coordinates of the experimental test vehicle, and calculate the world geographic coordinates of the experimental vehicle according to the calculated homography transformation matrix corresponding to the pixel coordinate system of the video camera image and the world geographic coordinate system, that is, Experimental test vehicle localization data for video camera detection. Collect the experimental vehicle positioning data and the synchronized vehicle RTK positioning data D ₀ output by the video camera of the experimental vehicle for a period of time (the duration must be greater than 2 minutes, and the experimental vehicle repeatedly appears in the field of view of the video camera), and establish D ₀ and D 0 _1. The space-time matching optimization model between the two data sets, solve the parameters to complete the coordinate calibration of the target detection frame of the video camera. 3) Millimeter-wave radar calibration: Calculate the relative coordinates of the experimental vehicle relative to the millimeter-wave radar according to the original data of the millimeter-wave radar (including the distance and azimuth data of the target relative to the millimeter-wave radar), that is, the coordinates of the experimental vehicle in the radar coordinate system . Collect the positioning data D ₂ of the experimental vehicle and the synchronized vehicle RTK positioning data D ₀ obtained by the millimeter-wave radar of the experimental vehicle for a period of time (the duration must be greater than 2 minutes, and the experimental vehicle repeatedly appears in the field of view of the millimeter-wave radar), Calculate the homography transformation matrix of the two data sets D ₀ and D ₂ and establish a space-time matching optimization model between the two data sets, and solve the parameters to complete the millimeter wave radar calibration. The millimeter wave radar calibration acquisition data and the target detection frame coordinate calibration acquisition data can be performed simultaneously or sequentially (in no particular order).

2. Data acquisition: use the video camera deployed on the roadside to acquire the video image data within the field of view of the camera, and use the millimeter-wave radar deployed on the roadside to acquire the multi-target vehicle data within the radar field of view, including the target ID of each vehicle and the millimeter-wave radar data (Positioning coordinates, speed, heading angle, etc. in the world geographic coordinate system). The data obtained by the video camera and the millimeter-wave radar (including video image data and millimeter-wave radar positioning data) are transmitted to the central computing server for subsequent calculation.

3. Vehicle target recognition and feature extraction: After the central computing server receives the data uploaded by the video camera and millimeter-wave radar (including video image data and millimeter-wave radar positioning data), it performs the steps of vehicle target recognition and feature extraction. For video image data: use the pre-trained deep learning target detection algorithm to identify the multi-target vehicles within the camera's field of view, and extract the target vehicle detection frame image (target detection frame selection) with high confidence (confidence degree greater than 0.6). The video image part within the range, which belongs to a subset of the complete video image), and calculates the image features of each target vehicle (including: vehicle color, model, size, license plate information (optional)) and geospatial features, that is, positioning data ( world geographic coordinates). For millimeter wave radar data: extract the geospatial features of the target vehicle, including: vehicle world geographic coordinates, speed, heading angle, etc.

4. Feature matching and data fusion: Match the geospatial features of multi-target vehicles obtained by video cameras and millimeter-wave radars, and use multi-sensor data fusion methods to fuse the positioning data obtained by video cameras and millimeter-wave radars to improve the accuracy of positioning data. .

5. Vehicle re-tracking:

1) Re-tracking of vehicles in the field of view of a single video camera: Use the pre-trained deep learning target detection algorithm to identify vehicles in the field of view of a single video camera and mark the target frame, use the Deepsort multi-target tracking algorithm to assign IDs to multi-target vehicles, and use the Deepsort multi-target tracking algorithm to assign IDs to multi-target vehicles. The target frame between them is re-identified and matched to realize the re-tracking of the vehicle within the field of view of the single video camera.

2) Vehicle re-tracking across video cameras: when a target vehicle V _x is lost in the field of view of camera 1, the vehicle re-identification system records the vehicle image features and geospatial features before the target vehicle V _x is lost, and transfers it to the lost vehicle database , when the lost target vehicle V _x enters the field of view of the camera 2, the vehicle re-identification system captures vehicle features (including image features and geospatial features) and performs feature matching in the vehicle lost database. 0.5), the ID re-assignment is performed. When the ID is re-assigned, the lost vehicle ID with the highest similarity in the lost database is taken for ID re-assignment to realize the vehicle re-tracking. If no matching vehicle is found in the vehicle lost database (that is, the similarity of all lost vehicle features with the vehicle lost database is lower than 0.5), a new ID is assigned to the target vehicle V _x .

Brief Description of Drawings

Figure 1 Flow chart of traditional vehicle re-identification technology route;

2 is a flow chart of the vehicle re-identification technology route of the present invention;

Figure 3 is a schematic diagram of sensor layout;

4 is a schematic diagram of camera calibration;

5 is a schematic diagram of a camera calibration image;

6 is a schematic diagram of coordinate conversion of a camera image vehicle inspection target frame;

Fig. 7 is the flow chart of the time synchronization optimization model between the target detection frame data and the vehicle RTK positioning data;

Figure 8 is a flow chart of spatial error calibration between target detection frame data and vehicle RTK positioning data;

Figure 9 is a schematic diagram of the rotation offset angle and target detection of the millimeter-wave radar;

Figure 10 is a schematic diagram of the calculation of the world geographic coordinates of the target detected by the millimeter wave radar;

Figure 11 is the flow chart of the optimization model for time synchronization between millimeter-wave radar detection data and vehicle RTK positioning data;

Figure 12 is a flow chart of spatial calibration of millimeter-wave radar detection data and vehicle RTK positioning data;

13 is a schematic diagram of vehicle re-identification with a single video camera;

Figure 14 is a flow chart of vehicle recognition across video cameras;

FIG. 15 is a schematic diagram of vehicle re-identification across video cameras.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

The invention relates to a multi-target vehicle detection and re-identification method based on Raivision fusion. The overall technical route is shown in Figure 2. The specific process is divided into five steps, including: sensor layout and data collection, sensor calibration (video). camera and mmWave radar), multi-sensor data fusion, and vehicle re-identification methods (single video camera vs. cross-video camera).

Step 1: Sensor layout and data collection

The invention realizes multi-vehicle target detection, tracking, data fusion and vehicle re-identification by relying on two types of sensor data of video cameras and millimeter-wave radars. Video cameras use directional cameras, and millimeter-wave radars use long-range radars in the 79GHz band. The basic layout plan is shown in Figure 3. The video camera and the millimeter-wave radar are placed in the same position on the roadside pole (to ensure that the distance between the longitude and latitude coordinates of the two sensors is less than 0.5 meters), so that the world geographic coordinates of the two are consistent. The detection range of the sensor can be adjusted according to the installation height and angle. When the installation height of the sensor is 6 meters, the downward tilt angle is 10° and the scene has no other obstructions, its field of view can reach 100-150m. Both types of sensors collect data at a frequency of 25Hz and access the central server for data storage and processing. For video image data: use the pre-trained deep learning target detection algorithm to identify the multi-target vehicles within the camera's field of view, and extract the image features of each target vehicle (including: vehicle color, model, size, license plate information (non- necessary)) and geospatial features (world geographic coordinates). For millimeter wave radar data: extract the geospatial features of the target vehicle (including: vehicle world geographic coordinates, speed, heading angle, etc.).

Variant A: The millimeter-wave radar and the video camera are arranged on different roadside poles to ensure that the overlap rate of the two sensors is greater than 90%. The sensing range of the sensing area of this road segment is the intersection of the visual range of the video camera and the millimeter-wave radar.

Variant B: The millimeter-wave radar and the video camera are arranged on the same gantry above the road to ensure that the overlap rate of the two sensors is greater than 90%. The sensing range of the sensing area of this road segment is the intersection of the visual range of the video camera and the millimeter-wave radar.

Variant C: The millimeter-wave radar and the video camera are arranged on different gantry above the road to ensure that the overlap rate of the two sensors is greater than 90%. The sensing range of the sensing area of this road segment is the intersection of the visual range of the video camera and the millimeter-wave radar.

Step 2: Calibration

(1) Video camera calibration

The calibration of the video camera in the present invention refers to establishing the mapping relationship between the pixel coordinate system in the camera image and the world geographic coordinate system, that is, the homography transformation matrix H, as shown in formula (1). The world geographic coordinate system is to select a reference coordinate system in the environment to describe the position of the camera and the object, which is called the world geographic coordinate system. The present invention stipulates that the world geographic coordinate system is the WGS-84 coordinate system in the geographic coordinate system. In the present invention, the world geographic coordinates are acquired by RTK equipment. High-precision GPS measurement must use carrier phase observations. RTK equipment uses differential positioning technology, that is, real-time dynamic positioning technology based on carrier phase observations. It can provide real-time three-dimensional positioning results of the station in the specified coordinate system, and achieve Centimeter-level accuracy. Because of its high positioning accuracy, the present invention regards the world geographic coordinates obtained through RTK as a relative true value. The pixel coordinate system represents the position of the image pixel in the image. Usually, the pixel in the upper left corner of the image is used as the origin, and the right direction is the positive direction of the x-axis, and the downward direction is the positive direction of the y-axis. The horizontal and vertical coordinates of the pixels represent the distance of the pixel respectively. The number of pixels in the y-axis and x-axis. Equation (2) describes that the world geographic coordinate system and the pixel coordinate system are obtained through homography matrix transformation, where longitude represents longitude, latitude represents latitude, x represents the abscissa of the pixel coordinate, and y represents the abscissa of the pixel coordinate. Formula (3) and formula (4) are conversion calculation formulas.

As shown in FIG. 4 and FIG. 5 , place the handheld RTK mobile station within the field of view of the video camera, and select at least four non-collinear key points to collect its world geographic coordinates. In the camera image, select the pixel coordinates of the bottom of the handheld RTK mobile station handheld stick as the pixel coordinates corresponding to the positioning point. Finally, the homography transformation matrix between the two coordinate systems is calculated, the mapping relationship between the pixel coordinates of the video image and the world geographic coordinates is established, and the parameter calibration of the video camera is completed.

(3) Coordinate calibration of target detection frame

The target detection frame coordinate calibration is to establish the mapping relationship between the pixel coordinates of the vehicle target detection frame in the camera image and the vehicle world geographic coordinates. The pixel coordinates of the vehicle target detection frame refer to the pixel coordinates of the midpoint pixel of the lower bottom edge of the detection frame. (The camera is located directly above the road coming from the road)

Variant A: The camera is located at the upper left of the road, and the pixel coordinates of the vehicle target detection frame refer to the pixel coordinates of the lower right vertex pixel of the detection frame.

Variant B: The camera is located on the road to the upper right, and the pixel coordinates of the vehicle target detection frame refer to the pixel coordinates of the vertex pixels at the lower left of the detection frame.

The target detection frame is obtained by the deep learning target detection algorithm, the data frequency is 25Hz, and the vehicle world geographic coordinates are obtained by the vehicle RTK, and the data frequency is 5Hz. Install on-board RTK for the experimental test vehicle to obtain the real-time world geographic coordinate data of the vehicle, drive the vehicle into the field of view of the camera, and use the deep learning target detection algorithm to mark and detect the experimental vehicle in the video image to obtain its pixel coordinates. According to the calculated homography transformation matrix corresponding to the pixel coordinates of the camera detection target and the world geographical coordinates, the world geographical coordinates of the detected target vehicle are calculated. It is a common phenomenon that the clocks between different sensors are out of synchronization. In addition, the world geographic coordinates converted from the pixel coordinates of the target detection frame are different from the real world geographic coordinates of the vehicle. It is necessary to establish a space-time matching optimization model for two types of world geographic coordinates, including time There are two parts of synchronous optimization model and spatial error calibration.

As shown in FIG. 7 , in the time synchronization optimization model, it is first necessary to unify the two types of data collection frequencies. The linear interpolation method is used to upsample the world geographic coordinates obtained by the vehicle RTK, so that the sampling frequency is consistent with the data frequency of the target detection frame. The objective function of the time synchronization optimization model is established as shown in formula (5). The optimization algorithm is used to find a set of suitable parameters Δt to minimize the Euclidean distance between the two world geographic coordinates in the time series.

Among them, longitude _{bb_t} and latitude _{bb_t} represent the world geographic coordinates corresponding to the pixel coordinates of the target detection frame at time t, and longitude _{rtk_t+Δt} and latitude _{rtk_t+Δt} represent the world geographic coordinates measured by the vehicle RTK at time t+Δt.

As shown in Figure 8, in the spatial error calibration, the spatial error distribution of two types of world geographic coordinates on the pixel coordinate system is established. The error calculation formula is shown in formula (6), and the error spatial distribution is calculated by the method of surface fitting. Curved surface, the world geographic coordinates calculated by the target detection frame are corrected according to formula (7).

Error=(longitude _{bb_t} , latitude _{bb_t} )-(longitude _{rtk_t+Δt} , latitude _{rtk_t+Δt} ) (6)

coordinate _correction = (longitude _{bb_t} , latitude _{bb_t} )-Error Curced Surface (7)

(3) Millimeter wave radar calibration

According to the characteristics of the millimeter-wave radar suitable for detecting moving targets, the millimeter-wave radar was calibrated using an experimental vehicle equipped with an on-board RTK. The data returned by the radar is target-level data, including the distance, azimuth, speed, etc. of the detected target. Figure 9 shows a schematic diagram of millimeter-wave radar target detection. The relative coordinates of the experimental vehicle relative to the radar are calculated according to the original data of the millimeter-wave radar. The calculation formulas are shown in equations (8) to (10).

x=dis·cosθ ₁ ·cosθ ₂ (8)

y=dis·cosθ ₁ ·cosθ ₂ (9)

z=-dis·cosθ ₁ (10)

Among them, distance represents the distance between the target vehicle and the radar, θ ₁ and θ ₂ represent the pitch angle and horizontal angle between the target vehicle and the radar, respectively, and x, y and z represent the coordinates of the radar detection target in the radar coordinate system.

As shown in FIG. 10 , there are pose angle deviations and spatial position deviations between the radar coordinate system and the world geographic coordinate system, where the pose angle deviations include deviation angles in three directions, denoted as α, β and γ, respectively. The spatial position deviation is the deviation between the origin of the millimeter-wave radar coordinate system and the origin of the world geographic coordinate system, that is, the coordinates of the millimeter-wave radar in the world geographic coordinate system, which are respectively recorded as longitude _radar , latitude _radar and height _radar .

The pose deviation angle correction matrices of the three directions are shown in Equation (11), Equation (12) and Equation (13), respectively.

The conversion formula from the radar coordinate system to the world geographic coordinate system is shown in formula (14).

Among them, x, y and z represent the coordinates of the radar detection target in the radar coordinate system, and longitude, latitude and height represent the transformation of the coordinates of the radar detection target in the radar coordinate system into the world geographic coordinates in the world geographic coordinate system.

Denote longitude, latitude and height as the world geographic coordinates of the detected target calculated from the radar coordinates, and at the same time obtain the vehicle world geographic coordinates obtained by the vehicle RTK, denoted as longitude _rtk , latitude _rtk and height _rtk . Two types of space-time matching optimization models of world geographic coordinates are established, including two parts: time synchronization optimization model and spatial error calibration.

As shown in FIG. 11 , in the time synchronization optimization model, it is first necessary to unify the two types of data collection frequencies. The linear interpolation method is used to upsample the world geographic coordinates obtained by the vehicle RTK, so that the sampling frequency is consistent with the data frequency of the target detection frame. The objective function is established as shown in formula (5), and the optimization algorithm is used to find a set of suitable parameters α, β, γ and Δt to minimize the Euclidean distance between the two world geographic coordinates in the time series.

Among them, longitude _t and latitude _t represent that the coordinates of the radar detection target in the radar coordinate system at time t are converted into world geographic coordinates in the world geographic coordinate system, and longitude _{rtk_t+Δt} and lattitude _{rtk_t+Δt} represent the time t+Δt obtained by the vehicle RTK The world geographic coordinates of the vehicle.

As shown in Figure 12, in the spatial error calibration, the spatial error distribution of two types of world geographic coordinates on the pixel coordinate system is established. The error calculation formula is shown in formula (14), and the error spatial distribution is calculated by the method of surface fitting. Curved surface, the world geographic coordinates calculated by the target detection frame are corrected according to formula (15).

Error=(longitude _t , latitude _t )-(longitude _{rtk_t+Δt} , latitude _{rtk_t+Δt} ) (16)

coordinate _correction = (longitude _t , latitude _t )-Error Curved Surface (17)

Step 3: Video image target recognition and feature extraction

Use the pre-trained deep learning target detection algorithm to perform target recognition on multi-target vehicles within the camera's field of view, and extract the image features of each target vehicle (including: vehicle color, model, size, license plate information (optional)) and geographic location. Spatial features (world geographic coordinates), when the license plate information cannot be obtained, this column is filled with a blank value. For millimeter wave radar data: extract the geospatial features of the target vehicle (including: vehicle world geographic coordinates, speed, heading angle, etc.).

Step 4: Feature matching and data fusion

The video and the target vehicle data obtained by the millimeter-wave radar are matched according to the geospatial features, and the multi-sensor data fusion method is used to fuse the video and the vehicle speed and geographic location data obtained by the millimeter-wave radar to improve the data accuracy. Among them, in the fusion algorithm, the high-precision positioning coordinates obtained by the on-board RTK of the experimental vehicle are used as the reference truth value, and sensor data fusion methods such as Kalman filtering, multi-Bayesian estimation, fuzzy logic reasoning and deep neural network are used to The geographic location data of the target vehicle is fused to calculate.

Among them, the Kalman filter algorithm is one of the algorithms widely used in multi-source data fusion. Kalman filtering is a recursive filtering algorithm, which is characterized by no need to save past historical information, and the new data is combined with the estimated value obtained in the previous frame (or previous moment) and the state equation of the system itself according to a certain method. Find a new estimate. The principle of Kalman filter can be expressed by the following five formulas:

predict:

renew:

The meaning of each parameter is as follows:

F: state transition matrix;

B: control matrix;

P: state covariance matrix;

Q: state transition covariance matrix;

H: observation matrix;

R: observation noise variance;

The state variable at this time is inferred from time t-1, and the observation value at this time has not been corrected;

The state variable at this moment is estimated from the previous moment, and the observed value at this moment has been corrected;

z: actual observed value;

k: Kalman coefficient;

t: time t.

Step 5: Vehicle re-tracking

1) Re-tracking of vehicles in the field of view of a single video camera: Use the pre-trained deep learning target detection algorithm to identify vehicles in the field of view of a single video camera and mark the target frame, and use the Deepsort multi-target tracking algorithm to assign IDs to multi-target vehicles and assign different frames to different frames. The target frame between them is re-identified and matched to realize the re-tracking of the vehicle within the field of view of the single video camera. A schematic diagram of the re-identification results is shown in FIG. 13 .

The Deepsort algorithm was proposed by Nicolai Wojke et al., which was improved from the Sort algorithm. Sort and Deepsort algorithms are commonly used algorithms in Multiple Object Tracking MOT. Good performance is obtained at frame rates. However, since the Sort algorithm ignores the surface features of the detected object, it is only accurate when the uncertainty of the state estimation of the object is low. Train on large-scale datasets and extract features that increase the network's robustness to loss and obstacles.

In the Deepsort algorithm, a detector and a tracker are constructed for object re-identification. The detector is input from the target detection frame, and the tracker is used for target tracking and re-identification. The input of the Deepsort algorithm includes: target detection frame, target detection confidence, target features (image features and motion features). The target detection confidence is mainly used to screen a part of the detection frame, and the target detection frame and target features are used for tracker construction and subsequent tracking calculation. In the algorithm prediction module, the Kalman filter is used to predict the tracker, and the model constructed by the Kalman filter is a uniform motion and linear observation model. The algorithm update module includes matching, tracker update and target feature set update.

Cascade matching algorithm: A tracker is assigned to each detector, and each tracker is set with a timer parameter. If the tracker is matched and updated, the timer parameter is reset to 0, otherwise it is incremented by 1 unit. In cascading matching, the trackers will be sorted according to the timer parameters. Smaller parameters will be matched first, and larger parameters will be matched later. That is, the tracker that matches the first frame in the previous frame is given high priority, and the tracker that has not been matched for several frames has a lower priority.

Feature comparison: Mahalanobis distance and cosine distance are introduced for comparison between motion information and image information. Mahalanobis distance avoids the risk of different variances of data features in Euclidean distance. A covariance matrix is added in the calculation to normalize the variance, so that the distance value is more in line with the data features and practical significance. Mahalanobis distance is a distance measure in the measure of difference, and different from Mahalanobis distance, cosine distance is a similarity measure. The former is based on location, while the latter is based on direction. When using cosine distance, it can be used to measure the differences between different individuals in dimensions. Combining the two types of feature distances can make the calculation results relatively comprehensive to measure the difference of different target features.

Feature matching: Based on the Mahalanobis distance and cosine distance features, target features are matched between different image frames to complete the process of target re-identification (ID transfer).

2) Cross-video camera vehicle re-tracking: The overall logic flow chart of vehicle cross-video camera re-tracking is shown in FIG. 14 . When the target vehicle is lost within the field of view of the camera 1, the system records the vehicle image features and geospatial features of the n frames (n≥10) before the loss, and transmits it to the lost vehicle database. In addition to the world geographic coordinates, the geospatial features include Vehicle speed and driving direction characteristics collected by millimeter-wave radar. After the vehicle is lost, linear fitting, Kalman filtering and other methods are used to calculate the predicted speed of the vehicle, and the predicted position of the vehicle is calculated according to the driving direction before the vehicle is lost, and the lost database is dynamically updated. When the target vehicle enters the field of view of camera 2, the system captures the features of the first n frames (n≥10) when the vehicle target appears and performs feature matching in the vehicle loss database, and assigns IDs to vehicles with high similarity to realize re-tracking of vehicles . If there is no matching vehicle in the vehicle lost database, a new ID is assigned to the target vehicle. In the present invention, the license plate information is not necessary. If the environmental factors are good (illumination, shooting angle, etc.), the camera can recognize part of the license plate information (such as key characters, license plate color, etc.) The confidence of the recognition result of the frame is given its dynamic weight, which can be used as the verification basis for multi-target vehicle re-tracking. Finally, cross-video camera re-tracking of multi-target vehicles is achieved, and the schematic results are shown in Figure 15.

Claims (11)

1. A multi-target vehicle detection and re-identification method based on Raivision fusion, using at least two video cameras and at least two millimeter-wave radars pre-arranged, including the following steps:

   1) Calibration:

   1.1) Video camera calibration;

   1.2) Coordinate calibration of target detection frame;

   1.3) Millimeter wave radar calibration;

   2) Data acquisition:

   2.1) Use the video camera deployed on the roadside to obtain the video image data within the field of view of the camera;

   2.2) Use the millimeter-wave radar deployed on the roadside to obtain multi-target vehicle data within the radar field of view;

   3) Vehicle target recognition and feature extraction: After receiving the video image data and the multi-target vehicle data, the central computing server performs vehicle target recognition and feature extraction;

   3.1) Feature extraction of video image data;

   3.2) Feature extraction of millimeter wave radar data;

   4) Feature matching and data fusion: The multi-target vehicle geospatial features obtained by the video camera and the millimeter wave radar are matched, and the on-board RTK positioning data (D ₀ ) of the experimental vehicle is used as the relative truth value, and the multi-sensor data fusion method is used to analyze the video. The positioning data (D ₁ ) obtained by the camera is fused with the positioning data (D ₂ ) obtained by the millimeter-wave radar;

   5) Vehicle re-tracking: includes the following two situations:

   5.1) Re-tracking of vehicles within the field of view of a single video camera;

   5.2) Vehicle re-tracking across video cameras.
2. The method according to claim 1, wherein the video camera adopts a directional camera, and the millimeter-wave radar adopts a long-distance radar in a frequency band of 79 GHz; the video camera and the millimeter-wave radar are arranged on the same roadside pole position, so that the latitude and longitude coordinates of the two are consistent.
3. The method of claim 1, wherein the video camera and the millimeter-wave radar are arranged by one of the following schemes:

   Scheme A: The millimeter-wave radar and the video camera are arranged on different roadside poles, and the overlap rate of the two fields of view is greater than 90%;

   Scheme B: The millimeter-wave radar and the video camera are arranged on the same gantry above the road, and the overlap rate of the two fields of view is greater than 90%;

   Scheme C: The millimeter-wave radar and the video camera are arranged on different gantry above the road, and the overlap rate of the two fields of view is greater than 90%.
4. The method according to claim 2 or 3, wherein the detection range of the video camera and the millimeter-wave radar is adjusted according to the installation height and angle: the installation height is 6 meters, and the downward tilt angle is 10 meters. °; Both the video camera and the millimeter-wave radar collect data at a frequency of 25 Hz, and access the central server for data storage and processing.
5. The method of claim 1, wherein the target detection frame coordinate calibration is to establish a mapping relationship between the pixel coordinates of the vehicle target detection frame in the camera image and the vehicle world geographic coordinates; the target detection frame coordinate calibration is divided into the following three a situation:

   1.2.1) When the video camera is located directly above the road, the pixel coordinates of the vehicle target detection frame refer to the pixel coordinates of the midpoint pixel of the bottom edge of the detection frame;

   1.2.2) When the video camera is located on the road to the upper left, the pixel coordinates of the vehicle target detection frame refer to the pixel coordinates of the vertex pixels at the lower right of the detection frame;

   1.2.3) When the video camera is located on the road to the upper right, the pixel coordinates of the vehicle target detection frame refer to the pixel coordinates of the lower left vertex pixel of the detection frame.
6. The method of claim 1, wherein the video image data includes the following target vehicle image characteristics: vehicle color, model, size, and geospatial characteristics.
7. The method of claim 4, wherein the video image data further includes license plate information.
8. The method of claim 1, wherein the millimeter-wave radar data includes the following geospatial features of the target vehicle: world geographic coordinates, speed, and heading angle.
9. The method of claim 1, wherein the data fusion adopts a Kalman filtering method.
10. The method according to claim 1, wherein the re-tracking of vehicles within the field of view of the single video camera comprises the following steps: 5.1.1) Use a pre-trained deep learning target detection algorithm to identify vehicles within the field of view of the single video camera and mark them target box;

    5.1.2) Use the Deepsort multi-target tracking algorithm to assign IDs to multi-target vehicles and re-identify and match the target frames between different image frames to achieve re-tracking of vehicles within the field of view of a single video camera.
11. The method of claim 1, wherein the cross-video camera vehicle re-tracking comprises the steps of:

    5.2.1) When a target vehicle is lost in the field of view of camera 1, the system records the image characteristics and geospatial characteristics of the vehicle before it is lost, and transmits it to the lost vehicle database;

    5.2.2) When the target vehicle enters the field of view of the camera 2, the system captures the vehicle features and performs feature matching in the vehicle lost database, and re-assigns the ID to the vehicle with high similarity to realize the re-tracking of the vehicle;

    5.2.3) If no matching vehicle is found in the vehicle lost database, assign a new ID to the target vehicle.

Images