WO2021259344A1 - Vehicle detection method and device, vehicle, and storage medium - Google Patents

Abstract

A vehicle detection method and device, a vehicle, and a storage medium. The vehicle detection method comprises: on the present vehicle, acquiring radar data by means of a millimeter-wave radar and acquiring an image by means of a camera (S101); establishing a dangerous area of the present vehicle (S102); determining, according to the radar data, a target vehicle located in the dangerous area and first detection data of the target vehicle (S103); inputting the image into a target detection model to obtain second detection data of the target vehicle (S104); and on the basis of a projection of a radar coordinate system of the millimeter-wave radar onto a pixel coordinate system of the image, fusing the first detection data and the second detection data to obtain detection data of the target vehicle (S105).

Description

Vehicle detection method, device, vehicle and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office with an application number of 202010591915.X on June 24, 2020, and the entire content of this application is incorporated into this application by reference.

Technical field

This application relates to the field of vehicle technology, for example, to a vehicle detection method, device, vehicle, and storage medium.

Background technique

With the development of autonomous driving technology, autonomous driving is slowly integrating into people's lives. Equipping with driving assistance functions has become one of the important references when people choose to buy a vehicle.

In autonomous driving technology, environment perception technology is the foundation and prerequisite for the development of intelligent driving vehicles. The sensors commonly used for environment perception include millimeter wave radar, camera, and lidar. However, it is difficult for a single sensor to meet the requirements of intelligent driving for all-weather, multi-information, high-precision, and high-robustness environmental perception. Millimeter wave radar is not susceptible to weather factors such as rain, snow, and fog. It is highly adaptable to the environment and has great advantages in extracting the depth of field information and speed information of the target. Camera-based visual recognition is used in target recognition and classification. It has a great advantage and can perceive real spatial information. Therefore, millimeter-wave radars and cameras are often used to perceive and integrate environmental information to assist autonomous driving. However, in the process of sensing environmental information based on millimeter-wave radars and cameras, the amount of fusion data is large, the efficiency is low, and the real-time performance of the perceived environmental information is poor. , Is not conducive to assisted automatic driving.

Summary of the invention

This application provides a vehicle detection method, device, vehicle, and storage medium to solve the problem that in the process of sensing environmental information based on millimeter wave radar and camera, the amount of fusion data is large, the efficiency is low, and the real-time performance of the environmental information obtained by perception is poor, which is not conducive to The problem of assisting autonomous driving.

A vehicle detection method is provided, which is applied to a vehicle equipped with a millimeter wave radar and a camera, including:

Collect radar data through millimeter-wave radar and collect images through cameras on the vehicle;

Establish the dangerous area of the own vehicle;

Determining a target vehicle located in the dangerous area and the first detection data of the target vehicle according to the radar data;

Inputting the image into a pre-trained target detection model to obtain second detection data of the target vehicle;

Based on the projection of the radar coordinate system of the millimeter wave radar to the pixel coordinate system of the image, the detection data of the target vehicle is obtained by fusing the first detection data and the second detection data.

Optionally, the establishing the dangerous area of the own vehicle includes:

A rectangular dangerous area is established in the driving direction of the host vehicle and in front of the host vehicle, the width of the dangerous area is M widths of the host vehicle, and the length of the dangerous area is all For the N vehicle lengths of the host vehicle, the width center line of the dangerous area coincides with the vehicle width center line of the host vehicle.

Optionally, the determining the target vehicle located in the dangerous area and the first detection data of the target vehicle according to the radar data includes:

Performing Kalman filtering on the radar data to obtain filtered radar data;

Determine the target detected by the millimeter wave radar based on the filtered radar data;

Removing false targets from the targets detected by the millimeter wave radar to obtain the vehicle detected by the millimeter wave radar;

Determine a vehicle located in the dangerous area from the vehicles detected by the millimeter wave radar and use the determined vehicle as a target vehicle;

Acquire first detection data of the target vehicle, where the first detection data includes at least one of radar coordinates, speed, acceleration, vehicle width, and vehicle length of the target vehicle in the radar coordinate system.

Optionally, the removing false targets from the targets detected by the millimeter wave radar to obtain the vehicle detected by the millimeter wave radar includes:

The false target is determined by the following formula:

Wherein, n-millimeter-wave radar is a sample number, T _r millimeter-wave radar sampling period, x (n + 1), y (n + 1) for the longitudinal and transverse coordinates of the coordinate sampling point n + 1 of the target, x(n), y(n) are the longitudinal and lateral coordinates of the target at n sampling points, v _x (n) and v _y (n) are the longitudinal and lateral speeds of the target at n sampling points, a _x And a _y are the maximum values of longitudinal acceleration and lateral acceleration,

Is the longitudinal speed difference,

Is the lateral velocity difference, a _x T _r is the longitudinal velocity difference threshold, and a _y T _r is the lateral velocity difference threshold. When the above formula is established, the target is a real target, otherwise the target is a false target;

The vehicle detected by the millimeter wave radar is obtained by removing the false target from the target detected by the millimeter wave radar.

Optionally, the first detection data includes radar coordinates of the target vehicle in the radar coordinate system, and the second detection data includes second pixel coordinates of the target vehicle in the pixel coordinate system, The fusion of the first detection data and the second detection data to obtain the detection data of the target vehicle based on the projection of the radar coordinate system of the millimeter wave radar to the pixel coordinate system of the image includes:

Projecting the radar coordinates from the radar coordinate system into the pixel coordinate system to obtain the target pixel with the geometric center of the target vehicle in the image;

Establishing a region of interest with the target pixel as the center;

The radar coordinates are corrected by the second pixel coordinates of the vehicle contained in the region of interest to obtain the final coordinates of the target vehicle.

Optionally, the projecting the radar coordinates from the radar coordinate system into the pixel coordinate system to obtain the target pixel with the geometric center of the target vehicle in the image includes:

Project the radar coordinates from the radar coordinate system to the pixel coordinate system to obtain the first pixel coordinate by the following conversion formula from the radar coordinate system to the pixel coordinate system:

H _c is the height of the installation position of the camera from the ground, α is the elevation angle of the camera installation, (x _r , y _r , z _r ) is the radar coordinates of the target vehicle in the radar coordinate system, and W _r is the installation of millimeter wave radar The distance between the position and the longitudinal symmetry plane of the vehicle, H _r is the height of the millimeter-wave radar installation position from the ground, and ΔL is the absolute value of the difference between the installation distance of the millimeter-wave radar and the camera in the driving direction of the vehicle. (u, v) is the first pixel coordinate of the target pixel of the target vehicle in the image, (u ₀ , v ₀ ) is the pixel coordinate of the center point of the image, f _x and f _y are the equivalent focal lengths of the camera;

It is determined that the pixel point corresponding to the first pixel coordinate is the target pixel point of the geometric center of the target vehicle in the image.

Optionally, the correcting the radar coordinates to obtain the final coordinates of the target vehicle by using the second pixel coordinates of the vehicle included in the region of interest includes:

The radar coordinates are corrected by the following formula to obtain the final coordinates of the target vehicle:

(x _a , y _a , z _a ) are the corrected final coordinates, (u _r , v _r ) are the second pixel coordinates of the target vehicle contained in the region of interest.

A vehicle detection device is also provided, which is applied to a vehicle equipped with a millimeter wave radar and a camera, and includes:

The radar data and image acquisition module is set to collect radar data through millimeter wave radar and images through the camera on the vehicle;

The dangerous area establishment module is set to establish the dangerous area of the vehicle;

A first detection data determination module, configured to determine a target vehicle located in the dangerous area and first detection data of the target vehicle according to the radar data;

A second detection data acquisition module, configured to input the image into a pre-trained target detection model to obtain second detection data of the target vehicle;

The fusion module is configured to fuse the first detection data and the second detection data to obtain the detection data of the target vehicle based on the projection of the radar coordinate system of the millimeter wave radar to the pixel coordinate system of the image.

A vehicle is also provided, including:

A millimeter wave radar and a camera, the millimeter wave radar is set to collect radar data, and the camera is set to collect images;

One or more processors;

Storage device, set to store one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the vehicle detection method according to any embodiment of the present application.

A computer-readable storage medium is also provided, which stores a computer program, and when the computer program is executed by a processor, the vehicle detection method according to any one of the embodiments of the present application is implemented.

Description of the drawings

FIG. 1 is a flow chart of the steps of a vehicle detection method provided by Embodiment 1 of the present application;

2A is a flowchart of the steps of a vehicle detection method provided in the second embodiment of the present application;

2B is a schematic diagram of the host vehicle and the target vehicle in an embodiment of the present application;

2C is a schematic diagram of the training and application of the target detection model in an embodiment of the present application;

FIG. 3 is a structural block diagram of a vehicle detection device provided in Embodiment 3 of the present application;

Fig. 4 is a schematic diagram of a vehicle provided in the fourth embodiment of the present application.

detailed description

The application will be described below with reference to the drawings and embodiments. The embodiments described here are only used to explain the application, but not to limit the application. For ease of description, the drawings only show a part of the structure related to the present application, but not all of the structure. In the case of no conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

Example one

Figure 1 is a flow chart of the steps of a vehicle detection method provided in the first embodiment of the application. The embodiment of the application can be applied to a situation where the vehicle detects surrounding vehicles during driving. The vehicle detection device may be implemented by hardware or software and integrated in the vehicle provided in the embodiment of the present application. As shown in FIG. 1, the vehicle detection method of the embodiment of the present application may include the following steps.

S101. Collect radar data through millimeter wave radar and collect images through a camera on the own vehicle.

In the embodiments of the present application, the host vehicle may be a vehicle that requires assisted driving. The host vehicle needs to perceive the surrounding environment through sensors mounted on the vehicle body, and obtain data of vehicles traveling in the surrounding environment to perform assisted driving through the data. decision making.

The vehicle can be equipped with millimeter-wave radar and camera. The millimeter-wave radar transmits radar waves to the front of the driving direction and receives the reflected radar waves. The transmitted radar waves and the received radar waves are mixed to obtain the radar signal as a radar. At the same time, the camera takes pictures in the direction of travel to obtain images. In practical applications, millimeter-wave radar and camera can collect data at the same sampling rate or at different sampling rates. When collecting data at different sampling rates, use the sampling time of the smaller sampling rate as the time. The benchmark acquires radar data and images at a time, that is, after acquiring one frame of data at a sampling time with a smaller sampling rate, acquire the data collected closest to the sampling time with a larger sampling rate as the sampling data.

Exemplarily, the camera collects 25 frames of images per second, and the millimeter wave radar collects 35 frames of radar data per second. The sampling rate of the camera is relatively small. When the camera collects one frame of images at a time, it is determined that the millimeter wave radar is closest to that time. The radar data collected by the millimeter wave radar and the image collected by the camera are obtained, and the target vehicle detected by the radar data and the target vehicle recognized by the image are fused to obtain the final detection data of the target vehicle.

S102. Establish a dangerous area of the own vehicle.

Compared with the vehicle, the dangerous area is a potentially dangerous area during driving. For example, it can be an area close to the vehicle in front of the vehicle in the driving direction of the vehicle. Move the distance to determine. Optionally, the dangerous area may be a rectangular area, the width of the rectangular area may be the width of two vehicle bodies, and the length may be N vehicle body lengths or determined according to the braking distance of the vehicle. The center line of the direction coincides with the center line of the vehicle body in the longitudinal direction of the vehicle body.

S103. Determine a target vehicle located in the dangerous area and first detection data of the target vehicle according to the radar data.

The radar data can be the radar signal after the transmitted radar wave and the received radar wave are mixed. After the radar signal is detected and detected, multiple targets in front of the vehicle detected by the millimeter wave radar can be obtained, and multiple targets can be obtained relative to each other. The radar coordinates, distance, target speed, acceleration, width and length of the first detection data of the vehicle. Furthermore, it can be determined whether the target is located in the dangerous area according to the radar coordinates of the multiple targets, and if so, the target vehicle located in the dangerous area is determined and the first detection data of the target vehicle is obtained.

S104. Input the image into a pre-trained target detection model to obtain second detection data of the target vehicle.

In the embodiments of the present application, the target detection model can be trained offline. The target detection model can recognize the types of multiple targets in the input image and the pixel coordinates of the geometric center of the target in the image. The target detection model can be based on Haar The target detection model combined with -like features and Adaboost can also be neural networks such as Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Deep Neural Network (DNN), etc. , The embodiment of the present application does not impose restrictions on the target detection model.

After the target detection model is trained offline, the image collected by the camera can be input into the target detection model to obtain the second detection data of multiple target vehicles in the image. The second detection data may include the type of the target vehicle and the information of the target vehicle. The pixel coordinates of the geometric center in the image, etc.

S105. Based on the projection of the radar coordinate system of the millimeter wave radar to the image coordinate system of the image, fusion of the first detection data and the second detection data to obtain the detection data of the target vehicle.

In an optional embodiment of the present application, the relative position of the camera and the millimeter wave radar, the equivalent focal length of the camera, the pitch angle of the camera after installation, the height of the camera and the millimeter wave radar to the ground, and the pixels of the center point of the image can be used. Coordinates, etc., establish a conversion formula from the radar coordinate system of the millimeter-wave radar to the image coordinate system of the image. Through this conversion formula, the radar coordinates of the geometric center of the target vehicle in the radar coordinate system are converted to the image coordinate system of the image to obtain the target vehicle The geometric center of the corresponding target pixel in the image, and the target pixel is used to establish a region of interest. The target identified in the region of interest and the target vehicle detected by the millimeter wave radar belong to the same target vehicle, so that The radar coordinates are corrected by the pixel coordinates of the geometric center of the target vehicle identified by the image to obtain the final detection data of the target vehicle, for example, the lateral distance, longitudinal distance, speed, acceleration, and target of the target vehicle relative to the vehicle. Vehicle (Identifier, ID), vehicle width, vehicle length, etc., output the detection data of the target vehicle to the driving assistance system, so that the driving assistance system makes driving assistance decisions based on the detection data of the target vehicle.

In the embodiment of the application, the radar data and the image collected by the camera are collected by the millimeter wave radar on the own vehicle to establish the dangerous area of the own vehicle, and the target vehicle located in the dangerous area and the first detection data of the target vehicle are determined according to the radar data; The second detection data of the target vehicle is obtained by inputting the image into the pre-trained target detection model; based on the projection of the radar coordinate system of the millimeter wave radar to the pixel coordinate system of the image, the first detection data and the second detection data are merged to obtain the target vehicle's Test data. The embodiment of the application realizes that only the first detection data and the second detection data of the target vehicle in the dangerous area are fused to obtain the target detection data of the target vehicle, which greatly reduces the amount of data of the vehicle that needs to be fused, and improves the data fusion Efficiency, the detection data of the target vehicle finally obtained has high real-time performance, which is conducive to the timely decision-making of the automatic driving assistance system.

Example two

2A is a flow chart of the steps of a vehicle detection method provided in the second embodiment of the application, and the embodiment of the present application is described on the basis of the foregoing first embodiment. As shown in FIG. 2A, the vehicle detection method of the embodiment of the present application may include the following steps.

S201. Collect radar data through millimeter wave radar and collect images through a camera on the own vehicle.

As shown in Figure 2B, a millimeter-wave radar can be installed at the air intake grid in front of the vehicle, and a camera can be installed inside the windshield. Both the millimeter-wave radar and the camera are installed on the longitudinal symmetry plane F of the vehicle. The millimeter-wave radar can also be offset with respect to the longitudinal symmetry plane F, and both the camera and the millimeter-wave radar collect data in the driving direction of the vehicle.

S202. Establish a rectangular dangerous area in the driving direction of the host vehicle and in front of the host vehicle, the width of the dangerous area is M widths of the host vehicle, and the length of the dangerous area Is the N vehicle lengths of the host vehicle, and the width center line of the dangerous area coincides with the vehicle width center line of the host vehicle.

As shown in FIG. 2B, in an optional embodiment of the present application, the dangerous area is a rectangular area, and the rectangular area is located in front of the driving direction of the vehicle. As shown in Figure 2B, the dangerous area A is in front of the driving direction of the vehicle. The width of the dangerous area is 2d and the length is L. The width center line of the dangerous area A coincides with the longitudinal symmetry plane F of the vehicle, d For the vehicle width of the own vehicle, L is equal to the length of the body of the N own vehicle, for example, N may be equal to 3. L can also be equal to the braking distance of the vehicle. The braking distance can be determined according to the current driving speed of the vehicle. In practical applications, the dangerous area can also be set according to the actual situation, which is not limited in the embodiment of the application. .

S203: Perform Kalman filtering on the radar data to obtain filtered radar data.

In the embodiment of the present application, the radar data may be a mixed radar signal of the transmitted radar wave and the received radar wave. The mixed radar signal is interfered by the environment and internal circuits and has noise, and the radar signal can be filtered by Kalman filtering. , In order to remove the noise and interference in the radar signal, and obtain the filtered radar data.

S204: Determine a target detected by the millimeter wave radar based on the filtered radar data.

The filtered radar signal can be detected and processed to obtain all the targets detected by the millimeter-wave radar. The targets detected by the millimeter-wave radar may include false targets caused by the disturbance of the road surface and the moving vehicle. In application, the target detection algorithm of the millimeter wave radar can be used to determine the target detected by the millimeter wave radar based on the radar data. In the embodiment of the present application, the determination of the target detected by the millimeter wave radar based on the radar data will not be described in detail.

S205. Remove false targets from the targets detected by the millimeter wave radar to obtain a vehicle detected by the millimeter wave radar.

In an optional embodiment of the present application, the false target can be determined by the following formula:

The above equation, n is the sample number millimeter-wave radar, T _r is the sampling period of a millimeter wave radar, x (n + 1), y (n + 1) in the longitudinal direction of the coordinate sampling point n + 1 of the target (OrXr ) And horizontal coordinates (OrYr), x(n), y(n) are the longitudinal and horizontal coordinates of the target at n sampling points, v _x (n) and v _y (n) are the target’s at n sampling points Longitudinal velocity and lateral velocity, a _x and a _y are the maximum values of longitudinal and lateral acceleration,

Is the longitudinal speed difference,

Is the lateral velocity difference, a _x T _r is the longitudinal velocity difference threshold, a _y T _r is the lateral velocity difference threshold, the target is the real target when the above formula is established, otherwise the target is a false target, that is, the lateral velocity difference is greater than the lateral velocity difference threshold, When the longitudinal speed difference is greater than the longitudinal speed difference threshold, the target is a false target.

After determining the false targets detected by the millimeter-wave radar, the false targets can be removed from the targets detected by the millimeter-wave radar to obtain the vehicles detected by the millimeter-wave radar, thus avoiding the image data fusion of the false targets and reducing the fusion The amount of data can improve the efficiency of vehicle detection.

S206. Determine a vehicle located in the dangerous area from the vehicles detected by the millimeter wave radar and use the determined vehicle as a target vehicle.

When the millimeter-wave radar detects a vehicle, it can also obtain the radar coordinates of the geometric center of the detected vehicle, and the horizontal and vertical distances of the geometric center of the detected vehicle relative to the vehicle. The longitudinal distance determines whether the vehicle is in the danger zone. Exemplarily, it can be determined whether the radar coordinates of the detected vehicle are located in the dangerous area, or whether the vehicle is located in the dangerous area is determined by the horizontal and longitudinal distances of the detected vehicle relative to the vehicle of the vehicle. It is determined that the vehicle is a vehicle located in a dangerous area, and the vehicle is regarded as the target vehicle. As shown in Figure 2B, for millimeter-wave radar, the detection range of millimeter-wave radar is relatively wide, and the detection distance can usually reach 100 meters. On roads with dense traffic, millimeter-wave radar can detect more vehicles. The target vehicle located in the dangerous area is determined from all detected vehicles. As shown in Figure 2B, the dangerous area includes target vehicle 1 and target vehicle 2, which greatly reduces the number of target vehicles, reduces the amount of fusion data, and improves Improve the efficiency of vehicle detection.

S207. Acquire first detection data of the target vehicle, where the first detection data includes at least one of radar coordinates, speed, acceleration, vehicle width, and vehicle length of the target vehicle in the radar coordinate system.

In the embodiments of the present application, the millimeter wave radar can not only detect the target vehicle, but also obtain the radar coordinates of the target vehicle, the horizontal and longitudinal distance relative to the own vehicle, the speed, acceleration, vehicle width, and length of the target vehicle, etc. Regarding these data as the first detection data, the first detection data of the above-mentioned target vehicle can be obtained by any kind of millimeter wave radar detection target algorithm, which will not be described in detail here.

S208. Input the image into a pre-trained target detection model to obtain second detection data of the target vehicle, where the second detection data includes second pixel coordinates of the geometric center of the target vehicle.

In an optional embodiment of the present application, the target detection model may be a target detection model based on a combination of Haar-like features and Adaboost algorithm. As shown in FIG. 2C, target detection model training based on a combination of Haar-like features and Adaboost algorithm And the application process is as follows.

In the offline training phase, collect vehicle sample images and non-vehicle sample images, normalize and grayscale the vehicle sample images and non-vehicle sample images, and perform features on the processed images based on the Haar-like features of the integral map Extract and input the extracted features into a cascaded classifier based on the Adaboost algorithm to iteratively train the classifier to obtain the final cascaded classifier.

In the online recognition phase, the camera detects the sample to be detected in real time, such as the image collected by the camera in the embodiment of this application, extracts the features of the image collected by the camera based on the Haar-like features of the integral map, and inputs the extracted features into training A good cascade classifier recognizes the target in the image, and obtains the target category and the pixel coordinates of the geometric center of the target. In the embodiment of the present application, the recognized target is a car, a bus, a truck, a motorcycle, a bicycle, etc.

In the embodiment of the present application, after the image collected by the camera is input into the trained target detection model, the target detection model can output the types of multiple vehicles in the image and the pixel coordinates of the geometric centers of the vehicles.

S209: Project the radar coordinates from the radar coordinate system to the pixel coordinate system to obtain the target pixel point of the geometric center of the target vehicle in the image.

The radar coordinates of the target vehicle can be converted to camera coordinates through the following conversion formula:

As shown in Figure 2B, the camera is installed on the longitudinal symmetry plane F of the vehicle. For example, the camera is fixed in the windshield of the vehicle or the upper part of the front air intake grille, and the horizontal angle and yaw angle of the camera are 0°, The pitch angle can be determined according to the front detection range of the vehicle; the millimeter wave radar is installed on the longitudinal symmetry plane F of the vehicle, or installed at a certain distance from the longitudinal symmetry plane F. For example, the millimeter wave radar is installed below At the intake grille, the horizontal angle, yaw angle, and pitch angle of the millimeter-wave radar installation are all 0°. According to the installation position of the camera and millimeter wave radar, the coordinate system of the camera and millimeter wave radar can be established.

As shown in Figure 2B, Oc-Xc-Yc-Zc is the camera coordinate system, and Or-Xr-Yr-Zr is the radar coordinate system of the millimeter wave radar. Among them, Oc is the origin of the camera coordinate system, that is, the optical center of the camera, the OcXc axis is perpendicular to the longitudinal symmetry plane F of the own vehicle and points to the right side of the direction of travel of the own vehicle, and the OcYc axis (not shown) is located in the own vehicle The longitudinal symmetry plane F of the vehicle is perpendicular to the optical axis of the camera, and the OcZc axis coincides with the optical axis of the camera. Or-Xr-Yr-Zr is the radar coordinate system, where Or is the geometric center of the millimeter-wave radar's working surface, the OrXr axis is perpendicular to the millimeter-wave radar's working surface and points to the driving direction of the vehicle, and the OrYr axis is perpendicular to the vehicle. The longitudinal symmetry plane F of the vehicle points to the left side of the forward direction of the vehicle, and the OrZr axis (not shown) is perpendicular to the ground and points to the roof of the vehicle.

In the above-mentioned conversion formula from radar coordinates to camera coordinates, H _c is the height of the installation position of the camera from the ground, α is the pitch angle of the camera installation, and (x _r , y _r , z _r ) is the target vehicle in the radar coordinate system Radar coordinates, W _r is the distance between the installation position of the millimeter-wave radar and the longitudinal symmetry plane F of the vehicle, H _r is the height of the installation position of the millimeter-wave radar from the ground, and ΔL is the distance between the millimeter-wave radar and the camera along the vehicle. The absolute value of the difference between the installation distances in the driving direction of, (x _c , y _c , z _c ) is the camera coordinate of the target vehicle in the camera coordinate system.

Convert the camera coordinates of the target vehicle to the first pixel coordinate in the pixel coordinate system, as follows:

In the above formula (2), (u, v) is the first pixel coordinate of the target pixel of the target vehicle in the image, (u ₀ , v ₀ ) is the pixel coordinate of the center point of the image, f _x and f _y are The equivalent focal length of the camera.

Combining the above formulas (1) and (2), the conversion formula from radar coordinate system to pixel coordinate system is obtained as follows:

When the radar coordinates of the geometric center of the target vehicle are obtained, the radar coordinates of the center point of the target vehicle can be projected to the pixel coordinates of the image through the above formula (3), and the pixel point corresponding to the pixel coordinate is the center point of the target vehicle The corresponding target pixel.

S210: Establish a region of interest with the target pixel as the center.

Taking the target pixel of the target vehicle in the image as the center, the width of the target vehicle as the width of the rectangle, and the length of the target vehicle as the length of the rectangle, a rectangular area is established as the region of interest.

S211: Correct the radar coordinates by using the second pixel coordinates of the vehicles included in the region of interest to obtain the final coordinates of the target vehicle.

In practical applications, the millimeter-wave radar detects the radar coordinates of the geometric center of the target vehicle. The radar coordinates are projected into the pixel coordinate system to obtain a first pixel coordinate. At the same time, the image is input into the pre-trained target detection model. The second pixel coordinates of the geometric center of the target vehicle are obtained, and the position of the target vehicle can be corrected through the conversion of the second pixel coordinates to the radar coordinate system.

The radar coordinates can be corrected by the following formula to obtain the final coordinates of the target vehicle.

_{Where, (x a, y a,} z a) as the final corrected coordinates, (u _r, v _r) is the target pixel coordinate by the second detection pattern to obtain the geometric center of the target vehicle.

After obtaining the final coordinates of the target vehicle, the final coordinates, the vehicle width and length detected by the millimeter wave radar, the distance relative to the vehicle, speed, acceleration, etc. can be output to the driving assistance system as the final detection data, so that the assistance The driving system makes auxiliary driving decisions based on the detection data of the target vehicle.

In the embodiment of this application, radar data is collected by millimeter wave radar and images are collected on the vehicle, and a rectangular dangerous area is established in the driving direction of the vehicle and in front of the vehicle, and the radar data is filtered to obtain the millimeter wave. The target detected by the radar, after filtering out false targets from the detected targets, determine the target vehicle located in the dangerous area, obtain the first detection data of the target vehicle and input the image into the target detection model to obtain the second detection data, The radar coordinates of the geometric center of the target vehicle in the first detection data are projected into the image to obtain the target pixel, and the area of interest is established based on the target pixel. The coordinates are modified to obtain the final coordinates of the target vehicle, and the detection data including the final coordinates are output. The fusion of radar data and camera vision data by filtering false targets, establishing a dangerous area and identifying target vehicles located in the dangerous area greatly reduces the number of vehicles that need to be fused, reduces the amount of data processing, and improves the vehicle detection effect .

Example three

FIG. 3 is a structural block diagram of a vehicle detection device provided in the third embodiment of the present application. As shown in FIG. 3, the vehicle detection device of the embodiment of the present application is applied to a vehicle equipped with a millimeter wave radar and a camera, and may include the following Module.

The radar data and image acquisition module 301 is configured to collect radar data through millimeter wave radar and collect images through a camera on the vehicle;

The dangerous area establishment module 302 is configured to establish the dangerous area of the own vehicle;

The first detection data determining module 303 is configured to determine the target vehicle located in the dangerous area and the first detection data of the target vehicle according to the radar data;

The second detection data acquisition module 304 is configured to input the image into a pre-trained target detection model to obtain second detection data of the target vehicle;

The fusion module 305 is configured to merge the first detection data and the second detection data to obtain the detection data of the target vehicle based on the projection of the radar coordinate system of the millimeter wave radar to the pixel coordinate system of the image.

Optionally, the dangerous area establishing module 302 includes: a dangerous area establishing sub-module.

The dangerous area establishment sub-module is set to establish a rectangular dangerous area in the driving direction of the host vehicle and in front of the host vehicle, and the width of the dangerous area is M vehicle widths of the host vehicle, The length of the dangerous area is N vehicle lengths of the host vehicle, and the width center line of the dangerous area coincides with the vehicle width center line of the host vehicle.

Optionally, the first detection data determining module 303 includes the following modules.

A filtering sub-module, configured to perform Kalman filtering on the radar data to obtain filtered radar data;

A target determination sub-module configured to determine the target detected by the millimeter wave radar based on the filtered radar data;

A false target removal sub-module, configured to remove false targets from the targets detected by the millimeter wave radar to obtain the vehicle detected by the millimeter wave radar;

The target vehicle determination sub-module is configured to determine a vehicle located in the dangerous area from the vehicles detected by the millimeter wave radar and use the determined vehicle as a target vehicle;

The first detection data acquisition sub-module is configured to acquire first detection data of the target vehicle, the first detection data including the radar coordinates, speed, acceleration, vehicle width, and vehicle width of the target vehicle in the radar coordinate system. At least one of the captains.

Optionally, the false target removal sub-module includes the following modules.

The false target determination unit is set to determine the false target through the following formula:

Wherein, n-millimeter-wave radar is a sample number, T _r millimeter-wave radar sampling period, x (n + 1), y (n + 1) for the longitudinal and transverse coordinates of the coordinate sampling point n + 1 of the target, x(n), y(n) are the longitudinal and lateral coordinates of the target at n sampling points, v _x (n) and v _y (n) are the longitudinal and lateral speeds of the target at n sampling points, a _x And a _y are the maximum values of longitudinal acceleration and lateral acceleration,

Is the longitudinal speed difference,

Is the lateral velocity difference, a _x T _r is the longitudinal velocity difference threshold, and a _y T _r is the lateral velocity difference threshold. When the above formula is established, the target is a real target, otherwise the target is a false target;

The false target removing unit is configured to remove the false target from the target detected by the millimeter wave radar to obtain the vehicle detected by the millimeter wave radar.

Optionally, the first detection data includes radar coordinates of the target vehicle, the second detection data includes second pixel coordinates of the target vehicle, and the fusion module 305 includes the following modules.

A coordinate projection sub-module, configured to project the radar coordinates from the radar coordinate system into the pixel coordinate system to obtain the target pixel with the geometric center of the target vehicle in the image;

The region of interest establishment sub-module is set to establish a region of interest with the target pixel as the center;

The coordinate correction sub-module is configured to correct the radar coordinates to obtain the final coordinates of the target vehicle by using the second pixel coordinates of the vehicle contained in the region of interest.

Optionally, the coordinate projection sub-module includes the following modules.

The coordinate projection unit is set to project the radar coordinates from the radar coordinate system to the pixel coordinate system to obtain the first pixel coordinate through the following conversion formula from the radar coordinate system to the pixel coordinate system:

H _c is the height of the installation position of the camera from the ground, α is the elevation angle of the camera installation, (x _r , y _r , z _r ) is the radar coordinates of the target vehicle in the radar coordinate system, and W _r is the installation of millimeter wave radar The distance between the position and the longitudinal symmetry plane of the vehicle, H _r is the height of the millimeter-wave radar installation position from the ground, and ΔL is the absolute value of the difference between the installation distance of the millimeter-wave radar and the camera in the driving direction of the vehicle. (u, v) is the first pixel coordinate of the target pixel of the target vehicle in the image, (u ₀ , v ₀ ) is the pixel coordinate of the center point of the image, f _x and f _y are the equivalent focal lengths of the camera;

The target pixel point determining unit is configured to determine the pixel point corresponding to the first pixel coordinate as the target pixel point of the geometric center of the target vehicle in the image.

Optionally, the coordinate correction sub-module includes the following modules.

The coordinate correction unit is set to correct the radar coordinates through the following formula to obtain the final coordinates of the target vehicle:

(x _a , y _a , z _a ) are the corrected final coordinates, (u _r , v _r ) are the second pixel coordinates of the target vehicle contained in the region of interest.

The vehicle detection device provided in the embodiment of the present application can execute the vehicle detection method provided in the first and second embodiments of the present application, and has the functional modules and effects corresponding to the execution method.

Embodiment four

Referring to Fig. 4, there is shown a schematic structural diagram of a vehicle in an example of the present application. As shown in FIG. 4, the vehicle may include: a processor 400, a memory 401, a display screen 402 with a touch function, an input device 403, an output device 404, a communication device 405, a camera 406, and a millimeter wave radar 407. The number of processors 400 in the vehicle may be one or more, and one processor 400 is taken as an example in FIG. 4. The number of memories 401 in the vehicle may be one or more. In FIG. 4, one memory 401 is taken as an example. The processor 400, the memory 401, the display screen 402, the input device 403, the output device 404, the communication device 405, the camera 406 and the millimeter wave radar 407 of the vehicle may be connected by a bus or other means. In Figure 4, the bus connection is taken as an example .

As a computer-readable storage medium, the memory 401 can be configured to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the vehicle detection method described in any embodiment of the present application (for example, the vehicle detection device described above) The radar data and image acquisition module 301, the dangerous area establishment module 302, the first detection data determination module 303, the second detection data acquisition module 304 and the fusion module 305) in the , The storage program area can store the operating device and application programs required by at least one function; the storage data area can store data created according to the use of the device, etc. In addition, the memory 401 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some examples, the memory 401 may include a memory remotely provided with respect to the processor 400, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

The display screen 402 is a display screen 402 with a touch function, and may be a capacitive screen, an electromagnetic screen or an infrared screen. Generally speaking, the display screen 402 is set to display data according to instructions of the processor 400, and is also set to receive touch operations on the display screen 402 and send corresponding signals to the processor 400 or other devices. Optionally, when the display screen 402 is an infrared screen, it also includes an infrared touch frame. The infrared touch frame is arranged around the display screen 402. It can also be set to receive infrared signals and send the infrared signals to the processor. 400 or other equipment.

The communication device 405 is configured to establish a communication connection with other devices, and it may be a wired communication device and/or a wireless communication device.

The input device 403 may be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 404 may include audio equipment such as a speaker. The composition of the input device 403 and the output device 404 can be set according to actual conditions.

The camera 406 is configured to collect images around the vehicle, and the millimeter wave radar 407 is configured to emit radar signals around the vehicle to collect radar data.

The processor 400 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 401, thereby realizing the aforementioned vehicle detection method.

In an embodiment, when the processor 400 executes one or more programs stored in the memory 401, the vehicle detection method provided in the embodiment of the present application can be implemented.

The embodiment of the present application also provides a computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the device, the device can execute the vehicle detection method described in the above-mentioned embodiment.

For the embodiments of the device, the vehicle, and the storage medium, since they are basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the part of the description of the method embodiment.

In the description of this specification, descriptions with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples", etc. mean the features described in conjunction with the embodiments or examples, The structure, material or feature is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above-mentioned terms do not necessarily refer to the same embodiment or example. Moreover, the described features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

Claims (10)

1. A vehicle detection method, which is applied to a vehicle equipped with a millimeter wave radar and a camera, includes:

   Collecting radar data through the millimeter wave radar and collecting images through the camera on the own vehicle;

   Establish the dangerous area of the own vehicle;

   Determining a target vehicle located in the dangerous area and the first detection data of the target vehicle according to the radar data;

   Inputting the image into a pre-trained target detection model to obtain second detection data of the target vehicle;

   Based on the projection of the radar coordinate system of the millimeter wave radar to the pixel coordinate system of the image, the detection data of the target vehicle is obtained by fusing the first detection data and the second detection data.
2. The method according to claim 1, wherein the establishing the dangerous area of the host vehicle comprises:

   A rectangular dangerous area is established in the driving direction of the host vehicle and in front of the host vehicle, the width of the dangerous area is M widths of the host vehicle, and the length of the dangerous area is For the N vehicle lengths of the host vehicle, the width center line of the dangerous area coincides with the vehicle width center line of the host vehicle.
3. The method according to claim 1 or 2, wherein the determining the target vehicle located in the dangerous area and the first detection data of the target vehicle according to the radar data comprises:

   Performing Kalman filtering on the radar data to obtain filtered radar data;

   Determine the target detected by the millimeter wave radar based on the filtered radar data;

   Removing false targets from the targets detected by the millimeter wave radar to obtain the vehicle detected by the millimeter wave radar;

   Determine a vehicle located in the dangerous area from the vehicles detected by the millimeter wave radar and use the determined vehicle as the target vehicle;

   Acquire first detection data of the target vehicle, where the first detection data includes at least one of the following: radar coordinates, speed, acceleration, vehicle width, and vehicle length of the target vehicle in the radar coordinate system.
4. The method according to claim 3, wherein the removing false targets from the targets detected by the millimeter wave radar to obtain the vehicle detected by the millimeter wave radar comprises:

   The false target is determined by the following formula:

   

   Wherein, n is the sample number of the millimeter-wave radar, T _r is the sampling period of the millimeter-wave radar, x (n + 1) and Y (n + 1) to n + 1 at the target sampling point Longitudinal and horizontal coordinates, x(n) and y(n) are the longitudinal and horizontal coordinates of the target at n sampling points, and v _x (n) and v _y (n) are stated at n sampling points The longitudinal velocity and lateral velocity of the target, a _x and a _y are the maximum values of longitudinal acceleration and lateral acceleration,

   

   Is the longitudinal speed difference,

   

   Is the lateral velocity difference, a _x T _r is the longitudinal velocity difference threshold, and a _y T _r is the lateral velocity difference threshold. When the above formula is established, the target is the real target, and when one of the above formulas is not established, The target is the false target;

   The vehicle detected by the millimeter wave radar is obtained by removing the false target from the target detected by the millimeter wave radar.
5. The method according to claim 1, wherein the first detection data includes radar coordinates of the target vehicle in the radar coordinate system, and the second detection data includes the target vehicle in the pixel coordinate system. The second pixel coordinates in the image are based on the projection of the radar coordinate system of the millimeter wave radar to the pixel coordinate system of the image, and the first detection data and the second detection data are merged to obtain the target vehicle's Test data, including:

   Projecting the radar coordinates from the radar coordinate system into the pixel coordinate system to obtain the target pixel point of the geometric center of the target vehicle in the image;

   Establishing a region of interest with the target pixel as the center;

   The radar coordinates are corrected by the second pixel coordinates of the target vehicle contained in the region of interest to obtain the final coordinates of the target vehicle.
6. The method according to claim 5, wherein the projecting the radar coordinates from the radar coordinate system into the pixel coordinate system obtains the target pixel point of the geometric center of the target vehicle in the image ,include:

   Project the radar coordinates from the radar coordinate system to the pixel coordinate system through the following conversion formula from the radar coordinate system to the pixel coordinate system to obtain the first pixel coordinate:

   

   Wherein, H _c is the height of the installation position of the camera from the ground, α is the elevation angle of the camera installation, (x _r , y _r , z _r ) is the radar of the target vehicle in the radar coordinate system The coordinates, W _r is the distance between the installation position of the millimeter wave radar and the longitudinal symmetry plane of the vehicle, H _r is the height of the installation position of the millimeter wave radar from the ground, and ΔL is the distance between the millimeter wave radar and the The absolute value of the difference in the installation distance of the camera along the traveling direction of the host vehicle, (u, v) is the first pixel coordinate of the target pixel of the target vehicle in the image, (u ₀ , V ₀ ) is the pixel coordinates of the center point of the image, f _x and f _y are the equivalent focal lengths of the camera;

   It is determined that the pixel point corresponding to the first pixel coordinate is the target pixel point of the geometric center of the target vehicle in the image.
7. The method according to claim 6, wherein the correcting the radar coordinates to obtain the final coordinates of the target vehicle by using the second pixel coordinates of the vehicle contained in the region of interest comprises:

   The radar coordinates are corrected by the following formula to obtain the final coordinates of the target vehicle:

   

   _{Where, (x a, y a,} z a) as the final corrected coordinates, (u _r, v _r) of the second pixel coordinates of the object of interest contained in the region of the vehicle.
8. A vehicle detection device, which is applied to a vehicle equipped with a millimeter wave radar and a camera, includes:

   A radar data and image acquisition module configured to collect radar data on the host vehicle through the millimeter wave radar and collect images through the camera;

   The dangerous area establishment module is set to establish the dangerous area of the vehicle;

   A first detection data determination module, configured to determine a target vehicle located in the dangerous area and first detection data of the target vehicle according to the radar data;

   A second detection data acquisition module, configured to input the image into a pre-trained target detection model to obtain second detection data of the target vehicle;

   The fusion module is configured to fuse the first detection data and the second detection data to obtain the detection data of the target vehicle based on the projection of the radar coordinate system of the millimeter wave radar to the pixel coordinate system of the image.
9. A vehicle including:

   A millimeter wave radar and a camera, the millimeter wave radar is set to collect radar data, and the camera is set to collect images;

   One or more processors;

   Storage device, set to store one or more programs,

   When the one or more programs are executed by the one or more processors, the one or more processors implement the vehicle detection method according to any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the vehicle detection method according to any one of claims 1-7.

Images