WO2022156276A1

WO2022156276A1 - Target detection method and apparatus

Info

Publication number: WO2022156276A1
Application number: PCT/CN2021/124194
Authority: WO
Inventors: 黄梓亮; 郑永豪; 王灿
Original assignee: 华为技术有限公司
Priority date: 2021-01-22
Filing date: 2021-10-15
Publication date: 2022-07-28
Also published as: CN114779229A

Abstract

Provided in the embodiments of the present application are a target detection method and apparatus. The method comprises: a first vehicle detecting the speed of a second vehicle on the basis of a millimeter wave radar; acquiring a road condition where the first vehicle and the second vehicle are located; determining a first threshold value according to the acquired road condition; then comparing the speed of the second vehicle with the first threshold value; and if the speed of the second vehicle exceeds the first threshold value, the first vehicle determining that the millimeter wave radar is in a state where a target is lost or a state where the target is about to be lost, otherwise, the first vehicle determining that the millimeter wave radar is not in the state where the target is lost or the state where the target is about to be lost. Therefore, under special road conditions (such as a curve or a ramp), a first vehicle can effectively identify, in a timely manner, the situation of losing a target of a millimeter wave radar due to the road condition, thereby preventing the first vehicle from producing an illusion of there being no target obstacle, and further improving the safety and stability of a vehicle during a driving process.

Description

A target detection method and device

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the Chinese patent application with the application number of 202110089080.2 and the application title of "A method and device for target detection" filed with the China Patent Office on January 22, 2021, the entire contents of which are incorporated into this application by reference .

technical field

The present application relates to the technical field of automatic driving, and in particular, to a target detection method and device.

Background technique

With the rapid development of economy and society, autonomous driving technology is gradually becoming mature and landing. However, some functions in special dangerous road conditions, such as the detection and tracking of targets at curves, are a major problem in the field of autonomous driving.

The existing technology is mainly aimed at tracking the millimeter-wave radar and the forward-looking camera first and then merging in flat and straight road conditions, with only some minor modifications to the fusion method; Curve lane lines to remove invalid targets, and continuous frame tracking of targets within the drivable space. However, due to the fixed detection range of sensors such as cameras and millimeter-wave radars, blind spots may still be undetectable in curved road conditions, causing the target obstacle to be considered lost, resulting in the illusion that the vehicle has no target obstacle.

Some technologies propose vehicle-to-vehicle (V2V) technology based on the Internet of Vehicles to achieve effective identification of target tracking vehicles in curves. For example, when the rear of the front car and the boundary line of the blind spot overlap, the vehicle obtains the speed of the car in the blind spot through the V2V system of the Internet of Vehicles, and fuses the speed with the blind spot boundary to form a virtual blind spot boundary following model with speed. This solution relies on the expensive V2X to obtain the information of the preceding vehicle, and the cost is extremely high; and the current V2X network deployment is not perfect, so the whole solution is very difficult to implement.

Therefore, how to realize the effective identification and tracking of the target vehicle in the curve with low cost and high efficiency, and improve the safety of the vehicle, is the technical problem to be solved by the present application.

SUMMARY OF THE INVENTION

The present application provides a target detection method and device for effectively identifying and tracking a target vehicle in a curve with low cost and high efficiency, thereby improving the safety and stability of the vehicle during driving.

In a first aspect, a target detection method is provided, which can be applied to a vehicle or a chip inside the vehicle. Taking the method applied to the first vehicle as an example, the method includes: the first vehicle detects the speed of the preceding vehicle based on a millimeter-wave radar, and the preceding vehicle is the second vehicle as an example, and the speed includes linear velocity and angular velocity; The road conditions where the vehicle and the second vehicle are located; the first vehicle determines the first threshold according to the acquired road conditions; the first vehicle compares the speed of the second vehicle with the first threshold, and if the speed of the second vehicle exceeds the first threshold, the A vehicle determines that the millimeter-wave radar is in a target-losing state or a target is about to be lost, otherwise the first vehicle determines that the millimeter-wave radar is not in a target-losing state or a target is about to be lost.

In the embodiment of the present application, the first vehicle determines the first threshold according to the road conditions, and compares the real-time speed of the preceding vehicle (ie, the second vehicle) with the first threshold, and then determines whether the millimeter-wave radar is in a target loss state or the target is about to be lost. state. In this way, under special road conditions (such as curves, ramps, etc.), the first vehicle can timely and effectively identify the loss of the millimeter-wave radar target caused by the road conditions, so as to avoid the illusion that the first vehicle has no target obstacles. , thereby improving the safety and stability of the vehicle during driving.

In a possible design, the first vehicle may use a camera to acquire an RGB image in front of the first vehicle, and then determine the road condition where the second vehicle is located according to the RGB image.

In this way, the road condition in front of the first vehicle can be identified efficiently and quickly, that is, the road condition where the second vehicle is located can be determined.

In a possible design, the first vehicle determines the road condition on which the second vehicle is located according to the RGB image. Specifically, the feature points may be extracted from the lane lines in the far field of view included in the RGB image, and then the feature points are extracted according to the extracted feature points. Determine the turning point and direction of the road where the second vehicle is located, and then obtain the road condition where the second vehicle is located.

Since the detection range of the millimeter-wave radar in the near field of view is larger than that in the far field of view, the target in the near field of view is generally not easily lost. Therefore, the embodiment of the present application only processes the lane lines in the far field of vision, which can ensure accurate Under the premise of stability, the amount of calculation is reduced and the calculation efficiency is improved.

In a possible design, the first vehicle may acquire location information of the first vehicle based on the positioning device, and then determine the road condition where the first vehicle is located according to the location information.

Exemplarily, the positioning device may be GPS, Beidou system or other positioning systems, for receiving satellite signals and locating the current position of the first vehicle. In addition, the positioning system may also be visual positioning, millimeter wave radar positioning, fusion positioning, etc., which is not limited in this application.

After obtaining the location information of the first vehicle, the first vehicle determines the road conditions where it is located based on the map, such as being on a curve, or a straight road, or going uphill or downhill.

In this way, the first vehicle can quickly and accurately obtain the road conditions where it is located.

In a possible design, the first threshold may be determined according to a speed threshold between the second vehicle being within the detection range of the millimeter-wave radar and being located outside the detection range of the millimeter-wave radar, for example, the first threshold is less than or equal to the speed threshold . When the speed of the second vehicle is much greater than the speed threshold, the second vehicle exceeds the detection range of the millimeter-wave radar and enters the blind spot for detection of the millimeter-wave radar, so the millimeter-wave radar is in a target loss state; when the speed of the second vehicle is at the critical speed When the value is near, the second vehicle may exceed the detection range of the millimeter-wave radar at any time, and is about to enter the blind spot of the millimeter-wave radar detection, so the millimeter-wave radar is in a state where the target is about to be lost. In this embodiment of the present application, the design of the first threshold may be different according to different road conditions, and three specific examples are given below for description.

Example 1. The road conditions where the first vehicle and the second vehicle are located are: the first vehicle is on a straight road, and the second vehicle is on a curve. Then, detecting the speed of the second vehicle based on the millimeter-wave radar may include: detecting the instantaneous angular velocity of the second vehicle making a circular motion around the curve based on the millimeter-wave radar; the first threshold N determined according to the road conditions conforms to:

Among them, ω _r is the angular velocity of the second vehicle making a circular motion around the curve from time t0 to time t1,

is the Euclidean distance between the first vehicle and the second vehicle at time t0;

is the displacement of the second vehicle from time t0 to time t1, α is half of the detection beam angle of the millimeter-wave radar, time t0 is the time when the millimeter-wave radar collects the first frame of data, and time t1 is the time when the millimeter-wave radar collects the second frame of data. At the moment of frame data, the first frame data and the second frame data are two consecutive frames of data; K is a coefficient greater than 0 and less than or equal to 1; ε is the position A of the first vehicle, the position B of the second vehicle at t0, The angle value of the central angle formed between the centers O of the curve.

Example 2. The road conditions where the first vehicle and the second vehicle are located are: the first vehicle is on a curve, and the second vehicle is on a straight road. Then, detecting the speed of the second vehicle based on the millimeter-wave radar may include: detecting the instantaneous running speed of the second vehicle on the straight road based on the millimeter-wave radar; and the first threshold N determined according to the road conditions conforms to:

in,

is the displacement of the second vehicle from time t0 to time t1; v _r is the driving speed of the second vehicle on the straight from time t0 to time t1; time t0 is the time when the millimeter-wave radar collects the first frame of data, time t1 For the moment when the millimeter-wave radar collects the second frame of data, the first frame of data and the second frame of data are two consecutive frames of data; P is a coefficient greater than 0 and less than or equal to 1.

Example 3. The road conditions where the first vehicle and the second vehicle are located are: both the first vehicle and the second vehicle are in a curve. Then, detecting the speed of the second vehicle based on the millimeter-wave radar may include: detecting the instantaneous angular velocity of the second vehicle making a circular motion around the curve based on the millimeter-wave radar; the first threshold N determined according to the road conditions conforms to:

Among them, ω _r is the angular velocity of the second vehicle making a circular motion around the curve from time t0 to time t1; v is the instantaneous speed of the first vehicle; α is half of the detection beam angle of the millimeter-wave radar, and time t0 is the millimeter-wave radar. The moment when the first frame of data is collected, the moment t1 is the moment when the millimeter-wave radar collects the second frame of data, the first frame of data and the second frame of data are two consecutive frames of data; Q is a coefficient greater than 0 and less than or equal to 1;

It should be noted that the above three types are only examples and not limitations. In practical applications, road conditions are not limited to curve scenarios. For other road scenarios (such as up/downhill, acceleration/deceleration, etc.), the same idea can also be used to design preset conditions.

In a possible design, after the first vehicle determines that the millimeter-wave radar is in a target loss state or a target is about to be lost, the first vehicle may also control the millimeter-wave radar to rotate around the Z _R axis and/or around the X _R axis according to road conditions The axis is rotated so that the millimeter wave radar can detect the second vehicle again; wherein the Z _R axis is perpendicular to the horizontal plane, and the X _R axis is parallel to the horizontal plane and perpendicular to the driving direction of the first vehicle.

It should be understood that the degrees of freedom of the above two directions (that is, the direction of rotation around the Z _R axis and the direction of rotation around the X _R axis) are independent of each other (or decoupled) in the mechanical structure and independent of each other in the control logic. (or decoupling), so the first vehicle can adjust only one of the degrees of freedom, or can adjust two degrees of freedom at the same time. For example, when the first vehicle and/or the second vehicle is in a curve, the first vehicle may control the millimeter-wave radar to rotate around the Z _R axis by a first angle. For example, when the first vehicle and/or the second vehicle is on a slope, the first vehicle may control the millimeter-wave radar to rotate about the X _R axis by a second angle. For example, when the first vehicle and/or the second vehicle is in a combination road section with curves and slopes, the first vehicle may control the millimeter-wave radar to rotate around the Z _R axis by a third angle and around the X _R axis by a fourth angle.

In the embodiment of the present application, the rotation of the millimeter-wave radar around the Z _R axis and the rotation around the X _R axis are independent of each other and not dependent on each other, which can improve the accuracy and efficiency of the attitude adjustment of the millimeter-wave radar, thereby further improving the accuracy and accuracy of target detection and tracking. efficiency.

In a possible design, after the first vehicle controls the millimeter-wave radar to rotate around the Z- _R axis and/or around the X- _R axis each time, the calibration matrix of the millimeter-wave radar can be updated in real time according to the angle value of the rotation. .

In this way, a fast and accurate response of the information fusion of the millimeter-wave radar and the camera can be achieved, and the reliability of the information fusion of the millimeter-wave radar and the camera during the attitude adjustment of the millimeter-wave radar can be guaranteed.

In a possible design, the first vehicle can detect and track targets based on cameras and millimeter-wave radars, respectively, and perform target fusion according to IOU fusion rules. Specifically, the first vehicle acquires the RGB image in front of the first vehicle based on the camera, and performs target recognition on the RGB image based on the target recognition model to obtain a first recognition result, where the first recognition result includes the location and type of the second vehicle, The input of the target recognition model is an RGB image, and the output is the position and type of the target; the first vehicle obtains the radar spot data in front of the first vehicle based on the millimeter wave radar, and processes the radar spot data to obtain the second recognition result. , wherein the second identification result includes the position and speed of the second vehicle; when the first vehicle determines that the area where the second vehicle is located in the RGB image and the IoU of the area where the second vehicle is located in the radar trace data is greater than the second threshold M, the The first recognition result and the second recognition result are fused to obtain fusion data, otherwise the first recognition result and the second recognition result are not fused. Wherein, the second threshold M and the curvature ρ of the curve where the first vehicle and/or the second vehicle are located, the driving speed V of the first vehicle, and the driving distance L of the first vehicle satisfy the following relationships:

M=a ² ρ+bV+L;

Among them, a and b are preset coefficients.

It should be understood that the above formula is only an example and not a limitation. During specific implementation, the second threshold K may also be related to other factors, such as the acceleration of the first vehicle, etc., which is not limited here.

In the embodiment of the present application, the second threshold value K is related to the road condition where the vehicle is located (the curvature ρ of the curve where the first vehicle and/or the second vehicle is located), and the driving state of the first vehicle (that is, the driving speed V, the driving distance L) correlation, which can improve the accuracy of fusion recognition, thereby further improving the safety and stability of the vehicle during driving.

In one possible design, when the millimeter-wave radar is in the target-missing state or the target is about to be lost, the first vehicle may be in The second vehicle is tracked by the first recognition result when the target is lost or the target is about to be lost; or, when the millimeter-wave radar is not in the target-lost state or the target is about to be lost, the first vehicle can be based on the continuous multi-frame fusion data pair. The second vehicle is tracked.

In the embodiment of the present application, the first vehicle adopts different tracking mechanisms according to different millimeter wave radar states, which can improve the accuracy of target tracking, thereby further improving the safety and stability of the vehicle during driving.

In a possible design, the first vehicle performs target recognition on the RGB image based on the target recognition model, which may specifically include: when the millimeter-wave radar is in the target loss state or the target is about to be lost, using the lightweight target recognition model to identify the camera The collected RGB images are used for target recognition; when the millimeter-wave radar is not in the target loss state or the target is about to be lost, the weighted target recognition model is used to perform target recognition on the RGB images collected by the camera; among them, the lightweight target recognition model is used. The recognition speed is higher than that of the heavyweight target recognition model, and the recognition accuracy of the lightweight target recognition model is lower than that of the heavyweight target recognition model.

In the embodiment of the present application, two different target recognition models are designed for camera target recognition. In the case of frame loss of millimeter-wave radar, a lightweight recognition model is used to perform target recognition on the data collected by the camera to improve the recognition speed. When the wave radar does not lose frames, the weighted recognition model is used to identify the data collected by the camera to improve the recognition accuracy, so that the speed and accuracy of fusion recognition can be achieved.

In a second aspect, a target detection device is provided, and the device is located in a first vehicle, for example, a chip provided inside the vehicle. The device includes modules or units or means corresponding to the steps performed by the first vehicle in the first aspect or any possible design of the first aspect, and the functions, units or means can be implemented by software, or by It can be realized by hardware, and can also be realized by executing corresponding software by hardware.

Exemplarily, the apparatus may include: a detection module for detecting the speed of the second vehicle based on a millimeter-wave radar; wherein the second vehicle is located in front of the first vehicle, and the speed includes a linear velocity and an angular velocity; and a processing module for obtaining The road conditions where the first vehicle and the second vehicle are located; the first threshold is determined according to the road conditions; if the speed of the second vehicle exceeds the first threshold, it is determined that the millimeter-wave radar is in a target loss state or a target loss state.

In a third aspect, a computer-readable storage medium is provided, comprising a program or an instruction, when the program or instruction is executed on a computer, the method as described above in the first aspect or any possible design of the first aspect is performed.

In a fourth aspect, a target detection device is provided, comprising a processor and a memory; wherein, the memory stores instructions that can be executed by the processor, and the processor executes the instructions stored in the memory to cause the device to execute the above-mentioned first aspect or the first Aspect the method in any possible design.

In a fifth aspect, a chip is provided, which is coupled to a memory and used to read and execute program instructions stored in the memory, so as to implement the method in the first aspect or any possible design of the first aspect.

A sixth aspect provides a computer program product comprising instructions, the computer program product having instructions stored in the computer program product that, when run on a computer, cause the computer to perform the above-mentioned first aspect or any possible design of the first aspect. method.

In a seventh aspect, a vehicle is provided, the vehicle includes a target detection device, a millimeter-wave radar, and a camera; the target detection device is configured to control the millimeter-wave radar and the camera to achieve the above-mentioned first aspect or any one of the first aspects possible. method in design.

Specifically, the target detection device can detect the speed of another vehicle in front of the current vehicle through a millimeter-wave radar, and obtain the road conditions where the first vehicle and the second vehicle are located through a camera, and the target detection device further determines the first threshold according to the road conditions; if The speed of another vehicle in front of the current vehicle exceeds the first threshold, and it is determined that the millimeter-wave radar is in a target loss state or a target loss state.

For the beneficial effects of the above-mentioned second to seventh aspects, refer to the beneficial effects of the corresponding designs in the first aspect, which will not be repeated here.

Description of drawings

Figure 1 is a schematic diagram of the relationship between the detection beam angle and the detection distance of the millimeter-wave radar;

FIG. 2 is a schematic diagram of the detection range of the camera and the detection range of the millimeter wave radar;

FIG. 3A is a possible application scenario applicable to the embodiment of the present application;

FIG. 3B is another possible application scenario applicable to the embodiment of the present application;

FIG. 3C is another possible application scenario applicable to the embodiment of the present application;

FIG. 3D is another possible application scenario to which the embodiment of the present application is applicable;

Fig. 4 is a possible vehicle architecture diagram;

FIG. 5 provides a flowchart of a target detection method according to an embodiment of the present application;

6 is a schematic diagram of a first vehicle in a straight road and a second vehicle in a curve;

7 is a schematic diagram of a first vehicle in a curve and a second vehicle in a straight road;

8 is a schematic diagram of a first vehicle in a curve and a second vehicle in a curve;

9 is a schematic diagram of a three-dimensional coordinate system of a millimeter-wave radar;

Figure 10 is a schematic diagram of adjusting the attitude of the millimeter wave radar;

Figure 11 is a schematic diagram of a millimeter-wave radar coordinate system and a camera coordinate system;

12 is a flowchart of a target tracking method provided by an embodiment of the present application;

13 is a schematic diagram of the ROI corresponding to the detection result of the millimeter wave radar and the ROI corresponding to the detection result of the camera;

Figure 14 is a schematic diagram of the intersection and union of the millimeter-wave radar detection results and the camera detection results;

Figure 15 and Figure 16 are schematic diagrams of an uphill scene;

FIG. 17 is a system architecture diagram provided by an embodiment of the present application;

FIG. 18 is a schematic structural diagram of a target detection apparatus 180 provided by an embodiment of the present application;

FIG. 19 is a schematic structural diagram of another target detection apparatus 190 provided by an embodiment of the present application.

Detailed ways

When the vehicle detects and tracks the target based on the camera and millimeter wave radar, the camera can basically cover the short-range target obstacle, while the long-distance target obstacle needs to rely on the millimeter wave radar.

The detection beam angle and installation position of the millimeter-wave radar are fixed, so the detection range (generally the sector angle) is limited. FIG. 1 is a schematic diagram of the relationship between the detection beam angle and the detection distance of the millimeter wave radar. As shown in Figure 1, the detection beam angle changes with the detection distance, and the detection angle of the millimeter-wave radar in the far field of view is smaller than that in the near field of view. Therefore, in curved road conditions, the target obstacle (such as the front vehicle) detected and tracked by the rear vehicle is likely to exceed the detection area of the rear vehicle millimeter-wave radar, as shown in Figure 2.

The detection range of the camera is generally a sector, and the field of view (fov) of the commonly used camera is 60°, 90°, 150° and so on. The detection distance of a camera is inversely proportional to the field of view. For example, a camera with a fov of 90° has a maximum detection distance of 80 meters, while a camera with a fov of 150° has a detection distance that is much shorter than 80 meters. As shown in Figure 2, the detection range of the camera (ie, the detection area of the camera) is wider than that of the radar, but the detection distance is shorter than that of the radar.

On vehicles equipped with millimeter-wave radar and camera sensors, the accurate position and speed of target obstacles come from the detection data of millimeter-wave radar. Therefore, when the target exceeds the detection range of the millimeter-wave radar, since the rear vehicle cannot obtain the accurate position, speed and other information of the target obstacle, the target obstacle will be considered lost by the rear vehicle, resulting in the illusion of no target obstacle.

But in fact, the target obstacle really exists in the objective world. Because millimeter-wave radar is the most reliable guarantee for forward side collision warning, the loss of target obstacles in such curved road conditions will cause the automatic driving system to fail to obtain an accurate safe distance, which will lead to advanced driving assistance. System (Advanced Driving Assistance System, ADAS) and automatic driving tracking have a very serious impact, resulting in major safety hazards.

In view of this, the embodiments of the present application provide a target detection method and device, which aim to introduce the consideration of road detection blind spots, quickly identify special road conditions according to the positioning information and visual information of the vehicle, and use kinematic physical quantities (such as target Angular velocity/velocity, etc.) to determine whether sensors such as millimeter-wave radar are in a state of target loss or the target is about to be lost, so as to avoid the illusion that the vehicle has no target obstacles. Further, when sensors such as millimeter-wave radar are in the state of target loss or the target is about to be lost, the attitude of sensors such as millimeter-wave radar is adjusted, so that sensors such as millimeter-wave radar can re-detect the target in time to ensure that the vehicle is in the driving process. security and stability. Furthermore, when adjusting the attitude of a sensor such as a millimeter-wave radar, the embodiment of the present application compensates and corrects the attitude of the radar from two degrees of freedom (for example, left and right (α) and up and down (β)). The adjustment of each degree of freedom is independent and does not depend on each other, so the accuracy and efficiency of the adjustment can be improved.

The technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings in the embodiments of the present application.

It should be understood that in the description of this application, a plurality of means two or more. At least one means one or more. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship. Words such as "first" and "second" are only used for the purpose of distinguishing and describing, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order.

FIG. 3A is a possible application scenario to which the embodiment of the present application is applicable, and the scenario may be scenarios such as automatic driving, driving assistance, and manual driving. The scene includes at least two vehicles, and FIG. 3A takes two vehicles as an example. A radar sensor, such as a millimeter-wave radar sensor, is also arranged in front of the rear vehicle (ie, the self-vehicle), and a camera is also arranged, and the rear vehicle can detect and track the preceding vehicle through the millimeter-wave radar and the camera. In the scenario shown in Figure 3A, both the leading and trailing vehicles are in a curve.

FIG. 3B is another possible application scenario to which the embodiment of the present application is applicable, and the scenario may be scenarios such as automatic driving, driving assistance, and manual driving. Different from FIG. 3A , in the scene shown in FIG. 3B , the preceding vehicle is on a curve, and the following vehicle (ie, the ego vehicle) is on a straight road.

FIG. 3C is another possible application scenario to which the embodiment of the present application is applicable, and the scenario may be scenarios such as automatic driving, driving assistance, and manual driving. Different from FIGS. 3A and 3B , in the scenario shown in FIG. 3C , the vehicle behind (ie, the ego vehicle) is on a curve, while the vehicle in front is on a straight road.

It should be noted that in the above-mentioned Figures 3A, 3B, and 3C, two vehicles are shown (that is, the vehicle has only one vehicle in front). It should be understood that this does not limit the number of vehicles in the application scenario. In the actual scenario More vehicles may also be included. For example, as shown in Fig. 3D, the ego vehicle is in a curve, the previous vehicle is in a curve, and the previous two vehicles are in a straight road. When there are multiple front vehicles, the detection principle of the self-vehicle for each preceding vehicle is the same, so in the embodiments of the present application, a preceding vehicle is mainly used as an example.

In addition, FIGS. 3A , 3B, 3C, and 3D take curved road conditions as an example, and the embodiments of the present application can also be applied to other road conditions that may lead to vehicle detection blind spots, such as up/downhill, acceleration/deceleration, etc. , this application does not limit.

The target detection method provided by the embodiment of the present application can be applied to the vehicle in the above-mentioned scenario. Specifically, the method can be carried in the computing device of the vehicle through software, and the computing device is, for example, a separate vehicle-mounted device (such as a vehicle-mounted computer, Driving control device, driving control device, etc.), or one or more processing chips, or integrated circuits, etc., which are not limited in this application.

Referring to FIG. 4 , it is a possible vehicle architecture diagram, including computing equipment, control equipment, and on-board sensors. The various components in the vehicle are described below.

Vehicle-mounted sensors (sensors for short) are used to collect various sensor data of the vehicle in real time. The sensors installed on the vehicle in the embodiments of the present application include, for example, a camera (or referred to as a camera), a positioning system, a radar sensor, an attitude sensor, and the like. In addition, other sensors may also be included, such as a shaft rotational speed sensor, a wheel speed sensor, etc., which are not limited in this application.

Among them, the radar sensor may be referred to as radar. The radar can measure the radar trace data of the surrounding environment, and can also measure the information of the obstacles in the set direction of the vehicle, such as the position and speed of the preceding vehicle (including linear velocity, angular velocity), etc.

The radar in the embodiments of the present application mainly takes a millimeter-wave radar as an example. Millimeter wave radar: It is a radar that works in the millimeter wave band (millimeter wave). Generally, millimeter wave refers to the 30-300GHz frequency band (wavelength is 1-10mm). The wavelength of millimeter wave is between centimeter wave and light wave, so millimeter wave has the advantages of microwave guidance and photoelectric guidance. Installing the millimeter-wave radar on the car can measure the distance, angle and relative speed from the millimeter-wave radar to the object to be measured. Using millimeter wave radar can realize adaptive cruise control, forward collision warning (Forward Collision Warning), blind spot detection (Blind Spot Detection), assisted parking (Parking aid), auxiliary lane change (Lane change assistant), autonomous cruise control, etc. Advanced Driver Assistance Systems (ADAS) features.

In the embodiment of the present application, the millimeter-wave radar may be linked with the follower mechanism, and when the follower mechanism rotates, the attitude of the millimeter-wave radar changes accordingly. Among them, the follow-up mechanism can be a component of the millimeter-wave radar, or it can be a supporting device of the millimeter-wave radar, which is not limited here.

Attitude sensor: Attitude sensor is a high-performance three-dimensional motion attitude measurement system based on sensor, namely Microelectro Mechanical Systems (MEMS) technology. In the embodiment of the present application, the attitude sensor is installed on the radar and is used to detect the attitude of the radar.

Cameras can be deployed around the vehicle and collect environmental parameters around the vehicle. For example, at least one camera may be installed on the front and rear bumpers, side mirrors, windshield, and roof of the vehicle, respectively. In this embodiment of the present application, the camera can at least acquire an image in front of the vehicle, so that the computing device can determine the type of the target obstacle (for example, the vehicle in front) in the driving direction of the vehicle according to the image, and can also determine the distance between the vehicle and the target obstacle. , the location of the target obstacle and the road conditions.

It should be emphasized that the vehicle-mounted sensor in the embodiment of the present application is a combination of a camera and a millimeter-wave radar as an example. However, this application does not limit the specific type of sensor. For example, the sensor of a vehicle can also be a combination of solid-state lidar and camera, a combination of millimeter-wave radar and solid-state lidar, a combination of camera and camera, millimeter-wave radar, solid-state laser and camera. combination, etc. As long as a sensor with a large detection range, short detection distance, and inaccurate detection results (obstacle position and speed, etc.) is used in conjunction with a sensor with a small detection range, long detection distance, and accurate detection results, the embodiments of the present application are applicable.

The positioning system is used to locate the current position of the vehicle. Exemplarily, the positioning system may be a global positioning system (Global Positioning System, GPS), a Beidou system or other positioning systems, for receiving satellite signals and locating the current position of the vehicle. The positioning system may also be visual positioning, radar positioning, fusion positioning, etc., which are not limited in this application, and GPS is mainly used as an example in the following text.

The computing device is responsible for the computing function, which is used to collect the parameter information of the surrounding environment of the vehicle in real time during the driving of the vehicle according to various sensors (such as cameras, radars, positioning systems, etc.) installed on the vehicle. The parameter information is calculated and analyzed, and the vehicle control command can be determined according to the analysis result.

Various computing functions may be provided in computing devices. For example, it includes: perceiving the surrounding environment information during the driving process of the vehicle according to the parameter information collected by the camera and the radar; determining the geographic location of the vehicle according to the parameter information collected by the positioning system; according to the parameter information collected by the camera, radar, positioning system and other sensors Perform calculation and analysis to determine the state of the vehicle, for example, determine whether the radar is in the state of target loss or the target is about to be lost; generate control instructions according to the parameter information collected by the above sensors and send them to the control device, so that the control device controls the corresponding sensors, such as When the radar is in the target loss state or is about to be in the target loss state, an instruction to control the rotation of the follower mechanism of the radar can be generated, and then the instruction is sent to the control device, so that the control device controls the rotation of the follower mechanism, and then adjusts the attitude of the radar, so that The radar was able to regain the target.

This embodiment of the present application does not specifically limit the type of the computing device. As an example, the computing device may be a Mobile Data Center (MDC). The MDC is the local computing platform of the autonomous vehicle. The autonomous driving software running on the MDC includes the camera perception algorithm, millimeter-wave radar perception algorithm, and target fusion tracking algorithm of this solution (that is, the target tracking method based on the fusion of radar information and camera information corresponds to algorithm) etc. The MDC also runs some simple operations, such as motor control.

The embodiment of the present application does not specifically limit the type of the control device. As an example, the control device may be a micro-control unit (MCU). Optionally, the MCU may also be referred to as a single-chip microcomputer (singlechip microcomputer, SCM) or a single-chip microcomputer.

It should be understood that the micro control unit MCU can appropriately reduce the frequency and specifications of the central processing unit (CPU), and combine the memory (memory), counter (timer), universal serial bus (universal serial bus, USB), Analog-to-digital (analog to digital, AD) conversion, universal asynchronous receiver transmitter (UART), programmable logic controller (PLC), direct memory access (direct memory access, DMA), etc. At least one peripheral interface, and even a liquid crystal display (LCD) driver circuit are integrated on a single chip to form a chip-level computing device that can be controlled in different combinations for different applications.

After receiving the vehicle control command sent by the computing device, the control device can control the vehicle sensor (eg, control the attitude of the radar) through the vehicle control interface, so as to realize the auxiliary control of the vehicle.

Those skilled in the art can understand that the structure of the in-vehicle device shown in FIG. 4 does not constitute a limitation on the in-vehicle device, and the in-vehicle device provided in the embodiments of the present application may include more or less modules than those shown in the figure, or a combination of certain modules may be included. Some modules or different component arrangements are not limited in this application. For example, the in-vehicle device may also include braking mechanisms (such as brakes, accelerators, gears, etc.), human-computer interaction input and output components (such as display screens, etc.), wireless communication modules, communication interfaces, and the like.

As shown in FIG. 5 , a flow chart of a method for object detection is provided in an embodiment of the present application. The method can be applied to the vehicle shown in FIG. 4 , and the method includes:

S501. The first vehicle acquires the road conditions where the first vehicle (own vehicle) and the second vehicle (the preceding vehicle) are located, and acquires the speed of the second vehicle.

Specifically, the first vehicle is equipped with sensors such as a camera, a millimeter-wave radar, and a positioning system, and the computing device collects sensor data collected by each sensor in real time. Among them, the camera can obtain the image information in front of the first vehicle in real time and detect and track the second vehicle (for example, detect the type and position of the second vehicle), the positioning system can locate the position of the first vehicle in real time, and the millimeter wave radar The target obstacle (taking the second vehicle as an example herein) can be detected and tracked (eg, the position and speed of the second vehicle are detected). For the specific structure of the first vehicle, reference may be made to the structure shown in FIG. 4 , which will not be described in detail here.

The computing device of the first vehicle may determine the road condition in which the first vehicle is located based on the positioning system. Specifically, the computing device of the first vehicle can obtain the position of the first vehicle on the map based on the positioning system, so as to determine the road condition on which it is located, such as a curve, a straight road, an uphill, or a downhill, based on the map.

The computing device of the first vehicle may also determine the distance traveled by the first vehicle based on the positioning system. For example, referring to FIG. 6 or FIG. 7 or FIG. 8 , at time t0 the positioning system locates the location of the first vehicle as point A; at time t1 the positioning system locates the location of the first vehicle as point A', and the computing device locates the location of the first vehicle as point A' according to point A and point A' The position can calculate the distance from point A to point A', namely

(Displacement of the first vehicle from time t0 to time t1).

The computing device of the first vehicle may also acquire the road conditions where the second vehicle is located through the camera. Specifically, the computing device of the first vehicle controls the camera to capture an image in front of the first vehicle, and the image includes information about the second vehicle and the road where the second vehicle is located, such as the curvature radius of the lane line or other road signs, etc., Then, based on the image, determine the road condition on which the second vehicle is located, such as a curve or a straight road, or an uphill, or a downhill, and the like.

The computing device of the first vehicle may also comprehensively determine information such as the position and speed of the second vehicle according to the data detected by the camera and the millimeter-wave radar. The specific implementation can refer to the target tracking method based on the fusion of millimeter-wave radar information and camera information, which is described later, and will not be introduced in detail here.

S502. If the speed of the second vehicle satisfies the preset condition corresponding to the road condition, the first vehicle determines that the millimeter-wave radar of the first vehicle is in a target loss state or a target loss state.

The speed of the second vehicle includes at least two kinds of angular speed and linear speed, and the preset condition includes whether the speed of the second vehicle is greater than or equal to the set first threshold.

Wherein, the set first threshold should be less than or equal to the speed of the second vehicle when driving on the boundary of the detection range of the millimeter-wave radar (that is, the second vehicle is located within the detection range of the millimeter-wave radar and located outside the detection range of the millimeter-wave radar. speed threshold in between).

For example, the first threshold is equal to the speed of the second vehicle when it is traveling on the boundary of the detection range of the millimeter-wave radar, then when the speed of the second vehicle exceeds the first threshold, the second vehicle will exceed the detection range of the millimeter-wave radar and enter the millimeter-wave radar The radar detects the blind spot. At this time, the millimeter wave radar is in the target loss state. When the speed of the second vehicle is equal to the first threshold (or the speed of the second vehicle is less than the first threshold and the difference between the speed of the second vehicle and the first threshold is less than When the preset difference is set, the millimeter-wave radar is in a state where the target is about to be lost.

For example, the first threshold is smaller than the speed of the second vehicle when it is traveling on the boundary of the detection range of the millimeter-wave radar, and the difference between the first threshold and the critical value is set to be ΔX, then the speed of the second vehicle exceeds the first threshold and the third When the difference between the speed of the second vehicle and the first threshold is less than or equal to ΔX, the millimeter-wave radar is in a state where the target is about to be lost, the speed of the second vehicle exceeds the first threshold, and the speed of the second vehicle is equal to the first threshold. When the difference is greater than △X, the millimeter-wave radar is in a target loss state.

In addition, in order to improve the safety of the vehicle, the state of the millimeter-wave radar can also be set as the target loss state when the millimeter-wave radar is actually in the target loss state (that is, the target loss state includes actual loss and impending loss.) .

It should be understood that during specific implementation, other terms can also be used to describe that the millimeter-wave radar is in a target-losing state or a state where the target is about to be lost, for example, the millimeter-wave radar is in a frame-loss state (this state can be understood as the millimeter-wave radar does not detect a target containing a target). data frame), which is not limited in this application.

In this embodiment of the present application, for different road conditions, the preset conditions may be different, for example, the first threshold is different. When the speed of the second vehicle satisfies the preset condition corresponding to the current road condition, the computing device of the first vehicle determines that the millimeter-wave radar of the first vehicle is in a target loss state or a target loss state.

It should be noted that, during the driving process, if the attitude of the millimeter-wave radar is not adjusted, the detection range of the millimeter-wave radar is fixed relative to the first vehicle. However, when the positions of the first vehicle and the second vehicle, and/or the relative positions of the first vehicle and the second vehicle are different, the first threshold value is different.

Several specific examples are given below to illustrate.

Example 1. Referring to FIG. 6 , the first vehicle is on a straight road and the second vehicle is on a curve, then the preset conditions include:

Each parameter is explained as follows:

Time t0 is the time when the millimeter-wave radar collects the first frame of data, and time t1 is the time when the millimeter-wave radar collects the second frame of data, where the first frame of data and the second frame of data can be two consecutive frames of data. It can also be two frames of data separated by a small number of frames (for example, the first frame data and the second frame data are separated by 1 frame or 2 frames). In the embodiments of the present application, two consecutive frames of data are mainly used as an example.

At time t0, the first vehicle is at position A, the second vehicle is at position B, and position B is the boundary position of the detection range of the millimeter-wave radar when the first vehicle is at position A. At time t1, the first vehicle is at the A' position, the second vehicle is at the B' position, and the B' position is the boundary position of the detection range of the millimeter-wave radar when the first vehicle is at the A' position. In this embodiment of the present application, the boundary position may be an area or a range. For example, the position within the ±L area from the boundary line of the radar detection range may be defined as the boundary position of the millimeter wave radar detection range.

ω _r ' is the instantaneous angular velocity of the second vehicle making a circular motion around the curve at time t2; N is the first threshold;

ω _r is the angular velocity of the second vehicle making a circular motion around the curve from time t0 to time t1 (the average angular velocity, when the interval between t0 and t1 is extremely short (for example, the time to collect a frame of data), the angular velocity can also be considered as instantaneous. speed), t2≥t1>t0;

is the displacement of the first vehicle from time t0 to time t1;

is the displacement of the second vehicle from time t0 to time t1; it should be noted that the actual path of the second vehicle from B to B' is a curve, but since time t0 to time t1 is very short, such as 30-50ms, so This section of the path can be approximated as a straight line, that is, the straight-line distance between the second vehicle from time t0 to time t1 is approximately equal to the straight line distance from point B to point B';

ε is the angle value of the central angle formed between A, B, and O at time t0;

α is half of the detection beam angle of the millimeter-wave radar, which is determined by the characteristics of the millimeter-wave radar;

Among them, the coordinates of A and A' can be obtained by the positioning system, and the position coordinates of B and B' can be obtained by the target tracking method based on the fusion of millimeter-wave radar information and camera information. The target tracking method will be further detailed later. introduce. According to the position coordinates of A, A', B, B', we can get

ε, ω _r , etc. can be calculated from the geometric points of the position coordinates of A, A', B, B' output by the target tracking method. ω _r ' may be detected by the millimeter-wave radar or calculated according to the linear velocity v _r ' of the second vehicle detected by the millimeter-wave radar.

K is the first preset coefficient, and the value range is (0, 1]. Optionally, the value of K may be related to the execution time of the first vehicle to adjust the follow-up mechanism of the millimeter-wave radar and/or the urgency of the curve, etc. .The longer the execution time, the smaller the K, the shorter the execution time, the larger the K;

Exemplarily, the value of K satisfies the following formula:

K(R,t)=e^-(t/aR);

Among them, t is the execution time, R is the radius of the curve, and a is the undetermined coefficient.

The following describes the derivation process of ω _r :

During the driving process of the automatic driving vehicle, the first vehicle is on a straight road, and the target vehicle is on a curve. At time t0, the first vehicle is at position A, the target vehicle is at position B, and the position B is the boundary position of the detection range of the millimeter-wave radar when the first vehicle is at position A. At time t1, the first vehicle is at A' and the target vehicle is at B', and the B' position is the boundary position of the detection range of the millimeter-wave radar when the first vehicle is at the A' position. O indicates that it is the center of the curve (the center of curvature), and the graph is analyzed and connected

are perpendicular to the tangent directions of B and B' respectively (because the time from t0 to t1 is very short, so

can be considered to be perpendicular to

).

Referring to Figure 6, according to the geometric relationship, there are:

It can be obtained by the above formula:

Due to road conditions,

can be seen as approximately perpendicular to

Available: ε≈γ;

The turning radius of the second vehicle in the curve is:

It can be considered that the first vehicle and the second vehicle are in the same lane, then:

Turning radius of the second vehicle:

According to the calculation formula of the circular motion of the car, we can get:

Example 2. Referring to FIG. 7 , the first vehicle is on a curve and the second vehicle is on a straight road, the preset conditions include:

Each parameter is explained as follows:

At time t0, the first vehicle is at position A, the second vehicle is at position B, and position B is the boundary position of the detection range of the millimeter-wave radar when the first vehicle is at position A. At time t1, the first vehicle is at the A' position, the second vehicle is at the B' position, and the B' position is the boundary position of the detection range of the millimeter-wave radar when the first vehicle is at the A' position. As in Example 1, the boundary position may be an area or a range.

is the displacement of the second vehicle from time t0 to time t1;

v _r ' is the instantaneous traveling speed of the second vehicle on the straight road at time t2 (the traveling speed in this paper refers to the linear speed), t2≥t1>t0; N is the first threshold;

v _r is the running speed of the second vehicle on the straight road from time t0 to time t1 (travel speed in this paper refers to the linear speed, where v _r is the average linear speed, when the interval between t0 and t1 is very short (for example, collecting a frame data time), the linear velocity can also be considered as the instantaneous linear velocity);

Among them, the coordinates of A and A' can be obtained by the positioning system, and the position coordinates of B and B' can be obtained by the target tracking method based on the fusion of millimeter-wave radar information and camera information. The target tracking method will be introduced in detail later. . According to the position coordinates of B and B', we can get

v _r can be calculated from the geometric points of the position coordinates of A, A', B, and B' output by the target tracking method. v _r ' can be detected by millimeter wave radar.

P is the second preset coefficient, and the value range is (0, 1]. P is the same as or different from K in Example 1. Optionally, the value of P is the same as that of the first vehicle to adjust the execution of the follow-up mechanism of the millimeter-wave radar. Time and/or the priority of the curve, etc. The longer the execution time, the smaller the P, the shorter the execution time, the larger the P;

Exemplarily, the value of P satisfies the following formula:

P(R,t)=e^-(t/aR);

The following describes the derivation process of v _r :

Referring to Figure 7, because the turning radius of the first vehicle:

θ=ω _r (t ₁ −t ₀ );

also because

From the trigonometric cosine theorem we have:

Among them, β is equal to α, which is half of the detection beam angle of the radar.

So by the trigonometric sine theorem we have:

And because ε=ξ+θ-γ, so

So the speed of the second vehicle in a straight line is:

Example 3. Referring to FIG. 8 , the first vehicle is on a curve and the second vehicle is on a curve, then the preset conditions include:

Each parameter is explained as follows:

At time t0, the first vehicle is at position A, the second vehicle is at position B, and position B is the boundary position of the detection range of the millimeter-wave radar when the first vehicle is at position A. At time t1, the first vehicle is at the A' position, the second vehicle is at the B' position, and the B' position is the boundary position of the detection range of the millimeter-wave radar when the first vehicle is at the A' position.

ω _r ' is the instantaneous angular velocity of the second vehicle making a circular motion around the curve at time t2;

ω _r is the angular velocity of the second vehicle making a circular motion around the curve from time t0 to time t1 (average angular velocity, when the interval between t0 and t1 is extremely short (for example, the time to collect one frame of data), the angular velocity can also be considered as the instantaneous velocity. ), t2≥t1>t0; N is the first threshold;

v is the traveling speed (instantaneous linear speed) of the first vehicle, which can be obtained from the chassis information of the first vehicle or the first vehicle positioning system;

Among them, the coordinates of A and A' can be obtained by the positioning system, and the position coordinates of B and B' can be obtained by the target tracking method based on the fusion of millimeter-wave radar information and camera information. The target tracking method will be introduced in detail later. . According to the position coordinates of A, A', B, B', we can get

ω _r ' may be detected by the millimeter-wave radar or calculated according to the linear velocity v _r ' of the second vehicle detected by the millimeter-wave radar.

Q is the third preset coefficient, the value range is (0,1], and Q is the same as or different from P and K. Optionally, the value of Q is the same as the execution time of the first vehicle to adjust the follow-up mechanism of the millimeter-wave radar and the / or the priority of the curve, etc. The longer the execution time, the smaller the Q, the shorter the execution time, the larger the Q; the more acute the curve, the smaller the Q, the slower the curve, the larger the Q.

Exemplarily, the value of Q satisfies the following formula:

Q(R,t)=e^-(t/aR);

The derivation process of _ωr is as follows:

When both the first vehicle and the target vehicle are in the process of driving on a curve, as shown in Figure 8, cross point O, make OC perpendicular to AB and cross AE to D;

Assuming that both cars are in uniform circular motion,

And ∠COA=∠CAD=α;

So the turning radius of the second vehicle is:

so

It should be noted that the above is only an example and not a limitation. In practical applications, the same idea can also be used to design the preset conditions for other road scenarios.

It should be understood that the above takes the target as the second vehicle as an example. In practical applications, other targets, such as pedestrians, animals, etc., can also be detected using the technical solutions of the embodiments of the present application, which are not limited in the present application.

It can be seen from the above that the first vehicle in the embodiment of the present application can quickly identify special road conditions through the positioning information and visual information collected by itself, and can monitor the kinematic physical quantities of the second vehicle (such as angular velocity/line under special road conditions) Speed, etc.) can quickly judge whether the sensors such as the millimeter-wave radar of the vehicle are in the target loss state or the target is about to be lost (the fastest only need to be identified according to the continuous two frames of data collected by the millimeter-wave radar), so as to prevent the vehicle from generating no noise. The illusion of target obstacles improves the accuracy of target detection.

The following describes a scheme for adjusting the attitude of the millimeter-wave radar by the first vehicle after the millimeter-wave radar is in a target-loss state or a target is about to be lost (or in other words, the millimeter-wave radar is in a frame-loss state).

For ease of understanding, here is a brief introduction to attitude sensors: attitude sensors are implemented by one or more of three types of sensors: acceleration sensor (ie accelerometer), angular velocity sensor (ie gyroscope), and magnetic induction sensor (ie magnetometer). Combination implementation. Attitude sensors include three-axis attitude sensors (or three-dimensional attitude sensors), six-axis attitude sensors (or six-dimensional attitude sensors), nine-axis attitude sensors (or nine-dimensional attitude sensors), and the like. Among them, the three-axis attitude sensor is implemented by one type of sensor (such as three-axis accelerometer or three-axis gyroscope or three-axis magnetometer); the six-axis attitude sensor is generally implemented by two types of sensors (such as three-axis accelerometer + three-axis gyroscope) ); the nine-axis attitude sensor generally consists of a three-axis gyroscope + three-axis accelerometer + three-axis geomagnetometer, there are also six-axis accelerometer + three-axis gyroscope, and there are also six-axis gyroscope + three-axis accelerometer.

The attitude sensor in the embodiment of the present application may be a six-dimensional attitude sensor or a nine-dimensional attitude sensor, etc., which is not limited in this application. The attitude sensor can detect the attitude of the millimeter wave radar in real time.

Referring to FIG. 9, the embodiment of the present application takes the center of the millimeter-wave radar as the center of the circle when the vehicle is driving on a straight road, and establishes a three-dimensional coordinate system of the millimeter-wave radar, wherein the X- _R axis is parallel to the ground direction (take the right direction of the millimeter-wave radar as the center of the circle). is positive, the left direction is negative), the Z _R axis is perpendicular to the ground (in the upward direction is positive, the downward direction is negative), the Y _R axis is perpendicular to the plane where the X _R axis and the Z _R axis are located (in the forward direction of the millimeter wave radar is positive and backward is negative), the attitude of the millimeter-wave radar can be represented by the following three parameters:

1) Yaw angle (Yaw), representing the rotation of the millimeter-wave radar around the Z _R axis, let Yaw=α;

2) Pitch angle (Pitch), indicating the rotation of the millimeter-wave radar around the X _R axis, set Pitch=β;

3) Roll angle (Roll), which represents the rotation of the millimeter-wave radar around the Y and _R axes, where Roll=γ.

Since the embodiment of the present application mainly considers the target detection of the preceding vehicle by the first vehicle, the attitude sensor mainly acquires α and β of the millimeter-wave radar. Assuming that when the vehicle is driving on a straight road, the attitude sensor detects that the attitude parameters α and β of the millimeter-wave radar are both 0, then when the vehicle is driving on a curve and/or a ramp, the attitude sensor detects that the millimeter-wave radar α and β are may be greater or less than 0.

In the embodiment of the present application, when the millimeter-wave radar is in a target loss state or the target is about to be lost, the attitude of the millimeter-wave radar is compensated and corrected by comprehensively considering the degrees of freedom of the millimeter-wave radar that are closely related to road characteristics.

Taking curved road conditions as an example, the attitude of the millimeter-wave radar can be adjusted from the left and right rotational degrees of freedom:

1) Left and right rotational degrees of freedom. As shown in Figure 9, the left and right rotation degrees of freedom are the degrees of freedom of the radar to rotate around the Z _R axis, and the corresponding rotation angle is α. A standard for closed-loop control of angle adjustment based on target re-detection by millimeter-wave radar. Among them, it is difficult to establish an accurate functional relationship between the approximate input angle and the output angle, so the fuzzy-PID adaptive control method is preferred.

Taking the slope condition as an example, the attitude of the millimeter-wave radar can be adjusted from the up and down swing degrees of freedom:

2) Up and down swing degrees of freedom. As shown in Figure 9, the degree of freedom of up and down rotation is the degree of freedom of the radar to rotate around the X- _R axis, and the corresponding rotation angle is β.

It should be understood that the degrees of freedom of the above two directions (ie up and down and left and right) are independent (or decoupled) in the mechanical structure and independent (or decoupled) in the control logic, so the computing device Only one degree of freedom can be adjusted. Of course, according to the actual road conditions, if the road feature is a combination of curves and slopes, the computing device can also adjust the attitude of the millimeter-wave radar from the left and right rotation degrees of freedom and the up and down swing degrees of freedom at the same time.

Referring to FIG. 10 , it is a schematic diagram of adjusting the attitude of the millimeter-wave radar by the computing device of the first vehicle (the MDC is taken as an example in FIG. 10 ). The target detection method described in the embodiment shown in FIG. 5 is run on the computing device, which can identify whether the millimeter-wave radar is in a target loss state or a target is about to be lost. In addition, the computing device also runs based on the millimeter-wave radar information and camera information. A fused target tracking method (the target tracking method may be a method in the prior art, or may be the target tracking method introduced in the related embodiment of FIG. 12 later, which is not limited here). The follow-up mechanism of the millimeter-wave radar includes a Z- _R -axis rotating motor and an X- _R -axis rotating motor. The Z- _R -axis rotating motor can rotate under the control of voltage to drive the millimeter-wave radar to rotate around the Z- _R -axis, and the X- _R -axis rotating motor can rotate. It rotates under the control of voltage and drives the millimeter-wave radar to rotate around the X _R axis.

The process that the computing device controls and adjusts the attitude of the millimeter-wave radar includes:

1) Based on the target detection method, the computing device detects whether the millimeter-wave radar is in the target loss state or the target is about to be lost (ie, whether the frame is lost);

2) When the millimeter-wave radar is in the target loss state or the target is about to be lost, the computing device determines the first angle adjustment value of the millimeter-wave radar according to the current attitude of the millimeter-wave radar (for example, the millimeter-wave radar needs to rotate around the Z _R axis by Δα. The angle needs to be rotated around the X _R axis by the angle of △β); the computing device sends the angle to the control device through an instruction (Figure 10 takes the MCU as an example); the control device saves the mapping relationship between the angle adjustment value and the voltage adjustment value. After receiving the instruction, the device converts the first angle adjustment value into the first voltage V1 that needs to be sent to the Z _R -axis rotating motor and the second voltage V2 that needs to be sent to the X _R -axis rotating motor according to the mapping relationship; the control device controls the Z _R The input voltage of the axis rotating motor is V1 and the input voltage controlling the X _R axis rotating motor is V2, so that the Z _R axis rotating motor rotates to drive the millimeter wave radar to rotate around the Z _R axis by an angle of Δα, and the X _R axis rotates and then Drive the millimeter-wave radar to rotate around the X _R axis by an angle of Δβ; or,

The computing device saves the mapping relationship between the angle adjustment value and the voltage adjustment value. When the millimeter-wave radar is in the target loss state or the target is about to be lost, the computing device determines the first angle adjustment value of the millimeter-wave radar according to the current attitude of the millimeter-wave radar. Then, according to the mapping relationship, the first angle adjustment value is converted into a first voltage V1 that needs to be delivered to the Z _R -axis rotary motor and a second voltage V2 that needs to be delivered to the X _R -axis rotary motor; the computing device sends V1 and V2 through the command To the control device, after the control device receives the command, the input voltage to control the Z _R axis rotating motor is V1 and the input voltage to control the X _R axis rotating motor is V2, so that the Z _R axis rotating motor rotates and drives the millimeter wave radar around the Z _R axis. The angle of rotation Δα, and the angle by which the X _R axis is rotated to drive the millimeter-wave radar to rotate Δβ around the X _R axis.

3) After each attitude adjustment of the millimeter-wave radar by the computing device (for example, the millimeter-wave radar rotates Δα around the Z _R axis and rotates Δβ around the X _R axis), the computing device re-detects the attitude of the millimeter-wave radar and judges the millimeter-wave radar. Whether the left and right rotation degrees of freedom and up and down swing degrees of freedom of the radar meet the above-mentioned criteria for closed-loop angle control; if so, the computing device determines that the attitude adjustment is completed; if not, the computing device continues to adjust the attitude of the millimeter-wave radar, and this cycle , until both the left and right rotation degrees of freedom and the up and down swing degrees of freedom of the millimeter-wave radar meet the corresponding angular closed-loop control standards.

After the attitude adjustment of the millimeter-wave radar is completed, the left and right rotation degrees of freedom and the up and down swing degrees of freedom of the millimeter-wave radar meet the corresponding angle closed-loop control standards, and the millimeter-wave radar can accurately detect the target again.

Optionally, after each attitude adjustment of the millimeter-wave radar by the computing device (for example, each time the radar is controlled to rotate around the Z- _R axis and/or rotate around the X- _R axis), the computing device updates the millimeter-wave radar in real time according to the adjustment angle value of the millimeter-wave radar. The calibration matrix of the wave radar ensures the reliability of the information fusion of the millimeter wave radar and the camera during the attitude adjustment of the millimeter wave radar.

As shown in Figure 11, it is a schematic diagram of the millimeter-wave radar coordinate system and the camera coordinate system. Among them, the millimeter-wave radar coordinate system can describe the relative position of the object and the millimeter-wave radar, expressed as (X _R , Y _R , Z _R ), and the camera coordinate system can describe the relative position of the object and the camera, expressed as (X _C , Y _C , Z _C ). Since the camera and the millimeter-wave radar are installed at different positions of the vehicle, the position data of each feature point collected by the camera is the coordinate of each feature point in the camera coordinate system, and the position data of each feature point collected by the millimeter-wave radar is the feature point. For the coordinates in the millimeter-wave radar coordinate system, the same object has different coordinate parameters in the camera coordinate system and the millimeter-wave radar coordinate system, so a calibration matrix is required to correspond the data collected by the millimeter-wave radar and the data collected by the camera to the same one In the coordinate system, it is convenient for the computing device to perform data operations.

For example, the computing device can convert the data collected by the millimeter-wave radar into the camera coordinate system, and the calibration matrix can include the rotation matrix R, translation matrix T, etc. required for conversion between the millimeter-wave radar coordinate system and the camera coordinate system. .

For example, the computing device can convert both the data collected by the millimeter-wave radar and the data collected by the camera into the world coordinate system (or image coordinate system or other coordinate system), then the calibration matrix can include the coordinate system for the millimeter-wave radar and the world coordinate system. Rotation matrix R, translation matrix T, etc. required for conversion between coordinate systems (or image coordinate systems or other coordinate systems).

From the above, it can be seen that the embodiment of the present application combines the road conditions characteristics of the road, and when the millimeter-wave radar loses the target or is about to lose the target, the millimeter-wave radar has two degrees of freedom in the left and right (α) and up and down (β) aspects of the millimeter-wave radar. The attitude is compensated and corrected, and the two degrees of freedom are independently adjusted and not dependent on each other, which can improve the accuracy and efficiency of the attitude adjustment of the millimeter-wave radar, thereby further improving the accuracy and efficiency of target detection and tracking. In addition, when adjusting the angle of the millimeter-wave radar, the embodiment of the present application can also compensate and correct the calibration matrix of the millimeter-wave radar in real time, so as to achieve a fast and accurate response of information fusion between the millimeter-wave radar and the camera, and ensure the millimeter-wave radar during the attitude adjustment process of the millimeter-wave radar. Reliability of radar and camera information fusion.

It should be noted that the above takes the curved road conditions as an example to introduce the process of adjusting the attitude of the millimeter-wave radar by the first vehicle when the millimeter-wave radar is in the target loss state or the target is about to be lost (or the millimeter-wave radar is in the frame-loss state). In practical applications, the attitude of the millimeter-wave radar can also be adjusted in the same way for other road scenarios (such as up/downhill, acceleration/deceleration, etc.), which is not limited in this application.

As shown in FIG. 12 , an embodiment of the present application also provides a target tracking method based on the fusion of millimeter-wave radar information and camera information. The target tracking method can be applied to the vehicle shown in FIG. 4 , including:

S1201. The first vehicle acquires a red green blue (Red Green Blue, RGB) image collected by a camera, and performs rough fitting on the lane lines in the far field of view included in the RGB image.

Specifically, the computing device of the first vehicle acquires the RGB image collected by the camera, analyzes the RGB image, and performs rough fitting on the lane lines in the far field of view included in the RGB image. Alternatively, the processing chip in the camera analyzes the RGB image, performs rough fitting on the lane lines in the far field of view included in the RGB image, and then transmits the fitting result to the computing device.

The rough fitting here refers to projecting certain pixels of the lane line onto the bird's-eye view, and extracting several points at equal intervals to determine the turning point and direction of the curve without spending a lot of computing power and consumption. Time to fit complex equations such as cubic curves of curves, and do not need to do a lot of lane lines and smooth post-processing.

Since the detection range of the radar in the field of view is larger than that in the far field of view, the target (such as the second vehicle) is generally not easily lost in the near field of view. On the premise of ensuring accuracy, the amount of calculation is reduced and the calculation efficiency is improved.

Optionally, the far field of view is a position where the distance from the vehicle exceeds a preset distance (eg, 50m, 60m, or 100m, etc.). During specific implementation, the preset distance may be determined according to the characteristics of the detection beam angle and detection distance of the millimeter wave radar.

S1202, the first vehicle simultaneously performs target recognition based on the millimeter-wave radar and the camera.

On the one hand, the computing device of the first vehicle performs target recognition on the RGB image obtained by the camera to obtain the first recognition result; or, the processing chip in the camera performs target recognition on the RGB image, and after obtaining the first recognition result, the first recognition result is obtained. The results are transmitted to the computing device (ie S1202.1, camera target recognition).

On the other hand, the computing device of the first vehicle performs target recognition on the radar trace data obtained by the millimeter-wave radar to obtain the second recognition result; or, the processing chip in the millimeter-wave radar performs target recognition on the radar trace data to obtain the first recognition result. After the second identification result, the second identification result is transmitted to the computing device (ie S1202.2, millimeter wave radar target identification).

In the embodiment of the present application, for the camera target recognition, the computing device saves a trained target recognition model corresponding to it. After obtaining the RGB image collected by the camera, the computing device inputs the RGB image into the target recognition model corresponding to the target recognition of the camera, so as to obtain the target recognition result corresponding to the camera, that is, the first recognition result. After the computing device obtains the radar spot trace data collected by the millimeter-wave radar, it can use the neural network model or traditional clustering and tracking algorithms to process the radar spot data, and the millimeter-wave radar can obtain the target recognition result corresponding to the millimeter-wave radar. , that is, the second recognition result.

As shown in FIG. 12 , the first target recognition result is represented as (Xc, Yc, C) ^T , where Xc and Zc represent the position data of the target, and C represents the type of the target. Let the second target recognition result be represented as (X _R , Y _R , v, w) ^T , where X _R and Y _R represent the position data of the target, v represents the linear velocity of the target, and w represents the angular velocity of the target. It should be noted that the position data of the target recognition of the millimeter-wave radar and the target recognition of the camera are based on the position data in the two-dimensional plane as an example, that is, the first recognition result only includes the data in the two directions of Xc and Yc, and the third 2. The identification result only includes the data in the two directions of X _R and Y _R. During specific implementation, the position data of the target recognition of the millimeter-wave radar and the target recognition of the camera may also include more parameters, which are not limited here.

Optionally, for the camera recognition in S1202.1 in this embodiment of the present application, different target recognition models may be designed according to different recognition scenarios. For example, in the case of frame loss of the millimeter-wave radar, the computing device uses a lightweight recognition model to identify the data collected by the camera, and the lightweight recognition model focuses on the recognition speed (that is, the algorithm processing delay is small). For example, lightweight recognition The recognition model can use You Only Look Once (YOLO) v3 (version number, representing the third edition). In the case that the millimeter-wave radar does not lose frames, the weighted recognition model is used to identify the data collected by the camera, and the weighted recognition model focuses on the accuracy of the algorithm (that is, the accuracy of the position, speed and type of the target output by the algorithm should be High), so that the computing device can adjust the attitude of the millimeter-wave radar as soon as possible based on the type and position of the target, and re-detect the target. For example, the weighted recognition model can use a regional convolutional neural network (RCNN).

S1203, the first vehicle performs time alignment and target alignment on the first recognition result and the second recognition result.

The so-called time alignment is to perform time synchronization between the first recognition result and the second recognition result. The so-called target alignment is to perform spatial synchronization between the first recognition result and the second recognition result, for example, converting the position data into the same coordinate system as described in the above-mentioned embodiment of FIG. 11 .

S1204, the computing device performs target fusion according to the IOU fusion rule.

In the embodiment of the present application, after the computing device of the first vehicle performs target recognition on the RGB image to obtain the first recognition result, it can also generate a region of interest (ROI) (that is, a target frame) in the RGB image; And, after the first recognition result is obtained by performing target recognition on the radar spot trace data, the ROI may also be generated in the radar spot trace data. Among them, ROI refers to the area that needs to be processed from the processed image in the form of box, circle, ellipse, irregular polygon, etc. in machine vision and image processing, which is called the region of interest. In the embodiment of this application, the ROI is The area where the target (ie the second vehicle) is located.

The ROI in the embodiment of the present application takes a rectangle as an example. As shown in FIG. 13 , it is the ROI corresponding to the detection result of the millimeter wave radar (the rectangular area indicated by A) and the ROI (the rectangular area indicated by B) corresponding to the detection result of the camera. schematic diagram.

The IoU is the ratio of the intersection and union of the ROI based on the millimeter wave radar detection output and the ROI based on the camera detection output. As shown in Figure 14, it is a schematic diagram of the intersection of the millimeter wave radar detection result A and the camera detection result B. IoU is the ratio of the intersection and union of the two rectangular areas, that is: IoU=(A∩B)/(A∪ B).

The value of IoU should be between [0,1]. The larger the value of IoU, the higher the probability that the target detected based on the camera and the target detected based on the radar are the same target. On the contrary, the smaller the value of IoU, the target detected based on the camera and the target detected based on the radar are The probability of the same target is lower.

In the embodiment of the present application, the IOU fusion rule may include: if IoU>M (or IoU≥M), the computing device fuses the first recognition result and the second recognition result, and outputs the third recognition result. The data that is the third recognition result is expressed as (X*, Y*, C*, v*, w*) ^T ; if IoU≤M (or IoU<M), then the computing device does not combine the first recognition result and the second The recognition results are fused, and the first recognition result and the second recognition result are output. Among them, M is the set second threshold, and the value range is between (0, 1).

Optionally, the second threshold M may be related to the curvature ρ of the curve where the first vehicle and/or the second vehicle is located, the driving speed V of the first vehicle, and the driving distance L of the first vehicle (that is, the distance to the second vehicle). ) and other factors.

Exemplarily, the second threshold M and the curvature ρ of the curve, the driving speed V, and the driving distance L satisfy the following formulas:

M=a ² ρ+bV+L;

Among them, a and b are coefficients, which can be set by technicians according to experiments or experience.

It should be understood that the above formula is only an example and not a limitation. In a specific implementation, the second threshold K may also be related to other factors, such as the acceleration of the first vehicle.

S1205.1. When the radar is not in the target lost state or the target is about to be lost, the first vehicle executes the millimeter-wave radar target not lost tracking mechanism: the computing device of the first vehicle performs target tracking based on continuous multi-frame fusion data;

For example, when the radar in FIG. 12 is not in the target loss state or the target is about to be lost, the computing device performs target tracking based on consecutive multiple frames (X*, Y*, C*, v*, w*) ^T.

S1205.2. When the radar is in the target lost state or the target is about to be lost, the first vehicle executes the millimeter-wave radar target not lost tracking mechanism: the computing device of the first vehicle is based on the radar in the target lost state or the target is about to be lost. Target tracking is performed with the recognition result and the last frame or multi-frame fusion data before the radar is in the target lost state or the target is about to be lost.

For example, when the radar in Fig. 12 is in the state of target loss or the target is about to be lost, the computing device fuses the data based on the last frame before the target is lost (that is, the original frame (X*, Y*, C* _, v*, w*) ^T ) and the first recognition result after the target is lost (ie, consecutive frames (X _C , Y _C , C) ^T ) to track the target.

It can be seen from the above that the embodiment of the present application only fits the lane line thickness in the far field of view, which can reduce the amount of calculation and improve the calculation efficiency on the premise of ensuring the accuracy. Secondly, the embodiments of the present application design at least two target recognition models for camera target recognition. In the case of frame loss of millimeter-wave radar, a lightweight recognition model is used to perform target recognition on the data collected by the camera to improve the recognition speed. When the wave radar does not lose frames, the weighted recognition model is used to identify the data collected by the camera to improve the recognition accuracy, thereby achieving both the speed and accuracy of fusion recognition. In addition, the embodiment of the present application also designs the second threshold K of the IoU in combination with the curve scene, which can further improve the recognition accuracy.

It should be noted that this paper takes the blind spot of the curve as an example to introduce the solution of how to detect whether the target is lost and how to re-acquire the detected target, but in practical applications, the technical solutions of the embodiments of the present application are also applicable to other similar blind spot scenarios. For example, in a ramp scene, due to the occlusion of the ramp, the vehicle will also have a blind spot, so the millimeter-wave radar can also be controlled to achieve the degree of freedom of its up and down swing. Or, for example, uneven road sections or the start/stop time of a car also have a great influence on the pitch angle of the millimeter-wave radar, so the millimeter-wave radar can also be controlled to achieve the degree of freedom of its up and down swing.

For example, see FIG. 15, which is a scene where the vehicle is on an uphill slope. At time t0, both the first vehicle and the second vehicle are on a flat road, and the second vehicle is within the detection range of the millimeter-wave radar of the first vehicle; at time t1, the second vehicle is driving on the uphill. The attitude is consistent with the attitude at time t0, so due to the existence of the ramp, the second vehicle exceeds (or is about to exceed) the detection range of the millimeter-wave radar of the first vehicle. Referring to Figure 16, after the second vehicle exceeds (or is about to exceed) the detection range of the millimeter-wave radar of the first vehicle, the computing device of the first vehicle adjusts the attitude of the millimeter-wave radar, so that the pitch angle of the millimeter-wave radar is adjusted upward, and then the first The millimeter-wave radar of one vehicle re-detects the second vehicle.

It should be further noted that the embodiments of this application mainly take millimeter-wave radar as an example to introduce the scheme of how to detect target loss and how to re-detect the target in special scenarios such as curves, ramps, acceleration and deceleration, but in practical applications Among them, the millimeter-wave radar can also be replaced with other sensors, such as laser millimeter-wave radar, etc., especially in the mass production scheme of autonomous driving, after the solid-state laser millimeter-wave radar is adopted, the laser millimeter-wave radar also has a definition similar to the field of view angle. Therefore, in the layout of a certain number of sensors, there will inevitably be blind areas with a certain detection range, so the same technical solution can also be applied to other similar sensors such as laser millimeter wave radar. In other words, as long as a sensor with a large detection range, short detection distance, and inaccurate detection results (obstacle position and speed, etc.) is used in conjunction with a sensor with a small detection range, long detection distance, and accurate detection results, there will be detection blind spots. The technical solutions of the embodiments of the present application are applicable.

The above-mentioned various implementation manners of the embodiments of the present application may be combined with each other to achieve different technical effects.

Based on the same technical concept, referring to FIG. 17 , which is a system architecture diagram provided in an embodiment of the present application, the system can execute the solutions in the foregoing method embodiments. The system includes: a positioning device, a camera, a millimeter-wave radar, a follow-up mechanism, and a computing device. The system is applicable to the vehicle shown in FIG. 4 , and is used to implement the above-mentioned target detection method in the embodiment of the present application.

The positioning device is used to obtain the positioning information of the vehicle.

A camera to obtain visual information in front of the vehicle.

Millimeter-wave radar, used to obtain the information of the millimeter-wave radar in front of the vehicle. It should be understood that the target detection in the present application takes the combination of a camera and a millimeter-wave radar as an example. However, this application does not limit the specific type of sensor, as long as it is a sensor with a large detection range, a short detection distance, and inaccurate detection results (obstacle position and speed, etc.), and a sensor with a small detection range, a long detection distance, and accurate detection results. When used in combination, all the embodiments of the present application are applicable.

The follower mechanism is set on the millimeter-wave radar and used to control the attitude of the millimeter-wave radar.

A computing device is a device with computing/processing capabilities. Specifically, it may be a vehicle-mounted device, or one or more processing chips, or an integrated circuit, etc., which is not limited here.

Referring to FIG. 17, according to the division of logical functions, the computing device may include the following modules:

The road condition identification module is used to identify the road conditions where the vehicle and the preceding vehicle are located according to the positioning information collected by the positioning device and the visual information collected by the camera; and according to the road conditions, start the corresponding threshold comparator (where different road conditions correspond to different threshold comparisons) Figure 12 takes the threshold comparators A, B, and C as an example, the actual number can be more or less).

The threshold comparator is used to determine whether the speed of the target obstacle (including the linear velocity or the angular velocity) exceeds the threshold of the threshold comparator; the specific implementation of each threshold comparator can refer to the embodiments shown in FIGS. 6 to 8 above. Related introduction.

Wherein, the image mark in FIG. 12 is an RGB image that carries a target mark (such as a target frame) output by the camera when the threshold comparator confirms as "Yes".

The millimeter-wave radar camera fusion target detection module is used to fuse the detection results of the millimeter-wave radar and the detection results of the camera. For the specific implementation, please refer to the relevant introduction in the embodiment shown in FIG. 12 .

The target loss tracking module is used to implement the millimeter wave radar target loss tracking mechanism. For details, please refer to the relevant introduction in S1205.1 above.

The target not lost tracking module is used to implement the millimeter wave radar target not lost tracking mechanism. For details, please refer to the relevant introduction in S1205.2 above.

Among them, the final output result can have various forms. For example, output the specific position, type, speed and other data of the target (such as X*, Y*, C*, v*, w*, etc.), or output the RGB image with the target frame, etc., there are no restrictions here.

Based on the same technical concept, an embodiment of the present application further provides a target detection device 180, where the device 180 is located inside the first vehicle, for example, a chip arranged inside the first vehicle. The apparatus 180 includes modules or units or means (means) for executing the steps in the methods shown in FIG. 5 , FIG. 10 , and FIG. 12 . The functions or units or means may be implemented by software, or by hardware, or The corresponding software implementation can be performed by hardware.

As shown in FIG. 18 , the apparatus 180 may include: a detection module 1801 for detecting the speed of the second vehicle based on a millimeter-wave radar; wherein the second vehicle is located in front of the first vehicle, and the speed includes a linear velocity and an angular velocity; a processing module 1802 , used to obtain the road conditions where the first vehicle and the second vehicle are located; the first threshold is determined according to the road conditions; if the speed of the second vehicle exceeds the first threshold, it is determined that the millimeter-wave radar is in the target loss state or the target is about to be lost.

In this embodiment of the present application, the device 180 determines a first threshold value according to road conditions, and compares the speed of the second vehicle with the first threshold value, thereby determining whether the millimeter-wave radar is in a target loss state or a target loss state. In this way, under special road conditions (such as curves, ramps, etc.), the device 180 can timely and effectively identify the loss of the millimeter-wave radar target caused by the road conditions, so as to avoid the first vehicle from generating the illusion that there is no target obstacle, and further Improve the safety and stability of the vehicle during driving.

Optionally, when acquiring the road conditions where the first vehicle and the second vehicle are located, the processing module 1802 is specifically configured to: use a camera to acquire an RGB image in front of the first vehicle, and then determine the location where the second vehicle is located according to the RGB image. road conditions. In this way, the road condition ahead can be identified efficiently and quickly, that is, the road condition where the second vehicle is located can be determined.

Optionally, when determining the road condition where the second vehicle is located according to the RGB image, the processing module 1802 is specifically configured to: extract feature points from the lane lines in the far field of view included in the RGB image, and then determine according to the extracted feature points. The inflection point and direction of the road where the second vehicle is located, so as to obtain the road condition where the second vehicle is located. Since the detection range of the millimeter-wave radar in the near field of view is larger than that in the far field of view, the target in the near field of view is generally not easy to be lost. The processing module 1802 only processes the lane lines in the far field of view, which can ensure accuracy. Under the premise, the amount of calculation is reduced and the calculation efficiency is improved.

Optionally, when acquiring the road conditions where the first vehicle and the second vehicle are located, the processing module 1802 may also acquire location information of the first vehicle based on the positioning device, and then determine the road conditions where the first vehicle is located according to the location information. In this embodiment of the present application, the positioning device may be a GPS, a Beidou system, or other positioning systems, and is used for receiving satellite signals and positioning the current position of the first vehicle. In addition, the positioning system may also be visual positioning, millimeter wave radar positioning, fusion positioning, etc., which is not limited in this application. After obtaining the location information, the processing module 1802 determines the road conditions where it is located based on the map, for example, in a curve, a straight road, an uphill, or a downhill.

In this way, the device 180 can quickly and accurately obtain the road conditions where the first vehicle is located.

It should be understood that the first threshold may be determined according to a speed threshold between the second vehicle being within the detection range of the millimeter wave radar and the speed threshold being outside the detection range of the millimeter wave radar, for example, the first threshold is less than or equal to the speed threshold. When the speed of the second vehicle is much greater than the speed threshold, the second vehicle exceeds the detection range of the millimeter-wave radar and enters the blind spot for detection of the millimeter-wave radar, so the millimeter-wave radar is in a target loss state; when the speed of the second vehicle is at the critical speed When the value is near, the second vehicle may exceed the detection range of the millimeter-wave radar at any time, and is about to enter the blind spot of the millimeter-wave radar detection, so the millimeter-wave radar is in a state where the target is about to be lost. In this embodiment of the present application, the design of the first threshold may be different according to different road conditions.

Three specific examples are given below to illustrate.

Example 1. The road conditions where the first vehicle and the second vehicle are located are: the first vehicle is on a straight road, and the second vehicle is on a curve. Then, when the detection module 1801 detects the speed of the second vehicle based on the millimeter-wave radar, it is specifically used for:

The instantaneous angular velocity of the circular motion of the second vehicle around the curve is detected based on the millimeter-wave radar; wherein, the first threshold N determined by the processing module 1802 according to the road conditions conforms to:

Example 2. The road conditions where the first vehicle and the second vehicle are located are: the first vehicle is on a curve, and the second vehicle is on a straight road. Then, when the detection module 1801 detects the speed of the second vehicle based on the millimeter-wave radar, it is specifically used for:

Detecting the instantaneous speed of the second vehicle on the straight road based on millimeter-wave radar;

Wherein, the first threshold N determined by the processing module 1802 according to the road conditions conforms to:

in,

Example 3. The road conditions where the first vehicle and the second vehicle are located are: both the first vehicle and the second vehicle are in a curve. Then, when the detection module 1801 detects the speed of the second vehicle based on the millimeter-wave radar, it is specifically used for:

Detecting the instantaneous angular velocity of the second vehicle in circular motion around the curve based on millimeter-wave radar;

It should be understood that the above three are only examples and not limitations. In practical applications, road conditions are not limited to curve scenarios. For other road scenarios (such as up/downhill, acceleration/deceleration, etc.), the same idea can also be used to design preset conditions.

Optionally, the processing module 1802 may also be used to: after determining that the millimeter-wave radar is in a target loss state or a target is about to be lost, control the millimeter-wave radar to rotate around the Z _R axis and/or around the X _R axis according to road conditions, to The millimeter wave radar is made to detect the second vehicle again; wherein, the Z _R axis is perpendicular to the horizontal plane, and the X _R axis is parallel to the horizontal plane and perpendicular to the traveling direction of the first vehicle.

It should be understood that the degrees of freedom of the above two directions (that is, the direction of rotation around the Z _R axis and the direction of rotation around the X _R axis) are independent of each other (or decoupled) in the mechanical structure and independent of each other in the control logic. (or decoupling), so the processing module 1802 can control and adjust only one of the degrees of freedom, or can control and adjust two degrees of freedom at the same time.

For example, when the first vehicle and/or the second vehicle is in a curve, the processing module 1802 may control the millimeter-wave radar to rotate around the Z _R axis by a first angle. For another example, when the first vehicle and/or the second vehicle is on a ramp, the processing module 1802 may control the millimeter wave radar to rotate around the X _R axis by a second angle. For another example, when the first vehicle and/or the second vehicle is in a combination road section with curves and slopes, the processing module 1802 may control the millimeter-wave radar to rotate around the Z _R axis by a third angle and around the X _R axis by a fourth angle. Since the rotation of the millimeter-wave radar around the Z _R axis and the rotation around the X _R axis are independent of each other and not dependent on each other, the accuracy and efficiency of the attitude adjustment of the millimeter-wave radar can be improved, thereby further improving the accuracy and efficiency of target detection and tracking.

The processing module 1802 may further update the calibration matrix of the millimeter-wave radar in real time according to the angle value of the rotation after each time the millimeter-wave radar is controlled to rotate around the Z _R axis and/or around the X _R axis. In this way, a fast and accurate response of the information fusion of the millimeter-wave radar and the camera can be achieved, and the reliability of the information fusion of the millimeter-wave radar and the camera during the attitude adjustment of the millimeter-wave radar can be guaranteed.

Optionally, the processing module 1802 can also be used to: detect and track the target based on the camera and the millimeter-wave radar respectively, and perform target fusion according to the IOU fusion rule.

The specific target fusion process includes:

The processing module 1802, after acquiring the RGB image in front of the first vehicle based on the camera, performs target recognition on the RGB image based on the target recognition model to obtain a first recognition result, wherein the first recognition result includes the position and type of the second vehicle, and the target recognition The input of the model is an RGB image, and the output is the position and type of the target; after the processing module 1802 obtains the radar spot data in front of the first vehicle based on the millimeter-wave radar, it processes the radar spot data to obtain a second recognition result, The second recognition result includes the position and speed of the second vehicle; when the processing module 1802 determines that the area where the second vehicle is located in the RGB image and the IoU of the area where the second vehicle is located in the radar trace data is greater than the second threshold M, the first The recognition result and the second recognition result are fused to obtain fusion data, otherwise the first recognition result and the second recognition result are not fused.

Wherein, the second threshold M and the curvature ρ of the curve where the first vehicle and/or the second vehicle are located, the driving speed V of the first vehicle, and the driving distance L of the first vehicle satisfy the following relationships:

M=a ² ρ+bV+L;

Among them, a and b are preset coefficients.

Since the second threshold K is related to the road conditions where the vehicle is located (the curvature ρ of the curve where the first vehicle and/or the second vehicle is located) and the driving state of the first vehicle (that is, the driving speed V, the driving distance L), it can be Improve the accuracy of fusion recognition, thereby further improving the safety and stability of the vehicle during driving.

Optionally, the processing module 1802 can also be used to: when the millimeter-wave radar is in the target-losing state or the target is about to be lost, based on the fusion data before the millimeter-wave radar is in the target-losing state or the target is about to be lost, and the millimeter-wave radar is in the target-losing state or the target is about to be lost. The second vehicle is tracked by the first recognition result when the target is lost or the target is about to be lost; or when the millimeter-wave radar is not in the target-loss state or the target is about to be lost, the second vehicle is tracked based on continuous multi-frame fusion data . Since the device 180 adopts different tracking mechanisms according to different millimeter wave radar states, the accuracy of target tracking can be improved, thereby further improving the safety and stability of the vehicle during driving.

Optionally, when the processing module performs target recognition on the RGB image based on the target recognition model, it is specifically used for: when the millimeter-wave radar is in the target loss state or the target is about to be lost, use the lightweight target recognition model to collect the RGB images collected by the camera. The image is used for target recognition; when the millimeter-wave radar is not in the target loss state or the target is about to be lost, the weighted target recognition model is used to perform target recognition on the RGB images collected by the camera; among them, the lightweight target recognition model The recognition speed is greater than The recognition speed of the weighted target recognition model and the recognition accuracy of the lightweight target recognition model are lower than the recognition accuracy of the heavyweight target recognition model.

For camera target recognition, two different target recognition models are designed to switch according to different scenarios (in the case of frame loss of millimeter wave radar, a lightweight recognition model is used to recognize the data collected by the camera to improve the recognition speed. When the millimeter-wave radar does not lose frames, the weighted recognition model is used to identify the data collected by the camera to improve the recognition accuracy), which can achieve both the speed and accuracy of fusion recognition.

It should be understood that the above only takes the switching of two models as an example, and in practical applications, more modules can be designed to perform switching.

It should be noted that, for the specific implementation of the method steps performed by each module in the foregoing apparatus 180, reference may be made to the specific implementation of the corresponding method steps performed by the first vehicle in the foregoing method embodiments, which will not be repeated here.

Based on the same technical concept, referring to FIG. 19 , an embodiment of the present application further provides a target detection apparatus 190, including a processor 1901 and a memory 1902; wherein, the memory 1902 stores instructions that can be executed by the processor 1901, and the processor 1901 passes Executing the instructions stored in the memory 1902 causes the apparatus 190 to execute the methods shown in FIG. 5 , FIG. 10 , and FIG. 12 . The processor 1901 and the memory 1902 may be coupled through an interface circuit, or may be integrated together, which is not limited here.

The specific connection medium between the processor 1901 and the memory 1902 is not limited in this embodiment of the present application. In the embodiment of the present application, the processor 1901 and the memory 1902 are connected through a bus in FIG. 19 , and the bus is represented by a thick line in FIG. 19 . The connection mode between other components is only for schematic illustration, and is not cited. limited. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is shown in FIG. 19, but it does not mean that there is only one bus or one type of bus.

It should be understood that the processor mentioned in the embodiments of the present application may be implemented by hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, or the like. When implemented in software, the processor may be a general-purpose processor implemented by reading software codes stored in memory.

Exemplarily, the processor may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC) , Off-the-shelf Programmable Gate Array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

It should be understood that the memory mentioned in the embodiments of the present application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Wherein, the non-volatile memory may be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically programmable read-only memory (Erasable PROM, EPROM). Erase programmable read-only memory (Electrically EPROM, EEPROM) or flash memory. Volatile memory may be Random Access Memory (RAM), which acts as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Eate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) ) and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, the memory (storage module) can be integrated in the processor.

It should be noted that the memory described herein is intended to include, but not be limited to, these and any other suitable types of memory.

Based on the same technical concept, the embodiments of the present application also provide a computer-readable storage medium, including a program or an instruction, when the program or instruction is run on a computer, the methods shown in FIG. 5 , FIG. 10 , and FIG. 12 can be implement.

Based on the same technical concept, an embodiment of the present application further provides a chip, which is coupled to a memory and used to read and execute program instructions stored in the memory to implement the methods shown in FIG. 5 , FIG. 10 , and FIG. 12 .

Based on the same technical concept, the embodiments of the present application also provide a computer program product containing instructions. The computer program product stores instructions. When the computer program product runs on a computer, the computer can execute the instructions shown in FIG. 5 , FIG. 10 , and FIG. 12 . method shown.

Based on the same technical concept, an embodiment of the present application also provides a vehicle, the vehicle includes a target detection device, a millimeter-wave radar, and a camera; The method shown in Figure 12. The structure of the vehicle may be as shown in FIG. 11 .

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

A target detection method, characterized in that, applied to a first vehicle, the method comprising:

Detecting a speed of a second vehicle based on a millimeter-wave radar; wherein, the second vehicle is located in front of the first vehicle, and the speed includes a linear speed and an angular speed;

acquiring road conditions where the first vehicle and the second vehicle are located, and determining a first threshold according to the road conditions;

If the speed of the second vehicle exceeds the first threshold, it is determined that the millimeter-wave radar is in a target loss state or a target loss state.
The method of claim 1, wherein acquiring the road conditions where the first vehicle and the second vehicle are located comprises:

The RGB image in front of the first vehicle is acquired based on the camera, and the road condition where the second vehicle is located is determined according to the RGB image;

The location information of the first vehicle is acquired based on the positioning device, and the road condition where the first vehicle is located is determined according to the location information.
The method according to claim 2, wherein the determining the road condition where the second vehicle is located according to the RGB image comprises:

Feature points are extracted from the lane lines in the far field of view included in the RGB image, and the inflection point and direction of the road where the second vehicle is located is determined according to the extracted feature points.
The method according to any one of claims 1-3, wherein the road conditions where the first vehicle and the second vehicle are located are: the first vehicle is on a straight road, and the second vehicle is on a curved road road;

Detect the speed of the second vehicle based on millimeter wave radar, including:

Detecting the instantaneous angular velocity of the circular motion of the second vehicle around the curve based on a millimeter-wave radar;

The first threshold N determined according to the road conditions conforms to:

Wherein, the ω r is the angular velocity of the second vehicle making a circular motion around the curve from time t0 to time t1,
is the Euclidean distance between the first vehicle and the second vehicle at time t0;
is the displacement of the second vehicle from time t0 to time t1, α is half of the detection beam angle of the millimeter-wave radar, and the time t0 is the time when the millimeter-wave radar collects the first frame of data, so The time t1 is the time when the millimeter-wave radar collects the second frame of data, the first frame of data and the second frame of data are two consecutive frames of data; K is a coefficient greater than 0 and less than or equal to 1; ε is The angle value of the central angle formed between the position A of the first vehicle, the position B of the second vehicle, and the center O of the curve at time t0.
The method according to any one of claims 1-3, wherein the road conditions where the first vehicle and the second vehicle are located are: the first vehicle is in a curve, and the second vehicle is in a curve straight;

Detect the speed of the second vehicle based on millimeter wave radar, including:

Detecting the instantaneous traveling speed of the second vehicle on the straight road based on the millimeter-wave radar;

The first threshold N determined according to the road conditions conforms to:

in,
is the displacement of the second vehicle from time t0 to time t1; v r is the running speed of the second vehicle on the straight road from time t0 to time t1; the time t0 is the millimeter-wave radar The time when the first frame of data is collected, the time t1 is the time when the millimeter-wave radar collects the second frame of data, the first frame of data and the second frame of data are two consecutive frames of data; P is greater than 0 and a factor less than or equal to 1.
The method according to any one of claims 1-3, wherein the road conditions where the first vehicle and the second vehicle are located are: both the first vehicle and the second vehicle are in a curve ;

Detect the speed of the second vehicle based on millimeter wave radar, including:

Detecting the instantaneous angular velocity of the circular motion of the second vehicle around the curve based on a millimeter-wave radar;

The first threshold N determined according to the road conditions conforms to:

Among them, ω r is the angular velocity of the second vehicle in circular motion around the curve from time t0 to time t1; v is the instantaneous speed of the first vehicle; α is the detection beam angle of the millimeter-wave radar half, the time t0 is the time when the millimeter-wave radar collects the first frame of data, the time t1 is the time when the millimeter-wave radar collects the second frame of data, the first frame of data and the second frame of data Frame data is two consecutive frames of data; Q is a coefficient greater than 0 and less than or equal to 1;
is the Euclidean distance between the first vehicle and the second vehicle at time t0.
The method according to any one of claims 1-6, wherein after determining that the millimeter-wave radar is in a target loss state or a target loss state, the method further comprises:

According to the road conditions, the millimeter wave radar is controlled to rotate around the Z R axis and/or around the X R axis, so that the millimeter wave radar can re-detect the second vehicle; wherein the Z R axis is perpendicular to The horizontal plane, the X R axis is parallel to the horizontal plane and perpendicular to the traveling direction of the first vehicle.
The method according to claim 7, wherein, according to the road conditions, controlling the millimeter-wave radar to rotate around the Z R axis and/or around the X R axis, comprising:

When the first vehicle and/or the second vehicle is in a curve, control the millimeter-wave radar to rotate around the Z R axis by a first angle; or,

When the first vehicle and/or the second vehicle is on a slope, control the millimeter-wave radar to rotate around the X- R axis by a second angle; or,

When the first vehicle and/or the second vehicle is in a curved and slope combined road section, the millimeter-wave radar is controlled to rotate around the Z R axis by a third angle and around the X R axis by a fourth angle.
The method of claim 7 or 8, wherein the method further comprises:

After each time the millimeter wave radar is controlled to rotate around the Z R axis and/or around the X R axis, the calibration matrix of the millimeter wave radar is updated in real time according to the angle value of the rotation.
The method according to any one of claims 1-9, wherein the method further comprises:

Obtaining an RGB image in front of the first vehicle based on a camera, and performing target recognition on the RGB image based on a target recognition model to obtain a first recognition result, where the first recognition result includes the location and type of the second vehicle; Wherein, the input of the target recognition model is an RGB image, and the output is the position and type of the target;

The radar spot data in front of the first vehicle is obtained based on the millimeter-wave radar, and the radar spot data is processed to obtain a second recognition result; the second recognition result includes the position of the second vehicle and speed;

When it is judged that the intersection ratio IoU of the area where the second vehicle is located in the RGB image and the area where the second vehicle is located in the radar trace data is greater than the second threshold M, the first recognition result and the The second recognition result is fused to obtain fused data;

Wherein, the second threshold value M and the curvature ρ of the curve where the first vehicle and/or the second vehicle are located, the driving speed V of the first vehicle, and the driving distance L of the first vehicle satisfy the following: relation:

M=a 2 ρ+bV+L;

Among them, a and b are preset coefficients.
The method of claim 10, wherein the method further comprises:

When the millimeter-wave radar is in the target-losing state or the target is about to be lost, based on the fusion data before the millimeter-wave radar is in the target-losing state or the target is about to lose the state, and the millimeter-wave radar is in the target-losing state or the target is about to be lost The second vehicle is tracked with the first recognition result in the lost state; or,

When the millimeter-wave radar is not in the target loss state or the target is about to be lost, the second vehicle is tracked based on continuous multi-frame fusion data.
The method according to claim 10, wherein, performing target recognition on the RGB image based on a target recognition model, comprising:

When the millimeter-wave radar is in a target loss state or a target is about to be lost, use a lightweight target recognition model to perform target recognition on the RGB images collected by the camera;

When the millimeter-wave radar is not in the target loss state or the target is about to be lost, use a heavyweight target recognition model to perform target recognition on the RGB images collected by the camera;

Wherein, the recognition speed of the lightweight target recognition model is greater than the recognition speed of the heavyweight target recognition model, and the recognition accuracy of the lightweight target recognition model is lower than the recognition accuracy of the heavyweight target recognition model.
A target detection device, characterized in that the device is located in a first vehicle, and the device comprises:

a detection module for detecting the speed of a second vehicle based on a millimeter-wave radar; wherein, the second vehicle is located in front of the first vehicle, and the speed includes a linear speed and an angular speed;

a processing module, configured to acquire the road conditions where the first vehicle and the second vehicle are located; determine a first threshold according to the road conditions, and determine the first threshold if the speed of the second vehicle exceeds the first threshold The millimeter-wave radar is in the target loss state or the target is about to be lost.
A target detection device, comprising a processor and a memory;

The memory stores instructions executable by the processor, and the processor executes the instructions stored in the memory to cause the apparatus to perform the method according to any one of claims 1-12.
A computer-readable storage medium, characterized by comprising programs or instructions, which, when executed on a computer, cause the method according to any one of claims 1-12 to be performed.
A vehicle, characterized in that the vehicle includes a target detection device, a millimeter-wave radar, and a camera;

The target detection device is configured to implement the method according to any one of claims 1-12 by controlling the millimeter-wave radar and the camera.