WO2023108931A1

WO2023108931A1 - Vehicle model determining method based on video-radar fusion perception

Info

Publication number: WO2023108931A1
Application number: PCT/CN2022/081188
Authority: WO
Inventors: 高超; 何煜埕; 张申浩; 谢争明
Original assignee: 江苏航天大为科技股份有限公司
Priority date: 2021-12-14
Filing date: 2022-03-16
Publication date: 2023-06-22
Also published as: CN114463660A

Abstract

The present invention relates to the technical field of target recognition. Disclosed is a vehicle model determining method based on video-radar fusion perception. The method comprises: framing a trapezoidal detection area along a lane in a video picture, and mapping, to a longitudinal distance and a transverse distance of a radar, the coordinates in the detection area in the video picture; extracting a length value and a width value of a target detected by the radar, extracting RGB value vectors of the picture in a vehicle detection frame, and forming a vector set representing the target by the length value and the width value of the target, and the RGB vectors; forming, into a training set, vector set parameter samples of a plurality of videos and the radar, and training same by an SVM trainer; and inputting the data acquired in real time into an SVM trainer objective function to identify the vehicle model. According to the present invention, the advantages of the radar and the video are combined, so that a radar and video all-in-one machine can stably and accurately detect the vehicle model under all-weather and all real-time conditions.

Description

A method for judging car models based on video radar fusion perception

technical field

The invention belongs to the technical field of target recognition, in particular to a method for judging vehicle models based on video radar fusion perception.

Background technique

In the face of increasingly complex road traffic conditions, with the development of science and technology, the relevant departments of road traffic management have increasingly applied radar and video sensing technology in intelligent transportation. Radar sensors transmit high-frequency electromagnetic waves and receive echoes The principle to measure the distance, speed and angle of surrounding objects. The video sensor detects the type and angle of surrounding objects by monitoring the video image in the lens. However, both radar sensors and video sensors have limitations in practical applications. For example, the limitations of radar technology are: first, the detailed resolution of the environment and obstacles, especially in terms of angular resolution, is not high; second, the target type cannot be identified. The limitations of video technology are: first, it is greatly affected by light and environment such as fog, rain, snow, etc. Second, it is impossible to accurately obtain the distance and speed information of the target; how to effectively integrate video and radar data is very important. necessary.

Contents of the invention

The present invention extracts the characteristic parameters of the radar recognition vehicle and the characteristic parameter of the video recognition vehicle through the combination of the radar and the video, and at the same time combines the processing of the abnormal situation of the video recognition caused by the harsh environment, selects different detection schemes and uses the support vector machine SVM to calculate , distinguish the vehicle type through multiple SVM binary recognition.

The method for judging a car model based on video radar fusion perception disclosed by the present invention includes the following steps: frame a trapezoidal detection area along the lane in the video screen, and map the coordinates in the detection area in the video screen to the longitudinal distance and lateral distance of the radar;

Extract the length value and the width of the target detected by the radar, extract the RGB value vector of the picture in the vehicle detection frame, and form the vector group representing the target with the target length value, width value and RGB vector;

The vector group parameter samples of multiple videos and radars are used to form a training set, which is trained by the SVM trainer;

Input the real-time collected data into the objective function of the SVM trainer to identify the vehicle type.

Further, the radar is installed at the vertical line of the detected lane.

Further, the frame trapezoidal detection area along the lane in the video picture includes:

Frame the trapezoidal detection area along the lane in the video screen, and record the coordinates of the four vertices in the screen (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ), (x ₄ , y ₄ ), wherein y ₁ =y ₂ , y ₃ =y ₄ , calculate the height h _v =y ₄ -y ₂ of the trapezoidal detection area in the video picture;

Calibrate the actual detection area width D _r and the number of lanes M;

Divide the height h _v of the detection area in the video into n equal parts from top to bottom, and calibrate the actual distance dh _i corresponding to each equal line of h _v (i=1,2,...,n,n+1);

Calculate the actual distance H _k corresponding to the vertical coordinate y _k (y ₂ ≤ _{y k} ≤ y ₄ ) of any point in the detection area in the video picture;

The function to calculate the left and right boundary lines L ₁ and L ₂ of the lane in the video frame:

Calculate the actual distance D _k corresponding to the coordinates (x _k , y _k ) of any point in the detection area in the video screen from the mid-perpendicular line of the lane, and map the coordinates (x _k , y _k ) in the detection area in the video screen Longitudinal distance H _k and lateral distance D _k to the radar.

Further, the calculation formula and steps of the H _k are as follows:

Further, the calculation formula and steps of the actual distance D _k are as follows:

Further, the extraction of the length and width of the target detected by the radar, and the extraction of the RGB value vector of the picture in the vehicle detection frame include:

Use pre-trained vehicle samples on the video side to detect vehicles in the video detection area through a feature classifier, extract the detected vehicle coordinates (x _k , y _k ) and the width WIDTH _k of the detected vehicle detection frame and high HEIGHT _k ;

Calculate the coordinates (D _k , H _k ) in the radar coordinate system corresponding to the detected vehicle coordinates in the video;

Find the coordinates of the target with the smallest relative (D _k , H _k ) deviation detected by the radar, and extract the length value C _l and width C _w of the target detected by the radar;

Convert the picture in the detection frame of the detected vehicle into a picture with a width of W and a height of H, extract the RGB value of the picture, and record it as a vector ω _p with a length of W*H*3.

Further, said training by SVM trainer includes:

Construct the SVM objective function:

Where y _i ∈ {-1,1}, δ is a positive integer, _xi is a vector group composed of parameters ω _p , C _L and C _w , when the video feasibility is judged to be low, the vector group is [C _L ,C _W ], otherwise [ω _p , _CL ,C _W ];

The training set is composed of n video and radar vector group parameter samples, and trained by the SVM trainer to obtain the minimum ||w||, that is, the minimum w ^T and b.

Further, the method for judging the low feasibility of the video is as follows:

Convert each frame of picture into a grayscale value, count the average value imgL of the picture brightness, set the upper and lower limit values imgL_max and imgL_min of the picture brightness, if the upper limit value is exceeded, the picture will be too bright, if it is lower than the lower limit value, the picture will be too dark, and the cumulative value will be lower than If the lower limit of brightness or the number of frames higher than the upper limit of brightness exceeds 3000 frames, it is an abnormal situation, and the reliability of video detection is low at this time.

Further, the method for judging that the video feasibility is low also includes:

Count the number of vehicles C_num_v in the detected video detection area and the number of vehicles C_num_r detected by the radar in the actual detection area. If |C_num_v-C_num_r| is greater than C_num_r*0.2 and the cumulative time is greater than 120 seconds, the video detection is abnormal , the feasibility of video detection is low.

Compared with prior art, the beneficial effect of the present invention is:

The present invention combines radar and video, adopts video and radar multi-parameter fusion mode for high-accuracy vehicle detection when the video scene is suitable, and uses radar parameters for detection when the video detection error is large in bad weather, thereby improving detection stability sexual method;

Combining the advantages of radar and video, the All-in-one camera can perform stable and accurate vehicle detection in all weather conditions.

Description of drawings

Fig. 1 is a schematic diagram of a trapezoidal detection area framed along the lane in the video picture of the present invention;

Fig. 2 is a flow chart of judging video credibility of the present invention;

Fig. 3 is a flow chart of the method for judging the vehicle type of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way, and any transformation or replacement done based on the teaching of the present invention belongs to the protection scope of the present invention.

Step 1: Unify the position of radar and video detection targets.

1) As shown in Figure 1, frame the trapezoidal detection area along the lane in the video screen, and record the coordinates (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ) of the four vertices in the screen , (x ₄ ,y ₄ ). Where y ₁ =y ₂ , y ₃ =y ₄ , calculate the height h _v =y ₄ -y ₂ of the trapezoid detection area in the video picture (the unit is the pixel in the video).

2) Calibrate the actual detection area width D _r and the number M of lanes.

3) Divide the height h _v of the detection area in the video into n equal parts from top to bottom, and calibrate the actual distance dh _i corresponding to each equal line of h _v (i=1,2,...,n,n+1 ).

4) Calculate the actual distance H _k corresponding to the ordinate y _k (y ₂ ≤y _k ≤y ₄ ) of any point in the detection area in the video frame. The calculation formula and steps are as follows:

(i takes a positive integer)

5) Calculate the function of the left and right boundary lines L ₁ and L ₂ of the lane in the video picture

6) The radar is installed at the mid-perpendicular line of the detected lane, and the actual distance D _k (left of the mid-perpendicular line) corresponding to the coordinates (x _k , y _k ) of any point in the detection area in the video picture from the mid-perpendicular line of the lane is calculated. side is negative, right is positive).

Thus, the coordinates (x _k , y _k ) in the detection area in the video frame can be mapped to the longitudinal distance H _k and the lateral distance D _k of the radar.

Step 2: Radar parameter extraction and video parameter extraction

1) Use pre-trained vehicle samples on the video side, use the haar+cascades feature classifier to detect vehicles in the video detection area, extract the detected vehicle coordinates (x _k , y _k ) and the detected vehicle detection The box's width WIDTH _k and height HEIGHT _k .

2) Calculate the coordinates (D _k , H _k ) in the radar coordinate system corresponding to the coordinates of the detected vehicle in the video through the formula in step 1. Find the coordinates of the target with the smallest relative (D _k , H _k ) deviation detected by the radar, and extract the length value C _l and width C _w of the target detected by the radar.

3) Set a unified width W and height H, convert the picture in the detection frame of the detected vehicle in step 2 1) into a picture with a width of W and a height of H, extract the RGB value of the picture, and record it as a length of W*H *3 vector ω _p .

4) Send ω _p , _CL and C _w to step 4 to identify the vehicle type.

Step 3: Troubleshoot video processing exceptions

Because video detection is greatly affected by changes in weather or light, when extreme weather such as heavy fog, heavy rain, strong light or external factors interfere, the accuracy of video vehicle detection will be reduced, which will affect the accuracy of Levision combined with judging the vehicle type Spend. Therefore, we need to monitor the abnormality of the video in real time and conduct investigations to reduce the credibility of the video parameters when the video is abnormal. When the video reliability is low, only radar data will be used for vehicle detection in step 4. As shown in Figure 2, the steps are as follows:

1) Convert each frame of picture into a grayscale value, count the average value imgL of the picture brightness, set the upper and lower limits of the picture brightness imgL_max and imgL_min, if the picture exceeds the upper limit, the picture will be too bright, if it is lower than the lower limit, the picture will be too dark, and the accumulated If the number of frames lower than the lower limit of brightness or higher than the upper limit of brightness exceeds 3000 frames, it is an abnormal situation, and the reliability of video detection is low at this time.

2) Count the number of vehicles C_num_v in the video detection area detected in step 2 1) and the number of vehicles C_num_r detected by the radar in the actual detection area. Calculate |C_num_v-C_num_r|, if the value is greater than C_num_r*0.2 and the cumulative time is greater than 120 seconds, the video detection is abnormal and the feasibility of video detection is low.

Step 4: Use SVM to identify vehicle models based on radar and video parameters

1) Construct the SVM objective function:

Where y _i ∈ {-1,1}, δ is a positive integer, and _xi is a vector composed of parameters extracted in step 2. When the video feasibility is judged to be low in step 3, the vector is [C _L , C _W ] , otherwise [ω _p , _CL ,C _W ].

Prepare n video and radar parameter samples in advance, and train them through the SVM trainer to get the smallest ||w||, which is the smallest w ^T and b.

2) Model identification

Use [ω _p , _CL ,C _W ] collected in step 2 and input it into the objective function. If the result is greater than or equal to δ, the vehicle is a large vehicle, otherwise it is a small vehicle. If it is necessary to distinguish multiple types of vehicles such as large, medium, and small, construct multiple objective functions and perform multiple binary selections to select vehicle types. Exemplarily, if it is necessary to identify large vehicles, medium vehicles and small vehicles, then construct two objective functions according to formula 4 (the first objective function and the second objective function, the δ values in the first objective function and the second objective function are different in size ), the training set of the first objective function is composed of small car sample parameters and middle car sample parameters, which are input into the first objective function for training. After the training, the collected real-time vehicle image parameters are input into the first objective function, If the result is greater than or equal to δ, the vehicle is a medium vehicle, otherwise it is a small vehicle. The training set of the second objective function is composed of the sample parameters of medium vehicles and large vehicles, which are input into the second objective function for training. After the training, the collected real-time vehicle image parameters are input into the second objective function, if If the result is greater than or equal to δ, the vehicle is a large vehicle, otherwise it is a medium vehicle.

Compared with prior art, the beneficial effect of the present invention is:

As used herein, the word "preferred" means serving as an example, instance or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, use of the word "preferably" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from context, "X employs A or B" is meant to naturally include either of the permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing instances.

Moreover, although the disclosure has been shown and described with respect to one or an implementation, equivalent alterations and modifications will occur to others skilled in the art upon the reading and understanding of this specification and the annexed drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. With particular regard to various functions performed by the above-mentioned components (eg, elements, etc.), terms used to describe such components are intended to correspond to any component that performs the specified function of the component (eg, it is functionally equivalent) Even if there are no structural equivalents to the disclosed structures that perform the functions in the exemplary implementations of the present disclosure shown herein (unless otherwise indicated). Furthermore, although a particular feature of the present disclosure has been disclosed with respect to only one of several implementations, such feature may be combined with one or other features of other implementations as may be desirable and advantageous for a given or particular application. combination. Also, to the extent the terms "comprises", "has", "comprising" or variations thereof are used in the detailed description or the claims, such terms are intended to be encompassed in a manner similar to the term "comprising".

Each functional unit in the embodiment of the present invention may be integrated into one processing module, or each unit may physically exist separately, or multiple or more of the above units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are implemented in the form of software function modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, and the like. Each of the above devices or systems may execute the storage method in the corresponding method embodiment.

In summary, the above-mentioned embodiment is an embodiment of the present invention, but the embodiment of the present invention is not limited by the embodiment, any other changes that deviate from the spirit and principle of the present invention, Modifications, substitutions, combinations, and simplifications should all be equivalent replacement methods, and are all included within the protection scope of the present invention.

Claims

A method for judging vehicle models based on video radar fusion perception, characterized in that it comprises the following steps:

Frame the trapezoidal detection area along the lane in the video screen, and map the coordinates in the detection area in the video screen to the longitudinal distance and lateral distance of the radar;

Extract the length value and the width of the target detected by the radar, extract the RGB value vector of the picture in the vehicle detection frame, and form the vector group representing the target with the target length value, width value and RGB vector;

The vector group parameter samples of multiple videos and radars are used to form a training set, which is trained by the SVM trainer;

Input the real-time collected data into the objective function of the SVM trainer to identify the vehicle type.
The method for judging a car model based on video radar fusion perception according to claim 1, wherein the radar is installed at the center perpendicular of the detected lane.
The method for judging a car model based on video radar fusion perception according to claim 1, wherein the framed trapezoidal detection area along the lane in the video picture comprises:

Frame the trapezoidal detection area along the lane in the video screen, and record the coordinates of the four vertices in the screen (x 1 , y 1 ), (x 2 , y 2 ), (x 3 , y 3 ), (x 4 , y 4 ), wherein y 1 =y 2 , y 3 =y 4 , calculate the height h v =y 4 -y 2 of the trapezoidal detection area in the video picture;

Calibrate the actual detection area width D r and the number of lanes M;

Divide the height h v of the detection area in the video into n equal parts from top to bottom, and calibrate the actual distance dh i corresponding to each equal line of h v (i=1,2,...,n,n+1);

Calculate the actual distance H k corresponding to the vertical coordinate y k (y 2 ≤ y k ≤ y 4 ) of any point in the detection area in the video picture;

The function to calculate the left and right boundary lines L 1 and L 2 of the lane in the video frame:

Calculate the actual distance D k corresponding to the coordinates (x k , y k ) of any point in the detection area in the video screen from the mid-perpendicular line of the lane, and map the coordinates (x k , y k ) in the detection area in the video screen Longitudinal distance H k and lateral distance D k to the radar.
The method for judging a car model based on video radar fusion perception according to claim 3, wherein the calculation formula and steps of H are as follows:
The method for judging a car model based on video radar fusion perception according to claim 3, wherein the calculation formula and steps of the actual distance D are as follows:
The method for judging a car model based on video radar fusion perception according to claim 1, wherein the length value and width of the target detected by the extraction radar, and the extraction of the RGB value vector of the picture in the vehicle detection frame include:

Use pre-trained vehicle samples on the video side to detect vehicles in the video detection area through a feature classifier, extract the detected vehicle coordinates (x k , y k ) and the width WIDTH k of the detected vehicle detection frame and high HEIGHT k ;

Calculate the coordinates (D k , H k ) in the radar coordinate system corresponding to the detected vehicle coordinates in the video;

Find the coordinates of the target with the smallest relative (D k , H k ) deviation detected by the radar, and extract the length value C l and width C w of the target detected by the radar;

Convert the picture in the detection frame of the detected vehicle into a picture with a width of W and a height of H, extract the RGB value of the picture, and record it as a vector ω p with a length of W*H*3.
The method for judging a car model based on video radar fusion perception according to claim 1, wherein said training by an SVM trainer includes:

Construct the SVM objective function:

Where y i ∈ {-1,1}, δ is a positive integer, xi is a vector group composed of parameters ω p , C L and C w , when the video feasibility is judged to be low, the vector group is [C L ,C W ], otherwise [ω p , CL ,C W ];

The training set is composed of n video and radar vector group parameter samples, and trained by the SVM trainer to obtain the minimum ||w||, that is, the minimum w T and b.
The method for judging a car model based on video radar fusion perception according to claim 1, wherein the method for judging that the video feasibility is low is as follows:

Convert each frame of picture into a grayscale value, count the average value imgL of the picture brightness, set the upper and lower limit values imgL_max and imgL_min of the picture brightness, if the upper limit value is exceeded, the picture will be too bright, if it is lower than the lower limit value, the picture will be too dark, and the cumulative value will be lower than If the lower limit of brightness or the number of frames higher than the upper limit of brightness exceeds 3000 frames, it is an abnormal situation, and the reliability of video detection is low at this time.
The method for judging a car model based on video radar fusion perception according to claim 1, wherein the method for judging that the video feasibility is low also includes:

Count the number of vehicles C_num_v in the detected video detection area and the number of vehicles C_num_r detected by the radar in the actual detection area. If |C_num_v-C_num_r| is greater than C_num_r*0.2 and the cumulative time is greater than 120 seconds, the video detection is abnormal , the feasibility of video detection is low.
The method for judging a car model based on video radar fusion perception according to claim 7, wherein the input of the real-time collected data into the SVM trainer objective function, and identifying the car model includes: grouping the collected vehicle vectors Parameters, input into the objective function, if the result is greater than or equal to δ, the vehicle is a large vehicle, otherwise it is a small vehicle.