CN111762155B

CN111762155B - Vehicle distance measuring system and method

Info

Publication number: CN111762155B
Application number: CN202010518686.9A
Authority: CN
Inventors: 张金华; 何君舰; 刘瑞祥; 曹红胜; 吴志桥
Original assignee: Anhui Singularity Intelligent New Energy Automobile Co ltd
Current assignee: Anhui Singularity Intelligent New Energy Automobile Co ltd
Priority date: 2020-06-09
Filing date: 2020-06-09
Publication date: 2022-06-28
Anticipated expiration: 2040-06-09
Also published as: CN111762155A

Abstract

The embodiment of the disclosure discloses a distance measuring system and method for a vehicle. Wherein the system includes: the system comprises an image acquisition device and an FPGA platform; the left view data and the right view data acquired by the image acquisition device are transmitted to the FPGA platform; the FPGA carries out binocular matching calculation on the left view data and the right view data to obtain a matching calculation result; performing distance calculation on the matching calculation result to obtain a distance calculation result; performing cost calculation on the distance calculation result to obtain a parallax space diagram; calculating the parallax space image by using a WTA algorithm to obtain a parallax image; performing median filtering on the disparity map to obtain a median filtering result; and carrying out consistency detection on the median filtering result, and outputting a corresponding parallax value if the median filtering result is consistent with the median filtering result. Therefore, real-time binocular ranging is realized, the requirements of accuracy and real-time performance are met, and the requirements of low power consumption and low cost are realized.

Description

Vehicle distance measuring system and method

Technical Field

The disclosure belongs to the technical field of automatic driving, and particularly relates to a vehicle distance measuring system and method.

Background

In the field of autonomous driving, collision warning is an important function of ADAS. The change of the distance needs to be concerned when the collision early warning is realized, the time about collision is estimated, and the distance measurement is carried out at the bottom. The accuracy and rapidity of obstacle ranging greatly influence the development process of automatic driving.

The binocular camera distance measuring method is used for measuring the distance of a front obstacle by performing parallax calculation on two images. In the distance measuring process, the requirements on real-time performance and power consumption are high; in the prior art, a common ARM embedded platform cannot meet corresponding real-time requirements, and a GPU platform has high power consumption and high price cost.

Disclosure of Invention

In view of this, the embodiments of the present disclosure provide a vehicle distance measuring system and method, which at least achieve the purposes of high accuracy, strong real-time performance, low power consumption and low cost.

In a first aspect, an embodiment of the present disclosure provides an automotive ranging system, including: the system comprises an image acquisition device and an FPGA platform, wherein the FPGA platform comprises an FPGA;

the left view data and the right view data acquired by the image acquisition device are transmitted to the FPGA;

the FPGA carries out binocular matching calculation on the left view data and the right view data to obtain a matching calculation result;

the FPGA carries out distance calculation on the matching calculation result to obtain a distance calculation result;

the FPGA carries out cost calculation on the distance calculation result to obtain a parallax space diagram;

the FPGA calculates the parallax space image by using a WTA algorithm to obtain a parallax image;

The FPGA carries out median filtering on the disparity map to obtain a median filtering result;

and the FPGA carries out consistency detection on the median filtering result, and if the median filtering result is consistent with the median filtering result, a corresponding parallax value is output.

Optionally, the FPGA performs consistency detection on the median filtering result, and further includes:

and if the disparity values are inconsistent, performing mean filtering on the corresponding disparity values, and outputting mean-filtered disparity values.

Optionally, the transmitting the left and right view data acquired by the image acquiring apparatus to the FPGA platform includes:

performing serial-parallel combination on the left view data and the right view data to obtain combined data;

and storing the merged data to a cache device.

Optionally, the FPGA performs binocular matching calculation on the left and right view data to obtain a matching calculation result, including:

enabling a parallax computation flag when the merged data stored in the cache device reaches a threshold;

reading the merged data in the cache device and temporarily storing the merged data in the two FIFOs respectively;

and when both FIFOs have data, performing binocular matching calculation.

Optionally, the binocular matching calculation includes:

simultaneously extracting data in the two FIFOs, and caching the extracted data line by utilizing N dual-port RAMs;

After the RAM buffers N rows of data, binocular matching calculation is started from N +1 rows of pixel input of the RAM.

Optionally, the binocular matching calculation further includes:

selecting the gray value of the corresponding pixel as a reference;

comparing the gray values of the rest pixels with the reference;

if the gray values of the rest pixels are smaller than the base criterion, the gray values are recorded as 1, otherwise, the gray values are 0, and the results are output in sequence.

Optionally, the distance calculation performed by the FPGA on the matching calculation result to obtain a distance calculation result includes:

the matching calculation result comprises left and right image frames;

taking one of the image frames as a reference, and setting a position pixel for exclusive OR operation after taking a corresponding position and a corresponding position from the other image frame;

and adding the result of the exclusive or operation of each bit of each disparity value in the matching calculation result.

Optionally, the FPGA performs cost calculation on the distance calculation result to obtain a disparity space map, including:

carrying out cost calculation in multiple directions through the FPGA to obtain cost values in multiple directions;

and summing the cost values in the multiple directions to obtain the parallax space diagram.

Optionally, the calculating, by the FPGA, the disparity space map by using a WTA algorithm to obtain the disparity map includes:

And selecting the parallax with the minimum matching cost of each pixel point in the parallax space image to obtain the parallax image.

Optionally, performing consistency detection on the median filtering result, including:

the parallax graphs are two, and whether the coordinate point of the pixel in the parallax graph serving as the reference and the parallax value corresponding to the coordinate point of the pixel in the other parallax graph are consistent or not is verified by taking one of the parallax graphs as the reference;

if the parallax exceeds a set threshold, setting the parallax of the corresponding pixel point as a set value; if the parallax does not exceed the set threshold, the original value is retained.

In a second aspect, an embodiment of the present disclosure provides a vehicle distance measuring method, including:

transmitting left and right view data acquired by an image acquisition device to an FPGA platform, wherein the FPGA platform comprises an FPGA;

performing distance calculation on the matching calculation result to obtain a distance calculation result;

performing cost calculation on the distance calculation result to obtain a parallax space diagram;

calculating the parallax space image by using a WTA algorithm to obtain a parallax image;

performing median filtering on the disparity map to obtain a median filtering result;

And carrying out consistency detection on the median filtering result, and outputting a corresponding parallax value if the median filtering result is consistent with the median filtering result.

Optionally, the step of performing consistency detection on the median filtering result further includes:

and storing the merged data to a cache device.

and when both FIFOs have data, performing binocular matching calculation.

Optionally, the binocular matching calculation includes:

Optionally, the binocular matching calculation further includes:

selecting the gray value of the corresponding pixel as a reference;

comparing the gray values of the other pixels with the reference;

if the gray value of the rest pixels is smaller than the base criterion, the gray value is recorded as 1, otherwise, the gray value is 0, and the results are sequentially arranged and output.

Optionally, the performing distance calculation on the matching calculation result to obtain a distance calculation result includes:

the matching calculation result comprises left and right image frames;

taking one of the image frames as a reference, and setting positioning pixels for exclusive OR operation after taking the corresponding position and the corresponding position of the other image frame;

and adding the exclusive or operation result of each bit of each parallax value in the matching calculation result.

Optionally, the performing cost calculation on the distance calculation result to obtain a disparity space map includes:

Optionally, the calculating the disparity space map by using a WTA algorithm to obtain a disparity map includes:

the parallax graphs are two, and whether the coordinate points of the pixels in the parallax graph serving as the reference and the parallaxes of the corresponding parallax values and the coordinate points of the corresponding pixels in the other parallax graph are consistent or not is verified by taking one of the parallax graphs as the reference;

if the parallax exceeds a set threshold, setting the parallax of the corresponding pixel point as a set value; and if the parallax does not exceed the set threshold value, keeping the original value.

The method is based on FPGA to perform parallax calculation, view data are read on an FPGA platform, binocular matching calculation, distance calculation and cost calculation are performed in the FPGA, calculation is performed by using a WTA algorithm, median filtering is performed, then consistency detection is performed on the filtered parallax value, and the parallax value is output according to the detection result, so that real-time binocular ranging is realized, the requirements of accuracy and real-time performance are met, and the requirements of low power consumption and low cost are also realized.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent by describing in greater detail exemplary embodiments thereof with reference to the attached drawings, in which like reference numerals generally represent like parts throughout.

FIG. 1 illustrates a functional block diagram of a vehicle ranging system according to one embodiment of the present disclosure;

FIG. 2 illustrates a schematic diagram of serial-to-parallel converting and merging data according to one embodiment of the present disclosure;

fig. 3 shows a flow diagram for FPGA to compute disparity according to one embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of an FPGA implementing Census computation according to one embodiment of the present disclosure;

FIG. 5 shows a diagram of median filtering according to one embodiment of the present disclosure;

fig. 6a to 6c are schematic views illustrating detection results output from a ranging system for a vehicle according to an embodiment of the present disclosure.

Detailed Description

Preferred embodiments of the present disclosure will be described in more detail below. While the following describes preferred embodiments of the present disclosure, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein.

Hamming (Hamming theorem);

WTA(Winner Take All)。

as shown in fig. 1, a ranging system for a vehicle includes: the system comprises an image acquisition device and an FPGA platform;

the image acquisition device is a video camera or a binocular camera and the like which are arranged on the vehicle and used for simulating human eyes to acquire binocular images.

The left and right view data acquired by the image acquisition device are transmitted to an FPGA platform, and the FPGA platform comprises an FPGA;

In a specific application scene, a binocular camera acquires a left image and a right image, and the left image and the right image are input into an FPGA for calculation through an MIPI interface. The FPGA platform utilizes the camera to transfer incoming images.

Optionally, the transmitting left and right view data acquired by the image acquiring apparatus to the FPGA includes:

and storing the merged data to a cache device.

In a specific application scenario, after a system is powered on, an MIPI CSI interface of an FPGA platform receives image data, and after data input by MIPI are subjected to serial-parallel conversion, the data are merged and stored in onboard DDR for caching.

The FPGA platform carries out binocular matching calculation on the left view data and the right view data to obtain a matching calculation result;

optionally, the performing binocular matching calculation on the left view data and the right view data by the FPGA platform to obtain a matching calculation result includes:

and when both FIFOs have data, performing binocular matching calculation.

In a specific application scenario, after binocular data in the DDR is cached for one frame, parallax calculation marks are enabled, left and right images are read and respectively temporarily stored in the FIFOs, and calculation is started when both the FIFOs have data.

Optionally, the binocular matching calculation includes:

Optionally, the binocular matching calculation further includes:

selecting the gray value of the corresponding pixel as a reference;

comparing the gray values of the rest pixels with the reference;

In a specific application scenario, a Census algorithm is adopted in binocular matching calculation, data in left and right FIFOs are extracted simultaneously, 4 (or 5) dual-port RAMs are utilized to perform line-by-line caching respectively, preparation is made for Census calculation of 5x5, and after 4 (or 5) lines of data are completely cached, the Census algorithm is calculated from the input of pixels in the next line; in Census algorithm 5x5, the gray value of the third row and the third column of pixels is used as a reference, the gray values of the other 24 pixels are compared with the reference, if the gray values are smaller than the reference, the gray values are marked as 1, otherwise, the gray values are 0, and the results are sequentially arranged into 24-bit data output.

the matching calculation result comprises left and right image frames;

In a specific application scenario, Hamming calculation is adopted for distance calculation, and a Census calculation result is input into a Hamming calculation module to perform Hamming theorem calculation of 64 parallaxes. Taking the left image frame as a reference, taking the corresponding position and 63 pixels behind the corresponding position in the right image to carry out exclusive OR operation, and adding each bit result of each parallax value to obtain a Hamming output result.

In a specific application scene, semi-global matching cost calculation is carried out, cost calculation in positive 4 directions is carried out through an FPGA, cost values in the positive direction of a line on a RAM cache are used for calculating the cost of the line, and pipeline operation is achieved. Obtaining a parallax space map S (p, d) after summing the 4 direction cost values; meanwhile, the parallax image space S _ R (p, d) calculated with reference to the right image pixel is calculated using the buffer data.

In a specific application scenario, based on a parallax space map S (p, d) and an S _ R (p, d), a wta (winner Take all) algorithm is respectively used for calculation, and in the parallax space map S (p, d), a parallax with the minimum matching cost of each pixel point p is selected, so that a left parallax map and a right parallax map are obtained.

in a specific application scenario, median filtering of 3x3 is performed on the obtained disparity map, fast median filtering is performed on 3 lines of cache by using RAM resources in an FPGA to calculate, and a disparity map median filtering result is output.

And the FPGA carries out consistency detection on the median filtering result, and if the median filtering result is consistent, a corresponding parallax value is output.

and if the disparity values are inconsistent with each other, performing mean filtering on the corresponding disparity values, and outputting mean-filtered disparity values.

That is, in the coincidence detection, if all the parallax values of the coincidence detection coincide, the parallax value result is output as the parallax output. If the disparity values are consistent and the disparity values are inconsistent, mean filtering is carried out on the inconsistent disparity values for filling, and the filled disparity values and the disparity values which are detected to be consistent are output as disparity.

And obtaining the distance between the vehicle and the object according to the output parallax value and the binocular ranging principle.

Optionally, the performing, by the FPGA, consistency detection on the median filtering result includes:

In a specific application scene, using a left disparity map as a reference, performing left-right consistency detection by using a left-right disparity map, verifying whether a point (x, y) in the left disparity map and the disparity value d thereof are consistent with the disparity of a corresponding point (x + d, y) in a right disparity map, if the difference exceeds 1, determining that the verification is failed, and setting the disparity of the point as 65; if the difference is not more than 1, the original value is retained. And (4) performing 3x3 mean filtering on the parallax value (65) which fails to be verified for filling, obtaining the final parallax, and outputting a detection result.

In a specific application scene, the binocular camera inputs data into the FPGA through the MIPI CSI interface, the FPGA caches the data into the DDR through the DDR controller, then left and right images are read from the DDR to perform parallax calculation, and finally the obtained parallax result is output.

As shown in fig. 2, the MIPI CSI interface of the FPGA platform receives the data flow, and the MIPI of the two links are decoded and merged by the serial-parallel merging unit to obtain the DDR pre-stored data.

As shown in fig. 3, which is a flowchart of the parallax calculation unit, after an image is input, the image passes through the Census calculation unit, the Hamming theorem calculation unit, the cost calculation unit, the WTA calculation unit, the median filtering calculation unit, the left-right consistency detection calculation unit, and the mean filtering calculation unit, and then the obtained parallax calculation result is output.

As above formula, it is Census calculation formula, where p refers to the middle pixel in 5x5 pixels, q refers to 24 pixels except the middle pixel in 5x5 pixels, t (p) is Census result of pixel p, ξ is the middle variable after pixel comparison, Np is the rest 24 pixels except the middle pixel in 5x5 pixels, i (p) is the middle pixel value, i (q) is the rest 24 pixel values. By comparison with the value of the middle position, the value smaller than the middle position is taken as 1, otherwise it is 0, and the Census value is formed.

As shown in fig. 5, in order to implement Census mode for FPGA, 4 lines (or 5 lines) of data are buffered, and calculation is performed by using a template with a size of 5 × 5, so as to obtain a 24-bit data value.

As the formula, Lr is the calculation result of cost aggregation in a certain direction, C is the current cost value, S is the sum of the costs in multiple directions, and p-r represents the calculation of a point p in the r direction; k. i is all values of the parallax d; r is the parallax propagation direction; p1 and P2 are both penalty items, and are respectively penalty items for the case that the parallax difference between adjacent pixels is 1 and the parallax difference exceeds 1.

And for the semi-global matching algorithm, calculating the cost of a certain parallax direction according to a formula. And respectively calculating the parallax costs of the positive 4 directions by utilizing a pipeline structure of the FPGA, namely four directions from top to bottom, from left to right, obliquely right to bottom and obliquely left to bottom. And finally, performing summation calculation according to the obtained 4 directional costs to finally obtain the total calculation cost. In the FPGA, a dual-port RAM is utilized to cache cost values of the data in the last line in four directions, and the calculation of the line is carried out, so that the algorithm flow of the pipeline structure is realized.

The parallax map is obtained by using a WTA method, and the specific calculation method is to select the parallax with the minimum matching cost of each pixel point p in the parallax space map S (p, d) to obtain the parallax map. Then, the median filtering is performed as shown in fig. 5, 3 lines of data are cached, calculation is performed according to a 3x3 template, noise is filtered, and a parallax result after the median filtering is obtained.

For the left-right consistency detection algorithm, the FPGA firstly caches the right disparity map for calculation, verifies whether the point (x, y) in the left disparity map and the disparity value d thereof are consistent with the disparity of the corresponding point (x + d, y) in the right disparity map, if the difference exceeds 1, the verification is considered to be failed, and the point disparity is set as 65; if the difference is not more than 1, the original value is retained.

And (3) performing 3x3 mean filtering on the parallax value which fails to be verified to fill up, performing same-median filtering in an FPGA (field programmable gate array), caching three lines of data, judging whether the middle value is an invalid parallax value, if so, performing mean filtering, otherwise, keeping the original value to obtain the final parallax, and outputting a detection result, wherein the steps are shown in fig. 6a to 6 c.

The binocular camera ranging method implemented by the vehicle ranging system adopts a full-pipeline structure, reasonably utilizes the parallelism processing architecture and the internal cache structure of the FPGA to process, can achieve a parallax processing effect of more than 50 frames when the input image of the camera is 1280x800, well meets the requirements of vehicle-mounted real-time performance, and meanwhile, the accuracy of parallax obtaining can also be well met.

The embodiment of the present disclosure further provides a vehicle distance measuring method, including:

and storing the merged data to a cache device.

Optionally, the performing, by the FPGA platform, binocular matching calculation on the left and right view data to obtain a matching calculation result includes:

enabling a parallax calculation flag when the merged data stored in the cache device reaches a threshold value;

and when both FIFOs have data, performing binocular matching calculation.

Optionally, the binocular matching calculation includes:

Optionally, the binocular matching calculation further includes:

selecting the gray value of the corresponding pixel as a reference;

comparing the gray values of the rest pixels with the reference;

the matching calculation result comprises left and right image frames;

Optionally, performing consistency detection on the median filtering result includes:

Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A vehicular distance measuring system, comprising: the system comprises an image acquisition device and an FPGA platform, wherein the FPGA platform comprises an FPGA;

the left and right view data acquired by the image acquisition device are transmitted to the FPGA;

the FPGA carries out consistency detection on the median filtering result, and if the median filtering result is consistent with the median filtering result, a corresponding parallax value is output;

the FPGA carries out consistency detection on the median filtering result, and the method further comprises the following steps:

if the disparity values are inconsistent, performing mean filtering on the corresponding disparity values, and outputting the mean-filtered disparity values;

the left and right view data acquired by the image acquisition device are transmitted to the FPGA, and the method comprises the following steps:

Storing the merged data to a cache device;

the FPGA carries out binocular matching calculation on the left view data and the right view data to obtain a matching calculation result, and the method comprises the following steps:

when both FIFOs have data, binocular matching calculation is carried out;

the binocular matching calculation includes:

2. The vehicular distance measuring system according to claim 1, wherein the binocular matching calculation further comprises:

selecting the gray value of the corresponding pixel as a reference;

comparing the gray values of the rest pixels with the reference;

3. The vehicular distance measuring system according to claim 1, wherein the FPGA performs distance calculation on the matching calculation result to obtain a distance calculation result, and the distance calculation result comprises:

The matching calculation result comprises left and right image frames;

4. The vehicular ranging system according to claim 1, wherein the FPGA performs cost calculation on the distance calculation result to obtain the parallax space map, and comprises:

and summing the cost values in multiple directions to obtain a parallax space diagram.

5. The vehicular distance measuring system according to claim 4, wherein the FPGA calculates the disparity space map by using a WTA algorithm to obtain a disparity map, and the method comprises the following steps:

6. The vehicular ranging system according to claim 5, wherein the FPGA performs consistency detection on the median filtering result, and the consistency detection comprises:

7. A distance measuring method for a vehicle is characterized by comprising the following steps:

the step of performing consistency detection on the median filtering result by the FPGA further comprises the following steps:

the left and right view data acquired by the image acquisition device are transmitted to the FPGA platform, and the method comprises the following steps:

storing the merged data to a cache device;

the FPGA carries out binocular matching calculation on the left and right view data to obtain a matching calculation result, and the method comprises the following steps:

when both FIFOs have data, binocular matching calculation is carried out;

the binocular matching calculation includes:

8. The vehicular distance measuring method according to claim 7, wherein the binocular matching calculation further comprises:

selecting the gray value of the corresponding pixel as a reference;

comparing the gray values of the rest pixels with the reference;

9. The vehicular distance measuring method according to claim 7, wherein the FPGA performs distance calculation on the matching calculation result to obtain a distance calculation result, and the distance calculation method comprises the following steps:

The matching calculation result comprises left and right image frames;

10. The vehicular distance measuring method according to claim 7, wherein the FPGA performs cost calculation on the distance calculation result to obtain a parallax space map, and the method comprises the following steps:

11. The vehicular distance measuring method according to claim 10, wherein the FPGA calculates the parallax space map by using a WTA algorithm to obtain the parallax map, and comprises:

12. The vehicular ranging method according to claim 11, wherein the FPGA performing consistency detection on the median filtering result comprises: