WO2018041898A1

WO2018041898A1 - Vision system and method for a motor vehicle

Info

Publication number: WO2018041898A1
Application number: PCT/EP2017/071769
Authority: WO
Inventors: Erik Narby; Martin Nilsson
Original assignee: Autoliv Development AB
Current assignee: Autoliv Development AB
Priority date: 2016-08-31
Filing date: 2017-08-30
Publication date: 2018-03-08
Anticipated expiration: 2019-02-28
Also published as: US20210279478A1; JP6820403B2; EP3291133A1; JP2019525346A; EP3291133B1; US11138445B2

Abstract

A vision system (10) for a motor vehicle comprises an imaging apparatus (11) adapted to capture images from a surrounding of the motor vehicle and a data processing device (14) adapted to perform image processing on images captured by said imaging apparatus (11) and to control at least one driver assistance device (18) depending on a result of said image processing. Said image processing comprises object detection through edge detection and said object detection comprises decomposing (21, 22) images captured by said imaging apparatus into a plurality of decomposed images (23, 24) having different color characteristics. Separate edge detection (35, 36) is performed on the decomposed images- The edges (37, 38) from said separate edge detections are merged (39) into a common list of edges. An edge distinguishability measure is calculated for each of said edges (37, 38), and the merging of said edges (37, 38) is based on said edge distinguishability measure.

Description

Vision system and method for a motor vehicle

The invention relates to a vision system for a motor vehicle, comprising an imaging apparatus adapted to capture images from a surrounding of the motor vehicle and a processing device adapted to perform image processing on images captured by said imaging apparatus and to control at least one driver assistance device depending on a result of said image processing, wherein said image processing comprises object detection through edge detection, said obj ect detection comprising decomposing images captured by said imaging apparatus into a plurality of decomposed images having different color characteristics, performing separate edge detection on the decomposed images, and merging the edges from said separate edge de ec ions into a common list of edges. The invention also relates to a corresponding vision method.

Yellow lane markings, for example, are difficult to detect in greyscaie images if the lines are old or faded for other reasons, and/or on a road having a light color surface such as concrete or light-colored asphalt. Also weather conditions may cause yellow road markings to be difficult to detect. For example, such lines may be easy to detect under clouded conditions, but difficul t to detect under sunny conditions . Other examples of conditions which are often difficult are sunrise and sunset.

US 8,750,567 B2 discloses a vision system for a motor vehicle adapted to detect yellow road markings according to the preamble of claim 1. The objective underlying the present invention is to provide a vision system and method enabling a more reliable detection of colored structures, like road markings, outside the motor vehicle even under difficult conditions.

The invention solves this objective with the features of the independent claims. According to the invention, an edge dis- tinguishability measure is calculated for each of the detected edges, and the merging of the detected edges is based on the distinguishability measure.

The invention provides a way of detecting colored road markings, and other colored objects like other vehicles, traffic signs etc., more reliably by performing edge detection in multiple images and combining the result in a new way. The decomposed images are preferably created by linearly combining color channels in a way which makes road markings of certain colors clearly visible. Edge detection is then performed in all decomposed images and the edges are combined, preferably by removing duplicate edges, i.e. edges close to each other in world coordinates, by preferably choosing the edge with the highest edge distinguishability measure, in particular the highest edge gradient, preferably normalized by a specific edge threshold. The edge threshold may preferably be calculated using a histogram of the calculated gradient image.

The invention performs merging on an edge level using edge distinguishability measures like relative gradient amplitudes, while US 8,750,567 B2 merges after grouping of edges, and does not use edge distinguishability measures like the invention. According to the invention, the edges are individually detected and processed. From ail image rows of an image captured by the imaging apparatus, preferably only a selected subset of rows is searched for edges. Preferably there is a number of rows between each searched row. Further preferably, the distance between the outer searched rows corresponds to a relatively small height, in particular less than 50 cm in a distance of 80 m in front of the car, like 30 cm in a distance of 80 m in front of the car. The searching for lane markings, for example, is prefera bly performed in less than the bottom half of the captured im age, and more preferably within a region of up to 80 m in front of the car. The above features contribute to an excellent computation efficiency of the inventive edge detection.

In the following the invention shall be illustrated on the ba sis of preferred embodiments with reference to the accompanying drawings, wherein:

Fig. 1 shows a schematic diagram of a vision system under the present invention; and

Fig. 2 shows a schematic flow diagram for illustrating the present invention.

The vision system 10 is mounted in a motor vehicle and comprises an imaging apparatus 11 for capturing images of a region surrounding the motor vehicle, for example a region in front of the motor vehicle. Preferably the imaging apparatus 11 comprises one or more optical imaging devices 12, in particular cameras, preferably operating in the visible and/or infrared wavelength range, where infrared covers near IR with wavelengths below 5 microns and/or far IR with wavelengths be yond 5 microns. In some embodiments the imaging apparatus 11 comprises a plurality imaging devices 12 in particular formin a stereo imaging apparatus 11. In other embodiments only one imaging device 12 forming a mono imaging apparatus 11 can be used. The imaging apparatus 11 is coupled to a data processi ng device 14 adapted to process the image data received from the imaging apparatus 11. The data processing device 14 is preferably a digital device which is programmed or programmable and preferably comprises a microprocessor, microcontroller a digi- tal signal processor (DSP) , and/or a microprocessor part in a System-On-Chip (SoC) device, and preferably has access to, or comprises, a data memory 15. The data processing device 14 may comprise a dedicated hardware device, like a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC) , or an FPGA and/or ASIC part in a Systera-On-

Chip (SoC) device, for performing certain, functions, for example controlling the capture of images by the imaging apparatus 11, receiving the electrical signal containing the image information from the imaging apparatus 11, rectifying or warping pairs of left/right images into alignment and/or creating disparity or depth images. The data processing device 14, or part of its functions, can be realized by a System-On-Chip (SoC) device comprising, for example, FPGA, DSP, ARM and/or microprocessor functionality. The data processing device 14 and the memory device 15 are preferably realised in an on-board electronic control unit (ECU) and may be connected to the imaging apparatus 11 via a separate cable or a vehicle data bus . I another embodiment the ECU and one o more of the imaging devices 12 can be integrated into a single unit, where a one box solution including the ECU and all imaging devices 12 can be preferred. All steps from imaging, image processing to possible activation or control of driver assistance device 18 are performed automatically and continuously during driving in real time.

Image and data processing carried out in the data processing device 14 advantageously comprises identifying and preferably also classifying possible objects (object candidates) in front of the motor vehicle, such as pedestrians, other vehicles, bicyclists and/or large animals, tracking over time the position of objects or object candidates identified in the captured images, and activating or controlling at least one driver assistance device 18 depending on an estimation performed with respect to a tracked object, for example on an estimated collision probability. The driver assistance device 18 may in particular comprise a display device to display information relating to a detected object. However, the invention is not limited to a display device. The driver assistance device 18 may in addition or alternatively comprise a warning device adapted to provide a collision warning to the driver by suitable optical, acoustical and/or haptic warning signals; one or more restraint systems such as occupant airbags or safety belt tensioners, pedestrian airbags, hood lifters and the like;

and/or dynamic vehicle control systems such as braking or steering control devices.

In the following, the invention is described using the flow diagram shown in Figure 2. Herein, color images 20 taken by the imaging apparatus 11 are processed in the data processing device 14, i.e. all steps 21 to 39 downstream of the color images 20 in Figure 2 are performed in the data processing device 14. The color images 20 are RGGB images in the present embodiment, but could as well be color images of other color schemes . In steps 21, 22 the raw RGGB image 20 is decomposed or de- mosaiced into an intensity image 23 (step 21: demosa icing intensity) and into a weighted color image 24 (step 22: demosa- icing weighted color) . Herein,, the demos ici g intensity step 21 yields the intensity image 23 which denotes the grayscale intensity of each pixel independent of its color.

The demosaicing weighted color step 22 is designed to show as clearly as possible the color object to be detected. In the following, the color object to be detected shall be a yellow road marking, where it is clear that the invention can be applied to any other color than yellow, and to any other object that a road marking.

The weighted color image 24 is preferably of the same color as the color object in the en ironme t of the motor vehicle to be detected. In the present example, therefore, the weighted color image 24 is advantageously a yellow image. The yellow image can preferably be calculated from an RGGB image by a linear combination between (R+G+2x (Bmax-B) ) /4 and (R+ (Bmax-B) ) 12, where R is red intensity, G is green intensity, B is blue intensity, and Bmax is maximum blue intensity in the image. This is a very calculation efficient way of calculating a yellow image providing a high yellow contrast level. For other colors than yellow, similar simple formulas can be set up to calculate the weighted color image 24 from linear combination of terms involving R, G and/or B values.

Following the decomposing of the color image 20 in steps 21 and 22, the intensity image 23 and the weighted color image 24 are processed separately, yielding two parallel processing branches 25, 26, namely the color-independent intensity processing branch 25 and the color processing branch 26. In both branches a gradient calculation 27, 28 is performed yielding a corresponding gradient image 29, 30, respectively. Methods for calculating a gradient image from an input image are known to the skilled person.

For the gradient image 29, next an intensity edge threshold 33 is calculated in a threshold calculation section 31. In the th eshold calculation section 31 preferably a histogram of all gradient values in the gradient image 29 is calculated first. The edge threshold 33 for the intensity image 23 may then be calculated as a factor times the median of all gradient values. Instead of the median, any other centre value regarding the statistical distribution of the gradient values could be used, for example the mean value, or the value providing the highest number in the histogram. Furthermore, the statistical centre value (median, mean, etc.) may be calculated in other ways than from a gradient values histogram, for example directly from the gradient image; in that case, the calculation of a histogram may not be necessary .

Similarly, for the gradient image 30 in the color image processing branch 26, an intensity edge threshold 34 is calculated in a threshold calculation section 32. In the threshold calculation section 32 preferably a histogram of all gradient values in the gradient image 30 is calculated first. The edge threshold 34 for the color image 24 may then be calculated as a factor times the median of all gradient values, or any other centre value regarding the statistical distribution of the gradient values. Also here, the statistical centre value may be ca lculated in other ways than from a gradient values histogram. Next, in the edge detection sections 35 , 36 , the gradient images 2 9 , 30 are thresholded with or against the corresponding calculated edge threshold 33 , 34 , respectively. Thresholding here means that every pixel of the gradient image 2 9 , 30 fall- ing below the corresponding edge threshold 33 , 34 is set to zero, and every pixel of the gradient image 29 , 30 reaching (in other embodiments, reaching or exceeding) the corresponding edge threshold 33 , 34 is set to some non-zero value. As result, all non-zero positions of the thresholded intensity gradient image 2 9 are stored as intensity edges 37 , and all non-zero positions of the thresholded color gradient image 30 are stored as color edges 38 .

In the next step, a normalized gradient value is calculated for each stored edge 37 , 38 by dividing the gradient value of the edge under consideration by the corresponding threshold value 33 , 34 . This normalized gradient value is a measure of how clearly the edge 37 , 38 can be seen in the image. The normalized gradient value calculation can be performed in the merging section 39 to be described in the following.

After the normalized gradient value calculation, the lists of edges 37 , 38 with normalized gradient values can be merged into a common list of edges in the merging section 39 .

Finally, duplicate edges are removed from the common list of edges 37 , 38 . Duplicate edges are multiple edges which are caused by the same physical structure outside the vehicle, for example, lane marking. In the present context two edges are considered duplicate if their positions are within a specified limit in world coordinates. For all duplicate edges, the edge having the lower or lowest normalized gradient is removed. Generally, for all duplicate edges, only the edge having the highest normalized gradient is kept. The output of the merging section 39 is a list 40 of detected edges free of duplicates. It is clear from the above that the edges contained in the final edge list 40 is a true subset of all edges 37, 38 due to the merging and duplicate removing process .

Further object detection processing, like road marking detection in the present example, can be performed on the list 40 of detected edges.

The invention can be readily generalized to more than two processing branches 25 , 26 by decomposing each color images 20 into more than two decomposed images 2.1, 22 , in particular one decomposed intensity image 21 and two or more decomposed color images 22 involving different color characteristics .

Claims

Claims :

A vision system ( 10 ) for a motor vehicle, comprising an imaging apparatus (11) adapted to capture images from a surrounding of the motor vehicle and a data processing device (14) adapted to perform image processing on images captured by said imaging apparatus (11) and to control at least one driver assistance device (18) depending on a result of said image processing, wherein said image processing comprises object detection through edge detection, said object detection comprising decomposing (21, 22) images captured by said imaging apparatus into a plurality of decomposed images (23, 24 ) having different color characteristics , performing separate edge detection (35, 36) on the decomposed images, and merging (39) the edges (37, 38) from said separate edge detections into a common list (40) of edges, characterized in that an edge dist inguishabi L ity measure is calculated for each of said edges (37, 38), and the merging of said edges (37, 38) is based on said edge distinguishability measure.

The vision system as claimed in claim 1, characterized in that the merging of said edges (37, 38) is based on selecting the edge among edges from di fferent decomposed images (23, 24 ) providing the highest distinguishability measure .

The vision system as claimed in claim 1 or 2, characterized in that: a gradient image (27 , 28 ) is calculated on each of said decomposed images (23, 24) .

The vision system as claimed in claim 3, characterized in that said edge distinguishability measure is a gradient amplitude of each detected edge (37, 38) .

5. The vision system as claimed in claim 3 or 4, characterized in that a normalized gradient is calculated for each detected edge (37, 38).

6. The vision system as claimed in claim 5, characterized in that said normalized gradient is calculated by dividing the gradient value of the detected edge (37, 38) by a corresponding edge threshold (33, 34) value.

7. The vision system as claimed in any one of the preceding claims, characterized in that an edge threshold (33, 34) is calculated for each of said decomposed images (23, 24) .

8. The vision system as claimed in claim 7, characterized in that the edge threshold (37, 38) is calculated from a statistical center value of gradient values, preferably a median of gradient values.

9. The vision system as claimed in claim 7 or 8, characterized in that said edge detection is performed by thresholding each gradient image (29, 30) against said edge threshold (37, 38) .

10. The vision system as claimed in any one of the preceding claims, characterized in that duplicate detected edges are removed from all edges (37, 38) detected in each of said decomposed images (23, 24) .

11. The vision system as claimed in any one of the preceding claims, characterized by considering two detected edges (37, 38) as being duplicates if their positions are within a specified limit in world coordinates.

12. The vision system as claimed in any one of the preceding claims, characterized in that said decomposed images (23, 24) comprise an illumination intensity image (23) and a weighted color image (24) .

13. The vision system as claimed in any one of the preceding claims, characterized in that said weighted color image (24) has the same color as the object in the environment of the motor vehicle to be detected.

14. The vision system as claimed in any one of the preceding claims, characterized in that said weighted color image (24) is a yellow image calculated from an RGB image by a linear combination between (R+G+2x (Bmax-B) ) /4 and

(R+ (Bmax-B) ) /2.

15. A vision method for a motor vehicle, comprising capturing images from a surrounding of the motor vehicle, per orming image processing on captured images and controlling at least one driver assistance device (18) depending on a result of said image processing, wherein said image processing comprises obj ect detection through edge detection, said object detection comprising decomposing (21, 22) captured images into a plurality of decomposed images (23, 24) having different color characteristics, performing separate edge detection (35, 36) on the decomposed images , and merging (39) the edges ( 37 , 38 ) from said separate edge detections into a common list of edges, characterized in that an edge distinguishability measure is calculated for each of said edges (37, 38), and the merging of said edges (37, 38) is based on said edge d tinguishability measure.