EP2219133A1

EP2219133A1 - A method and system of automatically detecting objects in front of a motor vehicle

Info

Publication number: EP2219133A1
Application number: EP09002172A
Authority: EP
Inventors: Ognjan Hedberg; Per Jonas Hammarström
Original assignee: Autoliv Development AB
Current assignee: Autoliv Development AB
Priority date: 2009-02-17
Filing date: 2009-02-17
Publication date: 2010-08-18
Anticipated expiration: 2029-02-17
Also published as: US8582818B2; ATE527620T1; WO2010094401A1; US20110311103A1; EP2219133B1

Abstract

A method of automatically detecting objects in front of a motor vehicle comprises the steps of pre-storing template objects (20a, 20b) representing possible objects (21a, 21b) in front of the motor vehicle, detecting images from a region in front of the vehicle by a vehicle mounted imaging means (11), generating a processed image (23) containing disparity or vehicle-to-scene distance information from said detected images, comparing said pre-stored template objects (20a, 20b) with corresponding regions-of-interest of said processed image (23), and generating a match result relating to the match between said processed image (23) and said template objects (20a, 20b). Each of said pre-stored template objects (20a, 20b) is a flat two-dimensional multi-pixel area (26a, 26b) of predetermined shape.

Description

The invention relates to a method and system of automatically detecting objects in front of a motor vehicle, comprising the steps of pre-storing template objects representing possible objects in front of the motor vehicle, detecting images from a region in front of the vehicle by a vehicle mounted imaging means, generating a processed image containing disparity or vehicle-to-scene distance information from said detected images, comparing said pre-stored template objects with corresponding regions-of-interest of said processed image, and generating a match result relating to the match between said processed image and said template objects.
US 7 263 209 B2 discloses a method according to the preamble of claim 1, comprising the steps of detecting images from a region in front of the vehicle, generating a depth map from the detected images, matching template objects to said depth map, and determining that an object is present in front of the motor vehicle if the result of said matching fulfils a predetermined condition.
The object of the invention is to provide a cost-effective method and system of automatically detecting objects in front of a motor vehicle with reduced memory requirements.
The invention solves this object with the features of the independent claims. A flat two-dimensional template object can be defined by only a few numbers requiring only a very small amount of memory. Therefore, the memory used to store and process the template objects can be significantly reduced. This has several advantages. For example, calculations involving template objects may be performed significantly faster because the number of memory accesses can be strongly reduced. As a result, the use of flat two-dimensional template objects can lead to a reduced processing time which is particularly advantageous on hardware with limited processing capabilities. Furthermore, the size of memory devices may be reduced which potentially leads to reduced costs.
Preferably the template objects are arranged orthogonally to a sensing plane. Furthermore, the template objects are preferably arranged to stand on a ground plane, because it can be assumed that objects to be detected stand on the ground; in this manner, the required number of templates can be strongly reduced.
Preferably the two-dimensional area has a rectangular shape which simplifies the image processing calculations to be performed over the region-of-interest. For the same reason, preferably to each template object at least one constant value is assigned, representing a disparity or distance value of the template object.
The processed image according to the invention preferably is a disparity image or disparity map wherein an individual disparity value, representing a difference of the corresponding point in the scene in the left/right stereo images, is assigned to every pixel of the disparity map. In general, the greater the computed disparity of an imaged pixel is, the closer the pixel is to the sensing arrangement. However, the processed image may also be a depth image or depth map wherein an individual depth value, representing the distance of the corresponding point in the scene to the vehicle, is assigned to every pixel of the depth map.
The processing time can be significantly reduced by preferably calculating an average disparity or average depth value for a region-of-interest in the measured image. The average disparity or depth value may then be subtracted from the expected disparity or depth value of the template object. The calculation of average disparity or depth values for different regions-of-interest in the measured image can be significantly simplified by the preferred generation of an integrated disparity or depth map. The integration of disparity or depth values then has to be performed only once over the whole disparity or depth map, and the calculation of the integrated disparity or depth for different regions-of-interest reduces to a simply subtraction operation involving only a few numbers.
Reliability and robustness of the object detection can be improved by calculating average disparity or depth values not over a whole region-of-interest in the measured image, but partitioning the region-of-interest into several multi-pixel sub-regions or sub-areas of predetermined shape and calculating an average disparity or depth value for every sub-region. This improves the matching reliability but still provides significant processing speed enhancement compared to the conventional pixel-by-pixel calculations.
In the following the invention shall be illustrated on the basis of preferred embodiments with reference to the accompanying drawings, in which:

Fig. 1: shows a schematic view of a safety system for a motor vehicle;
Fig. 2: shows a simplified disparity image of a region on front of a motor vehicle;
Fig. 3: shows a simplified integrated disparity image generated from a disparity image;
Fig. 4: shows a simplified disparity image with matching template objects;
Fig. 5: shows a simplified score map for the disparity map shown in Fig. 4; and
Fig. 6: shows a simplified disparity image with template objects according to another embodiment.

The safety system 10 is mounted in a motor vehicle and comprises an imaging means 11 for detecting images of a region in front of the motor vehicle. The imaging means 11 comprises one or more imaging sensors 12a, 12b, preferably one or more optical and/or infrared cameras, where infrared covers near IR and/or far IR. In one embodiment the imaging means 11 comprises stereo cameras 12a, 12b; alternatively a mono camera can be used.
The cameras 12a, 12b are coupled to an image pre-processor 14 which may be realized by a dedicated hardware circuit, and is adapted to calibrate the cameras 12a, 12b, control the capture and digitizing of the images, warp them into alignment, merge left/right images into single images, and perform pyramid wavelet decomposition to create multi-resolution disparity images, which per se is known in the art.
Image processing of the pre-processed images is then performed in image processing means 15, 16 comprising an object identification means 15 and an object tracking means 16. In the object identification means 15 preferably a disparity image or disparity map 23 is generated where an individual disparity value, representing a difference of the corresponding point in the scene in the left/right stereo images, is assigned to every pixel of the disparity map. A schematic view of an exemplary disparity map 23 with two objects to be identified is shown in Fig. 2, where however the disparity values in the third dimension are not indicated. In reality the disparity image may for example be a greyscale image where the grey value of every pixel represents a distance between the corresponding point in the scene in the left/right stereo images.
The disparity image 23 is then compared in the object identifying means 15 to template images or template objects 20a, 20b, ... contained in a template data-base 17 pre-stored in a memory. The template images 20a, 20b, ... are models of possible objects in front of the motor vehicle. The template images 20a, 20b, ... are two-dimensional images of predetermined shape which are flat in the third dimension. This means that only one constant value, for example a constant grey value, indicating the expected disparity or depth of the template object, is assigned to the template image as a whole. In particular, no information relating to individual image points or pixels is contained in the template images 20a, 20b, ....
The shape of the template objects 20a, 20b is preferably rectangular which allows to define the template object by four image coordinates of the corners and simplifies the calculation of the match score, which shall be described in the following.
In Fig. 4 only two template images 20a, 20b providing a perfect match with two exemplary objects 21a, 21b are shown. In reality, template images of different size and/or shape, representing different kind of objects, for example pedestrians 21a, vehicles 21b, bicyclists, motorbikes and so on may be provided. Furthermore, template images for a specific object are arranged at given intervals along the ground in the longitudinal and lateral direction in order to cover all possible positions of the object. Preferably the template objects 20a, 20b are arranged to stand on a ground plane 22, which is determined in advance by calibration or can be calculated from the image data, in order to reasonably reduce the number of template objects required. Furthermore, the template objects 20a, 20b are preferably arranged orthogonally to a sensing plane which is an essentially horizontal, vehicle-fixed plane through the imaging sensors 12a, 12b.
The comparison between every template image 20a, 20b, ... read out of the database 17 and the disparity image 23 may be based on a pixel-by-pixel calculation of the difference between the individual disparity value of each pixel in the detected image and the expected disparity value of the template object, where the calculation is performed over a region-of-interest 26a, 26b defined by the template object 20a, 20b. A match score may then be calculated for example as the percentage of pixels for which the absolute magnitude of the above mentioned difference is smaller than a predetermined threshold.
In comparison to the above described pixel-by-pixel difference calculation, the comparison between the template images 20a, 20b, ... and the disparity image 23 may be significantly speeded up by calculating an average disparity value of the disparity image 23 in the region-of-interest 26a, 26b defined by the template object 20a, 20b under inspection, and subtracting the constant disparity value of the template object under inspection, or vice versa. In general, the lower the absolute value of this difference of two single numbers is, the higher is the corresponding match score. In general, calculating an average disparity value of the disparity image 23 in the region-of-interest 26a, 26b can be performed significantly faster than calculating disparity differences for every pixel.
The average disparity value of the disparity image 23 in the region-of-interest can be calculated as the disparity value integrated over the region-of-interest in the disparity image 23, divided by the area of the region-of-interest. The disparity value integrated over the region-of-interest in the disparity image 23 is preferably calculated by using an integrated disparity map 27, a simplified example of which is shown in Fig. 3 where however the individual integrated disparity values assigned to the pixels are not visualized. In reality the integrated disparity map 27 may for example be a greyscale image where the greyscale value of every pixel indicates the corresponding integrated disparity value.
In the integrated disparity map 27, to each pixel an individual integrated disparity value is assigned. The integrated disparity value assigned to a pixel may be calculated as the sum of all disparity values of a region suitably related to that point. For example in Fig. 3, the value assigned to a pixel in the integrated disparity map 27 may be calculated as the sum of the disparity values of all pixels which are located to the top and to the left of the pixel under consideration. This means that the integrated disparity value of point P1 is the sum of all disparity values in the region R1 of the disparity map 23; the integrated disparity value of point P2 is the sum of all disparity values in the regions R1 and R2; and so on. The integrated disparity value in the region-of-interest R4 defined by the corner points P1, P2, P3, P4 is then the integrated disparity value of P1 plus the integrated disparity value of P4 minus the integrated disparity value of P2 minus the integrated disparity value of P3. In other words, once the integrated disparity map 27 has been calculated, the calculation of the difference between a template object and the disparity map in the corresponding region-of-interest reduces to a simple arithmetic operation involving only a few, here five, numbers.
Of course there is other ways of calculating an integrated disparity map 27; for example the integrated disparity value assigned to a pixel in the integrated disparity map may be calculated as the sum of the disparity values assigned to all pixels to the top and to the right of that point, etc.
If average disparity values are used as described above, the reliability and robustness of the object detection can be improved according to a further embodiment of the invention exemplary illustrated in Fig. 6. In this embodiment, the template images 20a, 20b are composed of a plurality of preferably rectangular sub-areas 28a1, 28a2, 28a3, ...; 28b1, 28b2, 28b3, ..., forming a contiguous, preferably rectangular template area 26a, 26b. Each sub-area 28a, 28b comprises a plurality of pixels, i.e. forms at least a (2 pixel)x(2 pixel) area, where pixel as usually means an elementary image point. Still only one constant disparity value, for example a constant grey value, indicating the expected disparity of the template object is assigned to each template object 20a, 20b. In this embodiment, the average disparity value is not calculated over the whole region-of-interest 26a, 26b, but an individual average disparity value is calculated for each sub-area 28a, 28b of a template object 26a, 26b under consideration. A match score may then be calculated for example as the percentage of sub-areas 28a, 28b for which the absolute magnitude of the difference between the disparity value of the sub-area under consideration and the expected disparity value of the corresponding template object is smaller than a predetermined threshold. The average disparity value in each sub-region can be calculated in the same manner as described above by using an integrated disparity map. In this embodiment, the risk of falsely detecting ghost objects can be strongly reduced, while still consuming significantly less processional resources as compared to a pixel-by-pixel calculation of disparity differences.
A combination of the above-described embodiments, comprising template images for which an overall average disparity value is calculated and template images for which average disparity values in a plurality of sub-regions are calculated, is also possible.
A preferred method of identifying an object based on the above comparison is carried out in the following manner. For every template image 20a, 20b, ... read out of the database 17 the calculated score value is saved to a score map 24, an example of which is shown in Fig. 5. In the score map 24, one axis (here the horizontal axis) corresponds to the horizontal axis of the detected image and the other axis (here the vertical axis) is the longitudinal distance axis. Therefore, the score map 24 may be regarded as a birds-eye view on the region in front of the vehicle. All calculated score values are inserted into the score map 24 at the corresponding position in form of a small patch field to which the corresponding score value is assigned for example in form of a grey value.
In order to ensure only as few detections as possible, optimally only one detection per object, in the final score map 24 preferably the local extrema (local maxima or local minima depending on how the score is calculated) are identified and a non-extremum suppression is carried out, leaving left only the extremal score values. For the exemplary disparity map 23 shown in Fig. 4, the score map 24 after non-extremum suppression is shown in Fig. 5, where for each object 20a, 20b to be identified one corresponding score value 25a, 25b is present in the score map 24.
An additional step of verifying or validating an object may be provided in the object identification means 15, for example discriminating the match score of the object against a predetermined threshold, and/or using a validation signal from a sensor independent of the imaging sensor 12.
An object identified in the object identification means 15 is then tracked by the object tracking means 16.
Based on the information on the object output by the image processing means 15, 16, for example object distance, position, speed, size and/or classification, a decision means 18 may activate or control safety means 29, 30. For example if the decision means 18 determines that there is a non-negligible risk of a collision with a detected object, a warning means 29 adapted to warn the driver is preferably activated. The warning means 29 suitably provides optical, acoustical and/or haptical warning signals. Further safety means not shown in Fig. 1 may be activated or suitably controlled, for example restraint systems such as occupant airbags or safety belt tensioners; pedestrian airbags, hood lifters and the like; or dynamic vehicle control systems such as brakes.
On the other hand, if the decision means 18 determines that the risk of collision with a detected object is negligible, for example in case of a pedestrian at the safe roadside, the decision means 18 may control a display means 30 to display information indicating the object, only.
The object identification means 15, object tracking means 16 and/or the decision means 18 are preferably realized in a programmable electronic control means which may comprise a microprocessor or micro-controller. The template data-base 17 is preferably stored in an electronic memory. The programmable electronic control means and the electronic memory may be realised for example in an on-board ECU and may be connected to the sensing arrangement 12a, 12b and the safety means 29, 30 via an on-board data bus. All steps from imaging, image preprocessing, image processing, to activation or control of safety 29, 30 means are performed continuously during driving in real time.
In the above described embodiments the pre-stored template objects are matched to a disparity map, which has the advantage that the time-consuming calculation of a depth map can be avoided. However, it is also possible to match the pre-stored template objects to a calculated depth map where an individual depth value representing the distance of the corresponding point in the scene to the vehicle, obtained from the distance information contained in the disparity image, is assigned to every pixel of the depth map.

Claims

A method of automatically detecting objects in front of a motor vehicle, comprising the steps of:
- pre-storing template objects (20a, 20b) representing possible objects (21a, 21b) in front of the motor vehicle,

- detecting images from a region in front of the vehicle by a vehicle mounted imaging means (11),

- generating a processed image (23), containing disparity or vehicle-to-scene distance information, from said detected images,

- comparing said pre-stored template objects (20a, 20b) with corresponding regions-of-interest of said processed image (23), and

- generating a match result relating to the match between said processed image (23) and said template objects (20a, 20b),
characterized in that each of said pre-stored template objects (20a, 20b) is a flat two-dimensional multi-pixel area (26a, 26b) of predetermined shape.
The method as claimed in claim 1, wherein said template objects (20a, 20b) are arranged orthogonally to a sensing plane.
The method as claimed in claim 1 or 2, wherein said template objects (20a, 20b) are arranged to stand on a ground plane.
The method according to any one of the preceding claims, wherein said two-dimensional area (26a, 26b) has a rectangular shape.
The method according to any one of the preceding claims, wherein to each template object (20a, 20b) one constant value preferably related to an expected disparity of the template object (20a, 20b) is assigned.
The method according to any one of the preceding claims, comprising the step of generating an integrated processed image (27) from said processed image (23).
The method according to any one of the preceding claims, comprising the step of calculating an average value of said processed image (23) in a region-of-interest (26a, 26b) defined by a template object(20a, 20b).
The method according to claim 7, wherein a template object (20a, 20b) is composed of a plurality of multi-pixel sub-regions (28a, 28b) of predetermined shape.
The method according to claim 8, comprising the step of calculating an average value of said processed image (23) for each of said sub-regions (28a, 28b).
The method according to claim 8 or 9, wherein said sub-regions (28a, 28b) have a rectangular shape.
The method according to any one of the preceding claims, comprising the step of calculating a match score for each template.
The method according to claim 11, comprising the step of generating a score map (24) from said match scores.
The method according to claim 12, comprising the step of identifying local extrema (25a, 25b) in said score map (24).
The method according to claim 13, comprising the step of performing a non-extremum suppression in said score map (24).
A safety system for a motor vehicle, comprising imaging means (11) adapted to detect images of a region in front of the motor vehicle, memory means (17) adapted to pre-store template objects (20a, 20b) representing possible objects (21a, 21b) in front of the motor vehicle, data processing means (15, 16, 18) adapted to carry out the data processing steps in the method according to any one of the preceding claims, and safety means (29, 30) controlled by said data processing means (15, 16, 18) depending on the result of said data processing.