WO2014095779A1

WO2014095779A1 - Method for differentiating features of a target object and ground features in an image of a camera, camera system for a motor vehicle and motor vehicle

Info

Publication number: WO2014095779A1
Application number: PCT/EP2013/076797
Authority: WO
Inventors: Ciaran Hughes; Sean Finn
Original assignee: Connaught Electronics Ltd
Current assignee: Connaught Electronics Ltd
Priority date: 2012-12-17
Filing date: 2013-12-17
Publication date: 2014-06-26
Anticipated expiration: 2015-06-17
Also published as: DE102012024662A1

Abstract

The invention relates to a method for detecting target objects (5) in an environmental region (6) of a motor vehicle (1) based on an image of the environmental region (6), which is provided by means of a camera (3) of the motor vehicle (1), wherein characteristic features in the image are identified by means of an electronic evaluation device of the motor vehicle (1) and the detection of the target objects (5) includes that it is differentiated between target features associated with the target objects (5) and ground features associated with a ground (4) of the environmental region (6). In the image, a differentiating line is defined depending on a position of the camera (3) on the motor vehicle (1), wherein the differentiation between the target features and the ground features is performed depending on a position of the features in the image with respect to the differentiating line.

Description

Method for differentiating between features of a target object and ground features in an image of a camera, camera system for a motor vehicle and motor vehicle

The invention relates to a method for detecting target objects in an environmental region of a motor vehicle based on an image of the environmental region, which is provided by means of a camera of the motor vehicle. Characteristic features (edges and/or corners or the like) are identified in the image by means of an electronic evaluation device of the motor vehicle, wherein the detection of the target objects includes that it is differentiated between target features associated with the target objects and ground features associated with a ground of the environmental region. In addition, the invention relates to a camera system for performing such a method as well as to a motor vehicle with such a camera system.

Camera systems for motor vehicles are already known from the prior art. As is known, a camera system includes at least one camera, which is attached to the motor vehicle and captures an environmental region of the motor vehicle. Multiple such cameras can also be employed, which capture the entire environment around the motor vehicle. The at least one camera mounted on the motor vehicle provides a temporal sequence of images of the environmental region, namely a plurality of images per second. This temporal sequence of images is then communicated to an electronic evaluation device processing the captured images and being able to provide very different functionalities in the motor vehicle based on the images. Presently, the interest is directed to the detection of target objects located in the imaged environmental region. Therein, an object having a certain height above the ground and thus being differentiated from objects or signs situated on the ground itself or applied to the ground is understood by the term "target object". If a target object is detected in the images, thus, this target object can be tracked in the sequence of images. For this purpose, the optical flow method is usually used, in which characteristic features such as for example edges and/or corners are detected in the images and a flow vector is calculated to each characteristic feature, which specifies the direction of movement and the velocity of movement of this characteristic feature in the sequence of images.

In the detection of target objects in the camera images, problems with respect to the differentiation between characteristic features of actual target objects on the one hand and characteristic features associated with the ground itself, namely for example with signs applied to the ground or else with objects having a very low height above the ground, on the other hand can arise. While the characteristic features of real target objects are relevant, the features of the ground do not present any obstacle to the motor vehicle. In order to be able to differentiate between the target features on the one hand and the ground features on the other hand, in the prior art, a flow vector is calculated to each characteristic feature and then the associated feature is classified based on the flow vector. However, the following problems arise here: if the motor vehicle itself moves, thus, flow vectors not only arise on movable objects, but also in the entire image frame because the camera also moves relatively to the imaged scene. Thus, the flow vectors arise both in those image regions, which represent the ground of the environmental region, and in those image regions, in which target objects located above the ground with a certain height are imaged. Thus, it is a particular challenge to differentiate the flow vectors associated with the target objects from the vectors of the ground. Therefore, in the prior art, the current movement vector of the motor vehicle itself is additionally also taken into account. By a comparison between the flow vectors and the movement vector of the motor vehicle, differentiation between the ground features on the one hand and the target features on the other hand can then be allowed. However, it has turned out that usually a very high number of characteristic features is detected in the images and thus it can occur that strong features less relevant are detected and evaluated, while more important, but weaker features are no longer evaluated due to the high number of features.

It is an object of the invention to demonstrate a solution how in a method of the initially mentioned kind, it can be reliably differentiated between the target features on the one hand and the ground features on the other hand in simple manner.

According to the invention, this object is solved by a method, by a camera system as well as by a motor vehicle having the features according to the respectively independent claims. Advantageous implementations of the invention are the subject matter of the dependent claims, of the description and of the figures.

A method according to the invention serves for detecting target objects in an

environmental region of a motor vehicle based on an image of the environmental region. The image is provided by means of a camera of the motor vehicle. An electronic evaluation device of the motor vehicle identifies characteristic features in the image and differentiates between target features on the one hand, which are associated with the actual target objects, and ground features on the other hand, which are associated with a ground of the environmental region. A differentiating line is defined in the image depending on a position of the camera on the motor vehicle and the differentiation between the target features and the ground features is performed depending on a position of the features in the image with respect to the differentiating line.

Thus, by a simple evaluation of the position of the features with respect to the

differentiating line - even with a single image - it can be reliably differentiated between the target features on the one hand and the ground features on the other hand. Therein, the invention is based on the realization that features situated higher than the camera itself in the three-dimensional vehicle coordinate system must be associated with actual target objects, while the features situated below the camera are possibly associated with the ground. Such a differentiation can be performed by defining a differentiating line in the image without much effort by examining on which side of the differentiating line the features are located without much computational effort. Thus, for example, a first sorting of the features can be performed: the features located on the one side of the

differentiating line can be uniquely classified as target features, while flow vectors can be additionally determined to the other features located on the other side in order to allow reliable classification of these features. This approach has the advantage that a plurality of features can be uniquely classified as target features solely based on the differentiating line, without having to calculate flow vectors to these features. The flow vectors can be calculated exclusively to the other features such that the number of the calculated flow vectors considerably decreases compared to the prior art and the weaker, but usually more important features located in the foreground can also be evaluated.

Generally, different methods can be applied to extract characteristic features from the image. For example, the so-called Harris points can be detected, or further methods such as SIFT, SURF, ORB or the like can also be used.

Preferably, the differentiating line is defined depending on a level or height, on which the camera is disposed on the motor vehicle. Thus, it is managed to differentiate those features located above the camera from those features located below the camera.

Therefore, target features can be detected without much effort.

The definition of the differentiating line can include that an artificial horizon on the level of the camera is defined in a three-dimensional coordinate system, in particular the vehicle coordinate system, and the differentiating line is determined by projection of the artificial horizon of the camera into the two-dimensional coordinate system of the image. In this manner, a differentiating line can be defined, which corresponds to the artificial horizon of the camera in the image - also considering the optics of the camera (it can be a fish eye camera). All of the features located above the differentiating line can therefore be interpreted as features located above the camera in the three-dimensional vehicle coordinate system. These features can then be uniquely categorized as target features. In contrast, those features located below the differentiating line, can be interpreted either as target features or as ground features - preferably, additional plausibility check is performed to these features. In particular, this approach has the advantage that those features located above the differentiating line, can be uniquely classified as target features without having to calculate flow vectors to these features. Now, the flow vectors can be calculated exclusively to those features, which are located below the differentiating line and thus cannot be uniquely classified as ground features or target features. Compared to the prior art, thus, the number of the calculated flow vectors greatly decreases. Therein, the horizon can either be defined at the level of the camera or it can be provided a little higher than the level of the camera in order to compensate for the pitch angle and the roll angle of the vehicle.

In an embodiment, it is therefore provided that those features located above the differentiating line in the image are interpreted as target features. Preferably, further plausibility check is not performed to these features. Namely, if the features are above the differentiating line, thus, it can be uniquely determined solely based on this line that these features are associated with actual target objects having a certain height above the ground.

In contrast, optical flow vectors can additionally be determined to those features located below the differentiating line in the image based on at least two images of the camera. Depending on the flow vectors, the decision can then be made plausible if these features are classified as target features or as ground features. Thus, the optical flow method can be applied to these features located below the differentiating line in order to be able to perform a final classification. Herein, the flow vectors can also be compared to a current movement vector of the motor vehicle in order to be able to uniquely differentiate between target features on the one hand and ground features on the other hand.

It can also be provided that the image is divided in a plurality of image cells, which each are composed of multiple pixels. Then, it can be differentiated between target cells including at least one target feature and the remaining image cells. Such a division of the image in multiple image cells has the advantage that the computational effort can be overall reduced because not the individual features, but larger image cells are evaluated. It can also occur that characteristic features cannot be identified in an image region of the image, for example because this image region has a homogenous coloration. For example, this can occur if a wall of a garage is depicted in the image. If such an image region is detected, which has a certain size and does not have any characteristic features, thus, this image region can be segmented according to a predetermined segmentation criterion. The segmentation criterion can for example include that image regions are segmented, which have a homogenous color distribution and/or at least a preset size. The segmented image region can then be interpreted as associated with an actual target object (for example a wall) if at least a part of the image region is located above the differentiating line. If a part of the image region is located above the differentiating line, thus, it can be assumed with very high likelihood that this segmented image region is associated with one and the same target object, which presents an actual obstacle to the motor vehicle. This segmented image region can therefore be classified to the effect that it is associated with a target object. Thus, target objects can also be detected, to which characteristic features cannot be identified.

A camera system according to the invention for a motor vehicle includes a camera formed for providing an image of an environmental region of the motor vehicle, as well as an electronic evaluation device adapted to identify characteristic features in the image and to differentiate between target features and ground features. The evaluation device can differentiate between the target features and the ground features depending on a position of the features in the image with respect to a differentiating line defined in the image depending on a position of the camera on the motor vehicle.

A motor vehicle according to the invention, in particular a passenger car, includes a camera system according to the invention.

The preferred embodiments presented with respect to the method according to the invention and the advantages thereof correspondingly apply to the camera system according to the invention as well as to the motor vehicle according to the invention.

Further features of the invention are apparent from the claims, the figures and the description of figures. All of the features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations or else alone. Now, the invention is explained in more detail based on a preferred embodiment as well as with reference to the attached drawings.

There show:

Fig. 1 in schematic illustration a motor vehicle with a camera system according to an embodiment of the invention; and

Fig. 2 to 4 an exemplary image of an environmental region of the motor vehicle.

wherein a method according to an embodiment of the invention is explained in more detail.

In Fig. 1 , in schematic and perspective illustration, a motor vehicle 1 according to an embodiment of the invention is shown. The motor vehicle 1 is for example a passenger car. It has a camera system 2 including a camera 3, in particular a mono camera. The motor vehicle 1 is on a ground 4, for example a road. On the ground 4, there is for example also a target object 5, i.e. an obstacle with a certain height above the ground 4. The camera system 2 is for example a collision warning system and serves for warning the driver of the motor vehicle 1 of the presence of the target object 5 in an environmental region 6 of the motor vehicle 1 . In particular, the camera system 2 serves for detecting the target object 5 and preferably also for tracking the target object 5. Therein, images are captured by means of the camera 3, which are then processed by means of an electronic evaluation device (signal processor) not illustrated in more detail. The evaluation device receives the captured images and processes them. The evaluation device can be integrated in the camera 3 or it can be a component separate from the camera 3. The images can also be displayed on a display in the motor vehicle 1 , wherein the detection of the target object 5 can occur to the effect that the target object 5 is provided with a border in the images

For example, the camera 3 has a capturing angle, which can be in a range of values from 90° to 200°. The camera 3 can be a CMOS camera or a CCD camera or any image capturing device formed for detecting light in the visible spectral range. Preferably, the camera 3 is a video camera continuously providing a sequence of images. The electronic evaluation device then processes the image sequence in real time and can detect and track the target object 5 based on this image sequence. In the embodiment according to Fig. 1 , the camera 3 is disposed in a front region of the motor vehicle 1 and captures the environmental region 6 in front of the motor vehicle 1. For example, the camera 3 can be disposed on the front bumper 7 or else behind a windshield 8. However, the invention is not restricted to such an arrangement of the camera 3 in the front region. The arrangement of the camera 3 can be different according to embodiment. For example, the camera 3 or an additional camera can also be disposed in a rear region of the motor vehicle 1 and capture the environmental region behind the motor vehicle 1. Multiple cameras 3 can also be employed, which each capture a separate environmental region.

In the development of the camera system 2 or in the manufacture of the motor vehicle 1 - i.e. in initialization of the camera 3 - an artificial horizon 9 is defined for the camera 3. This artificial horizon 9 can be determined in the form of a polygon to reduce the computational effort. Thus, a line can be defined as the artificial horizon 9, which connects several points 10 to each other. Preferably, a line is defined as the artificial horizon 9, which is situated on the level of the camera 3. This means that a horizontal plane defined by the artificial horizon 9 also contains the camera 3 such that the camera 3 is located in this plane. For determining the artificial horizon 9, preferably, an annular line is defined, which extends around the camera 3, wherein the center of this annulus is defined by the camera 3 itself. The radius of this annulus can be arbitrarily selected.

Alternatively, it can also be provided that the points 10 and thus the artificial horizon 9 are situated in a vertical offset to the camera 3. This means in particular that the plane defined by the points 10 can be placed a little higher than the camera 3 and thus over the camera 3. This approach is advantageous regarding the pitch angle and the roll angle of the motor vehicle 1 .

If such an artificial horizon 9 is defined, thus, this horizon 9 can be projected from the vehicle coordinate system x, y, z into a two-dimensional image coordinate system x', y'. An exemplary image 1 1 of the camera 3 is illustrated in Fig. 2. By projection of the artificial horizon 9 into the image coordinate system x', y', a differentiating line 9' is provided. This differentiating line 9' therefore represents the artificial horizon of the camera 3 in the image coordinate system x', y'. In the transformation of the horizon 9 into the image coordinate system x', y', the position of the camera 3 on the motor vehicle 1 on the one hand and also the configuration of the lens of the camera 3 on the other hand is taken into account. As is apparent from Fig. 2, the camera 3 can be a fish eye camera having a relatively wide opening angle. The differentiating line 9' extends from the left side towards the right side of the image 1 1. It has the shape of an arc. This shape applies for a fisheye camera. But, if the camera 3 is not a fisheye camera, then the differentiating line 9' will have a shape of a straight line.

Furthermore, if the horizon 9 is modelled in the 3D space as a series of points 10 that generate a 3D polygon, then the projected arc or the differentiating line 9' on the image is also modelled as a polygon, i.e. open polygon.

Now, characteristic image features 12 such as for example edges and/or corners are extracted from the image 1 1 by means of the evaluation device. Such algorithms serving for detecting characteristic image features are already prior art. The evaluation device then examines the position of the features 12 in the image 1 1 relative to the differentiating line 9'. Namely, the evaluation device examines if the features 12 are located above or below the differentiating line 9'. Those features 12, which are located above the differentiating line 9', are classified as target features, which are associated with actual target objects having a certain height above the ground 4. Those features located below the differentiating line 9' in turn are subjected to further plausibility check. In the next image 1 1 , the associated features are detected to these features, and flow vectors are calculated. The plausibility check is then effected depending on the flow vectors, namely for example by a comparison of the flow vectors to a movement vector of the motor vehicle 1 . Based on this plausibility check, then, the features 12 located below the differentiating line 9' are classified either as target features or as ground features.

As is apparent from Fig. 3, the image 1 1 can be divided in a plurality of image cells 13. To each image cell 13, it can be examined if this image cell 13 includes features above the differentiating line 9' and/or features below the differentiating line 9'. Those image cells 13, which include features above the differentiating line 9', are classified as target cells - they are denoted by 13a in Fig. 3. In contrast, image cells 13 having features 12 below the differentiating line 9' are first classified as neutral cells 13b. If image cells 13 are detected, which include both features 12 above the differentiating line 9' and features 12 below the differentiating line 9', thus, two alternative embodiments are possible: those image cells 13 including at least one feature 12 above the differentiating line 9' can be classified as target cells 13a. Alternatively, the number of the features 12 above the differentiating line 9' can also be compared to the number of the features 12 below the differentiating line 9' within a single image cell 13 and this image cell 13 can be classified as a target cell 13a or as a neutral cell 13b depending on whether this image cell 13 includes more features above the line 9' or below the line 9'.

The target cells 13a are interpreted to the effect that they are associated with an actual target object. These target cells 13a do not have to be tracked in further images 1 1 . Flow vectors are calculated exclusively to those image cells 13, which are classified as neutral cells 13b.

The image cells 13 classified as neutral cells 13b are now made plausibly by means of the optical flow method. In the further step, these neutral cells 13b can be classified as target objects or else as the ground itself and thus either as target cells or ground cells. Herein, it is differentiated between two situations: if the motor vehicle 1 does not move, thus, the neutral cells 13b are interpreted as associated with an actual target object 5, if flow vectors can be detected to these neutral cells 13b. If a flow vector is not detected in a neutral cell 13b, thus, this cell 13b is interpreted to the effect that it is associated with the ground 4 - ground cell.

If the motor vehicle 1 moves, flow vectors arise also on those neutral cells 13b, which are associated with the ground 4 itself. These flow vectors can now be corrected with the current movement vector of the motor vehicle 1 , and it can also be differentiated between neutral cells 13b associated with the ground 4 and cells 13b associated with an actual target object 5.

Overall, the described method has the advantage that the flow vectors have to be calculated exclusively to those features 12, which are located below the differentiating line 9'. Especially these features 12 are important because they could for example be associated with a leg of a pedestrian. Thus, exclusively those features have to be further tracked, which are located below the differentiating line 9'.

It can also occur that an area is detected in the image 1 1 , which has a homogenous coloration and thus does not have any characteristic features 12. Such an image region 14 is exemplarily illustrated in Fig. 4. In this image region 14, a wall of a garage is depicted. Such an image region 14 can for example be detected in that multiple image cells 13 next to each other are identified without characteristic features. If the size of the image region 14 (number of image cells 13) is greater than a limit value and/or the coloration is homogenous, thus, this image region 14 can be segmented and it can be examined if at least a part of the image region 14 is located above the differentiating line 9'. Such a situation is depicted in Fig. 4. If this is the case, thus, this image region 14 can be classified to the effect that it is associated with an actual target object 5, in the present case with the wall of the garage.

Claims

1 . Method for detecting target objects (5) in an environmental region (6) of a motor vehicle (1 ) based on an image (1 1 ) of the environmental region (6), which is provided by means of a camera (3) of the motor vehicle (1 ), wherein characteristic features (12) in the image (1 1 ) are identified by means of an electronic evaluation device of the motor vehicle (1 ) and the detection of the target objects (5) includes that it is differentiated between target features (12) associated with the target objects (5) and ground features (12) associated with a ground (4) of the

environmental region (6),

characterized in that

a differentiating line (9') is defined in the image (1 1 ) depending on a position of the camera (3) on the motor vehicle (1 ), and that the differentiation between the target features (12) and the ground features (12) is performed depending on a position of the features (12) in the image (1 1 ) with respect to the differentiating line (9').

2. Method according to claim 1 ,

characterized in that

the differentiating line (9') is defined depending on a level, on which the camera (3) is disposed on the motor vehicle (1 ).

3. Method according to claim 1 or 2,

characterized in that

the definition of the differentiating line (9') includes that an artificial horizon (9) of the camera (3) is defined in a three-dimensional coordinate system (x, y, z), in particular a vehicle coordinate system, and the differentiating line (9') is determined by a projection of the artificial horizon (9) into a two-dimensional image coordinate system (χ', y').

4. Method according to any one of the preceding claims,

characterized in that

those features (12) located above the differentiating line (9') in the image (1 1 ) are interpreted as target features (12).

5. Method according to any one of the preceding claims,

characterized in that

to those features (12) located below the differentiating line (9') in the image (1 1 ), optical flow vectors are determined based on at least two images (1 1 ) of the camera (3), and depending on the flow vectors it is decided if these features (12) are interpreted as target features (12) or as ground features (12).

6. Method according to any one of the preceding claims,

characterized in that

the image (1 1 ) is divided in a plurality of image cells (13) and the detection of the target objects (5) includes that it is differentiated between target cells (13a) including at least one target feature (12) and other image cells (13b).

7. Method according to any one of the preceding claims,

characterized in that

if characteristic features (12) are not identified in an image region (14) of the image

(1 1 ) , this image region (14) is segmented according to a predetermined

segmentation criterion, wherein the segmented image region (14) is interpreted as associated with a target object (5) if at least a part of the image region (14) is located above the differentiating line (9').

8. Camera system (2) for a motor vehicle (1 ), including a camera (3) for capturing an image (1 1 ) of an environmental region (6) of the motor vehicle (1 ), and including an electronic evaluation device adapted to identify characteristic features (12) in the image (1 1 ) and to differentiate between target features (12) associated with a target object (5) and ground features (12) associated with a ground (4) of the

environmental region (6),

characterized in that

the evaluation device is adapted to perform the differentiation between the target features (12) and the ground features (12) depending on a position of the features

(12) in the image (1 1 ) with respect to a differentiating line (9') defined in the image (1 1 ) depending on a position of the camera (3) on the motor vehicle (1 ).

9. Motor vehicle (1 ) with a camera system (2) according to claim 8.