DE102011108468A1 - Method of generating three-dimensional environment information of motor vehicle, involves increasing occupancy probability of voxels of voxel based environment map, such that voxels are contacted or cut lines of sight - Google Patents

Abstract

The method involves generating images (B) by image capture device such as camera (2) at time period (t1) and detecting image feature in the image. The position and/or orientation between time periods are changed. The lines of sight (S1,S2) are determined by image capture device for imaging the image features respectively. An occupancy probability of voxels of voxel based environment map (6) is increased, such that the voxels are contacted or cut lines of sight. An independent claim is included for a device for generating three-dimensional environment information of motor vehicle.

Description

The invention relates to a method and a device for generating three-dimensional environment information, in particular of three-dimensional environment information of a motor vehicle.

Especially in modern motor vehicles, it is desirable to determine environmental information by means of technical devices to assist a motor vehicle driver in the operation of the motor vehicle. So z. For example, obstacles may be extracted from environmental information and a motor vehicle driver warned if a potential collision between the motor vehicle and the obstacle occurs or can occur.

Three-dimensional environmental information provides a high level of information, which enables high-quality support of the motor vehicle driver.

The DE 2009 016 562 A1 discloses a method for detecting objects in an environment of a vehicle, wherein a first image of the surroundings of the vehicle is detected by means of at least one camera arranged on the vehicle at a first vehicle position, a second image of the surroundings of the vehicle is detected at a second vehicle position and both captured images are evaluated in an image processing and evaluation. In this case, a change in position of the vehicle between the first vehicle position and the second vehicle position is determined by determining an optical flow based on at least one characteristic point shown in both images and / or by sensors for determining a travel path of the vehicle. Using the determined position change positions and dimensions of objects shown in the images are determined in the environment of the vehicle and generates a three-dimensional environment map of the detected environment of the vehicle with a height profile.

In this case, it is disadvantageous that in all captured images at least one characteristic point, which is represented in all captured images, must be determined in order to create the three-dimensional map. For this purpose, complex and compute-intensive image processing algorithms are necessary because corresponding points in the captured images must be determined.

Another disadvantage is that incorrectly assigned corresponding point pairs lead to incorrect environment information.

The technical problem arises of providing a method and a device for generating three-dimensional surroundings information, in particular of three-dimensional surroundings information of a motor vehicle, which reduce requirements for computing power or computing capacity for generating this environment information.

The solution of the technical problem results from the objects with the features of claims 1 and 8 further advantageous embodiments of the invention will become apparent from the dependent claims.

Proposed is a method for generating three-dimensional environment information, in particular of three-dimensional environment information relating to an environment of a motor vehicle.

In this case, a first image is generated by means of a device for image acquisition at a first time and at least one further time at least one further image. Preferably, by means of the device for image acquisition, images are generated at a predetermined acquisition rate, for example 30 images per second (30 fps) or 60 fps, environment information being generated as a function of all or at least part of all acquired images.

The first time is different from the at least one other time. In particular, the first time is before the at least one further time. Furthermore, the device for image acquisition changes a position and / or orientation between the first and the at least one further point in time. In this case, a change in the position and / or orientation of the device for image acquisition with respect to a predetermined three-dimensional reference coordinate system can be detected, for. Example by means of at least one suitable sensor for detecting an absolute position and / or an absolute orientation or by means of at least one suitable sensor for detecting a change in position and / or orientation change. The reference coordinate system can in this case be geostationary or geofest, that is geographically fixed referenced. For example, the reference coordinate system may be a GPS-based coordinate system.

The image generated by the device for image acquisition hereby is a two-dimensional, preferably pixel-based, image which contains no depth information.

At least a first image feature is detected in the first image. The first image feature here denotes a characteristic feature in the first image. The first image feature may be, for example, an intensity-based feature and / or a color-value-based feature and / or a model-based feature. For this purpose, methods known in the art for feature recognition or feature detection in images can be used.

For example, methods for feature recognition may be known methods for object recognition. Analogously, methods for feature detection can be known methods for object detection. For example, model-based features can be detected by means of known model-based methods for object detection. The model-based methods for object recognition include, for. B. Method for edge detection. Of course, further, known in the art features in the first image can be detected.

Analogously, at least one further image feature is detected in the at least one further image.

Thus, therefore, image features are detected at different times in different images. It is essential here that the at least one first image feature of the first image does not have to be a feature corresponding to the at least one further image feature of the at least one further image. The at least one first image feature of the first image may thus be different from the at least one further image feature of the at least one further image; in particular, these image features may have been generated by different objects.

According to the invention, a first line of sight is determined by the device for image acquisition to the first image feature. This determination can be made for example in the geofesten reference coordinate system described above.

For example, the device for image acquisition, a coordinate system, for. As a camera coordinate system, be assigned, whose position and orientation to the reference coordinate system described above is known or determinable. Further, image coordinates of the image features with respect to the coordinate system of the image capture device may be determined. For example, an image coordinate of a, z. B. geometric, center of gravity of the image feature. Due to the known or determinable position and orientation of the coordinate system of the device for image acquisition with respect to the reference coordinate system, these, in particular pixel-related, image coordinates can be transformed into the reference coordinate system.

An origin of the coordinate system of the device for image acquisition can also be transformed into the reference coordinate system, wherein three-dimensional coordinates of the origin of the coordinate system with respect to the reference coordinate system are known or determinable.

Since the two-dimensional image contains no depth information, only one z. B. from the origin of the transformed coordinate system of the device for image acquisition outgoing line of sight are determined on which the detected image feature is located. Thus, the first line of sight is determined by the image capture device, particularly an origin of a coordinate system of the image capture device, to the first image feature in the reference coordinate system.

Furthermore, an occupancy probability of voxels (volume pixels) of a voxel-based environment map is increased, which are touched or cut from the first line of sight. In the voxel-based environment map in this case, the three-dimensional environment information in the previously described voxels can be stored, with a geometric relationship between voxel coordinates and the reference coordinate system is known.

The occupancy probability can z. By means of a counter variable associated with a voxel, the counter variable being incremented if the first line of sight touches or intersects the respective voxel. In this case, a height of the increment z. B. be dependent on a distance of the line of sight from a voxel center. For example, the further the line of sight from the center of the voxel touches or intersects the voxel, the lower the height of the increment can be.

Analogously, a further line of sight is determined by the device for image acquisition to the at least one further image feature of the at least one further image, whereby an occupancy probability of voxels of the voxel-based environment map is increased, which are touched or cut by the further line of sight.

This advantageously results in a method for generating three-dimensional environment information which requires no computation-intensive determination of corresponding points or corresponding image features in images recorded at different times and at different positions and / or with different orientations.

Here it is assumed that the device for image acquisition calibrated and thus a geometric transformation of the image coordinate system in the reference coordinate system is possible. For example, intrinsic and extrinsic camera parameters may be known if the image capture device is a camera.

Increasing the occupancy probability for voxels that are intersected by lines of sight advantageously leads, in particular, to an occupancy probability of voxels or voxel areas that correspond to a position or an area. at which or in which an actually existing object is arranged in the environment of the device for image acquisition, be increased more than an occupancy probability of voxels or voxel areas that correspond to a position or an area. at which or in which no object is arranged.

In this way, a robust method for generating three-dimensional environment information is created in an advantageous manner, since individual erroneous determinations of image characteristics lead to a faulty increase in the occupancy probability, but these individual erroneous elevations in relation to the increase due to line of sight, which are determined for real existing objects , is much lower. For the proposed method, it can be readily assumed that increasing an occupancy probability for voxels or voxel areas that correspond to a position or an area. at which or in which an actually existing object is arranged in the environment of the device for image acquisition, occurs significantly more frequently than for voxels or voxel regions which correspond to a position or an area at which or in which no object is arranged.

A line of sight can hereby be characterized or described by an orientation with respect to the reference coordinate system. For example, a line-of-sight may be characterized or described by angles, for example by angles described according to the so-called yaw-pich-roll convention.

Advantageously, the proposed method requires less computational intensity, since no corresponding point pairs have to be detected in different images and thus no computationally intensive operations have to be performed.

In a further embodiment, the occupancy probability is evaluated for each individual voxel or for at least one voxel area with a predetermined size. Furthermore, at least one maximum of the occupancy probability is determined, which was evaluated in the previous step. Furthermore, the voxel or the at least one voxel area is assigned a detected object which has the at least one maximum. Thus, in the environment map, an object is detected at the voxel or at the voxel area having the at least one maximum.

For example, a voxel region of a predetermined size may be a 3x3x3 voxel region. Of course, depending on the resolution of the environment map and the size of the objects to be detected, different sizes of the voxel area can also be imagined.

The occupancy probability can be evaluated after each time of image acquisition. This results in an advantageously updated constantly evaluation of the maximum or local maxima of the occupancy probability in the environment map. Also, the occupancy probability after a predetermined number of times of image capture, z. B. after five consecutive times of image acquisition, be re-evaluated. As a result, a required computing power for evaluating the occupancy probability is advantageously reduced.

In this case, it is assumed that in the geographically fixed referenced area map only those voxels or those voxel areas that correspond to a position or an area at which or in which an actually existing object is arranged in the environment of the device for image acquisition, a high occupancy probability and thus, will have a maximum occupancy probability relative to the remaining voxels or voxels.

By evaluating the maximum, an object can be detected and determined in a simple manner in a simple manner in the three-dimensional environment map.

In a preferred embodiment, local maxima of the occupancy probability are determined, wherein the voxels or the voxel areas are each assigned a detected object having the local maxima. Thus, in the environment map, therefore, objects are detected at the voxels or at the voxel areas, which each have a local maximum. This results in an advantageous manner that not only one, but several objects can be detected and registered in the environment map.

In a further embodiment, an object is assigned to a voxel or a voxel area only if one Occupancy probability is greater than a predetermined probability. An object in the environment map is thus detected only if the occupancy probability is greater than a predetermined occupancy probability. Accordingly, a measure of the occupancy probability, z. This results in an advantageous manner against a misdemeanor robust method, wherein a random and not based on real existing objects increase the occupancy probability is not necessarily a resulting assignment of an object to a Voxel or a voxel area of the environment map leads.

In a further embodiment, the device for image acquisition is a camera arranged on a motor vehicle. This results in an advantageous manner that a partially already existing anyway camera in the motor vehicle can also be used to generate three-dimensional environment information by means of the proposed method, wherein the camera for other methods, such. As a distance measurement or a lane keeping assistance system can be used. The camera can be a forward camera or a backward camera. In particular, a position and orientation, i. H. extrinsic and also intrinsic parameters known to the camera relative to the reference coordinate system of the motor vehicle. Next, a position and orientation of the motor vehicle with respect to the geofesten reference coordinate system, z. B. by a GPS sensor. In this way, an image coordinate system with respect to the geofesten reference system can then be referenced.

In a further embodiment, a scale of the environment map is determined as a function of a distance covered and a speed of the device for image acquisition.

If the device for image acquisition is a camera arranged on a motor vehicle, then the distance traveled and the speed can be determined as the distance traveled and the speed of the motor vehicle. The distance covered and the speed can be determined, for example, by means of displacement sensors and speed sensors which are generally already present in the motor vehicle. As a result, an actual size of an object detected and one or more voxels of the environment map can advantageously be determined.

Also, therefore, a scaling of the environment map in response to a change in orientation of the device for image capture done, which is also detected, for example, by means of suitable sensors. A speed of the image acquisition device may also be determined depending on the distance traveled and a time to travel the route.

In a preferred embodiment, a distance traveled by the image capture device is determined by an odometric method, for example, if the image capture device is fixedly connected to a motor vehicle. In this case, an odometric method refers to a method which determines a traveled path on the basis of data from a propulsion system of the device for image acquisition. For example, if the device for image acquisition firmly connected to a motor vehicle, then a number of Radumdrehungen between two times of image acquisition can be determined and determined assuming a known wheel diameter of the distance traveled. An orientation of the motor vehicle can be determined, for example, as a function of a steering angle of the motor vehicle. A speed of the image acquisition device can be determined as a function of a time interval between two times of the image acquisition and the distance traveled.

This results in an advantageous manner that for a determination of the scaling of the environment map and thus a determination of an actual size of an object necessary parameters can be determined in a simple manner. If the device for image acquisition is, in particular, a camera arranged on a motor vehicle, already existing sensor signals of the motor vehicle can be used for the determination in an advantageous manner.

Further proposed is a device for generating three-dimensional environment information, in particular of three-dimensional environment information of a motor vehicle. The device comprises at least one device for image acquisition, at least one evaluation device and at least one storage device. By means of the proposed device, one of the aforementioned methods can be carried out in an advantageous manner.

In a preferred embodiment, the device for image acquisition is a camera arranged on a motor vehicle.

The invention will be explained in more detail with reference to an embodiment. The figures show:

1 a schematic representation of a motor vehicle with a device for generating three-dimensional environment information,

2 a schematic representation of a generation of visual axes and

3 a schematic representation of an increase in occupancy probability.

Hereinafter, like reference numerals designate elements having the same or similar technical features.

In 1 is a schematic motor vehicle 1 shown. In a front area of the motor vehicle 1 is a camera 2 (Forward camera) whose detection range is shown schematically by dashed lines E1. The detection area here comprises an area lying in the forward direction of travel with respect to the motor vehicle 1 , Also shown is an evaluation device arranged in the motor vehicle 3 , a storage device 4 and a GPS sensor 5 , The camera 2 , the storage device 4 and the GPS sensor 5 are data technology with the evaluation device 3 connected. By means of the camera 2 is at a first time t1 (see 3 ) a first image B (see 2 ) and at a second time t2 (see 3 ) a second image can be generated. A position and orientation of the camera 2 is variable between the first time t1 and the second time t2, in particular by a forward or reverse drive of the motor vehicle 1 , By means of the evaluation device 3 In the first image B at least a first image feature is detectable. Analogously, at least one further image feature can be detected in the second image.

Next is by means of the evaluation 3 a first line of sight S1 (see 2 and 3 ) from the camera 2 determined to the first image feature, this being done in a geofesten reference coordinate system. The first line of sight S1 becomes a voxel-based environment map 6 registered (see 3 ), where values of voxels of the voxel-based environment map 6 in the storage device 4 are stored. Further, the occupancy probability of a voxel-representing counter variable is increased from those voxels touched or cut by the first line of sight S1. Analogously, at the second time t2 another line of sight 52 from the camera 2 determines to the at least one further image feature, wherein in 3 it is shown that the at least one further image feature is equal to the first image feature. Further, the further line of sight S2 in the three-dimensional environment map 6 registered, whereby counter variables of voxels of the environment map 6 which are touched or cut by the further line of sight S2. The determination of the lines of sight S1, S2 takes place by means of the evaluation device 3 , The environment map 6 , that is the voxels of the environment map 6 associated values of the counter variables are in the memory device 4 saved.

By means of the evaluation device 3 the local maxima of the occupancy probability can be determined after each time point t1, t2 of the image acquisition, with the voxels or voxel areas having a predetermined size each having a detected object O (see FIG 3 ), which have the local maxima. Here, the object O is assigned to a voxel or a voxel area only if an occupancy probability greater than a predetermined probability, the value of the counter variables is greater than a predetermined value.

In 2 the method according to the invention is shown schematically. Shown on the left is a first image B which, for example, by means of the in 1 illustrated camera 2 at a first time t1 (see 3 ) was generated. Shown in the first image B is a road 7 , a town sign 8th and a church 9 which has a characteristic church tower 10 having. The place sign 8th and the church tower 10 , in particular their geometric centers, can be detected in this case by means of methods for object detection in the image B known to the person skilled in the art, the image of the place sign 8th and the church tower 10 into the pixel-based image of the camera 2 generated detectable image features. By means of the evaluation device 4 (please refer 1 ) is a first line of sight S1, for example a line of sight from the camera 2 to the place sign 8th , in particular a line of sight from an origin of an image coordinate system of the camera 2 to a geometric center of the detected city sign 8th , determinable and in an environment map 6 , which is three-dimensional, registerable. Analog is another line of sight S11 of the camera 2 to the church tower 10 in the environment map 6 registrable. The entry takes place here in the reference coordinate system, wherein the detected image features and the origin of the image coordinate system of the camera 2 be transformed into the geofeste reference coordinate system.

In 3 is a schematic increase of a probability of occupancy in the three-dimensional environment map 6 shown. At a first time t1 is the z. In 1 illustrated vehicle 1 at a first position. This is done in one of the camera 2 generated first image B (see 2 ) an object O, for example, the in 2 illustrated sign 8th , detected and a corresponding first line of sight S1 in an environment map 6 entered. wherein counter variables, each representing an occupancy probability of a voxel, of those voxels of the voxel-based environment map 6 are raised or cut from the first line of sight S1. At a second time t2, the vehicle has 1 in one by means of an arrow 11 direction shown. Analogously, in a second image generated at the second time t2, a second line of sight S2 to the object O is determined, wherein the object O in the image generated at the second time t2 is only detected next to optionally further image features, but does not have to be identified as object O. Next, the second line of sight S2 is also in the environment map 6 entered. It can be seen that the first line of sight S1 and the second line of sight S2 are in a voxel, not shown, or an unillustrated voxel area of the environment map 6 whereby an occupancy probability, in particular the counter variable of this voxel or the counter variables of this voxel area, is increased. Consequently, this voxel or voxel region will have a higher occupancy probability than the other voxels touched or cut by the first line of sight S1 or the second line of sight S2. Consequently, by evaluating local maxima of the environment map 6 the object O in the environment map 6 be detected. It is assumed that the environment map 6 is georeferenced, that is, a position of voxels of the environment map with respect to the global geofeste reference system is known.

LIST OF REFERENCE NUMBERS

1

motor vehicle

2

camera

3

evaluation

4

memory device

5

GPS sensor

6

three-dimensional environment map

7

Street

8th

ortsschild

9

church

10

steeple

11

arrow

E1

lines

B

image

S1

first line of sight

S11

further line of sight

S2

second line of sight

t1

first time

t2

second time

O

object

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 2009016562 A1 [0004]

Claims (9)

Method for generating three-dimensional surroundings information, in particular three-dimensional surroundings information of a motor vehicle ( 2 ), wherein by means of a device for image acquisition at a first time (t1) a first image (B) and at least one further time at least one further image is generated, wherein the first time (t1) is different from the at least one further time, wherein the means for image acquisition changes a position and / or orientation between the first time (t1) and the at least one further time, wherein in the first image (B) at least a first image feature is detected, wherein in the at least one further image another image feature is detected, characterized in that a first line of sight (S1) is determined by the device for image acquisition to the at least one first image feature, wherein an occupancy probability of voxels of a voxel-based environment map ( 6 ), which is touched or cut by the first line of sight (S1), wherein a further line of sight (S2) is determined by the device for image acquisition to the at least one further image feature, wherein an occupancy probability of the voxels of the voxel-based environment map ( 6 ) which are touched or cut by the further line of sight (S2). Method according to Claim 1, characterized in that the occupancy probability is evaluated for each individual voxel or for at least one voxel area of a predetermined size, wherein at least one maximum occupancy probability is determined, a detected object (O) being associated with the voxel or the at least one voxel area which has the at least one maximum. A method according to claim 2, characterized in that local maxima of the occupancy probability are determined, wherein a detected object (O) is respectively assigned to the voxels or the voxel areas having the local maxima. Method according to claim 2 or 3, characterized in that a detected object (O) is assigned to a voxel or a voxel area only if an occupancy probability is greater than a predetermined probability. Method according to one of the preceding claims, characterized in that the device for image acquisition on a motor vehicle ( 1 ) arranged camera ( 2 ). Method according to one of the preceding claims, characterized in that a scaling of the environment map ( 6 ) is determined as a function of a distance traveled and a speed of the device for image acquisition. A method according to claim 6, characterized in that a zurückgelegter path of the device for image acquisition by means of an odometric method is determined. Device for generating three-dimensional surroundings information, in particular three-dimensional surroundings information of a motor vehicle ( 1 ), wherein the device has at least one device for image acquisition, at least one evaluation device ( 3 ) and at least one memory device ( 4 ), wherein by means of a device for image acquisition at a first time (t1) a first image (B) and at least one further time at least one further image can be generated, wherein a position and / or orientation of the device for image acquisition between the first time (t1) and the at least one further time is variable, wherein by means of the evaluation device ( 3 ) in the first image (B) at least one first image feature is detectable, wherein at least one further image feature is detectable in the at least one further image, characterized in that by means of the evaluation device ( 3 ) a first line of sight (S1) can be determined by the device for image acquisition to the first image feature, wherein an occupancy probability of voxels one in the storage device ( 4 ) stored environment map ( 6 ), which are touched or cut by the first line of sight (S1), wherein by means of the evaluation device ( 3 ), a further line of sight (S2) can be determined by the device for image acquisition to the at least one further image feature, wherein an occupancy probability of the voxels of the environment map stored in the memory device can be increased, which are touched or cut by the further line of sight (S2). Apparatus according to claim 8, characterized in that the means for image acquisition on a motor vehicle ( 1 ) arranged camera ( 2 ).

Images