AT518438A1 - Method for self-localization of vehicles - Google Patents

Abstract

For self-localization of vehicles based on stereo vision, images of at least two cameras fixed statically on the vehicle are successively recorded and read out, whereby disparity or depth information is calculated from the recorded images and the self-motion is deduced from the distance difference of the same objects in different images. In accordance with the invention, the disparity or depth information is understood as images, and disparity clusters or deep clusters are extracted and these disparity clusters or deep clusters are found again in the subsequent images. It is expedient to calculate a ground plane as a disparity or depth value image, to raise it mathematically and to form the difference between the ground plane as a disparity or depth value image and the disparity or depth information before the disparity or depth information is compared become. This will hide the ground level where no useful information is available.

Description

The present invention relates to a method for self-localization of vehicles based on stereo vision, in which successively recorded and read images of at least two cameras fixed to the vehicle static, in which from the recorded images disparity or. Depth information are calculated and in which is derived from the distance difference of the same objects in different images on the proper motion.

Self-localization should be understood in this context to mean the calculation of the movement or position of a vehicle without the aid of external signals such as GPS over a time course.

Localization of autonomous systems based on internal signals is not a new idea. The basic approaches to this are known by the term odometry. Odometry calculates movements based on sensor values. In the simplest case, the wheel speeds are read out and integrated. Many of the basic algorithms of robotics are still not suitable for use in highly dynamic environments (dirt road, forest, ...). The problem is shown by drift over time or the complete failure of localization. The desire for a system stable over time is therefore very high for the application of autonomous systems.

The subject method is a camera-based approach using Stereovision - a technology for calculating depth information based on at least two camera images.

The previously known such methods record a camera image of at least two cameras and extract therefrom the image features, which are representative features such as, for example, SIFT features. Thereafter (e.g., one second or several seconds later), images are again taken and, in turn, image features extracted. Finally, an attempt is made to retrieve image features from the first images in the second images, after which the distance difference can be determined based on the calibration information.

Such systems are problematic, e.g. in the forest. Here, the conditions are constantly changing due to wind, leaf movement, etc. Therefore, no representative image features can be extracted.

It is an object of the present invention to provide a system of the type mentioned above, which does not have the mentioned disadvantages, so it works reliably even in dynamic conditions in non-structured environment.

This object is achieved by a method of the aforementioned type according to the invention that the disparity or depth information are interpreted as images that it disparity or depth clusters are extracted and that these disparity or depth clusters are found again in the subsequent images ,

Thus, according to the invention, not (successive) images are compared with each other, but disparity or depth information clusters (ie contiguous disparity contours in the disparity image in which the disparity value is within a predetermined value) are determined in the images. This has proven to be much more reliable than the direct comparison of the images, because the appearance of an object can change rapidly, but its distance remains constant for stationary objects except for their own motion. The self-localization is therefore based exclusively on the disparity and thus depth information of objects.

In order to be able to calculate the disparity clusters of objects at all, ground information should be removed from the disparity image. A description of the floor area is therefore necessary. By subtracting the ground level creates an "obstacle image". Positive pixel values after this subtraction mean elevations and negative values holes. As a result, the soil level containing hardly usable information is "hidden" as it were. Masking the original disparity image with the obstacle image creates an obstacle image with undistorted disparity values.

If the ground level is raised, the calculation may be limited to clearly outstanding obstacles.

A concrete embodiment of the present invention will be explained in more detail with reference to the accompanying drawings. 1 shows a possible arrangement of three cameras-two stereo cameras designed for different working spaces; Fig. 2 shows these three cameras with fictitious objects; and FIG. 3 shows the disparity of a ground plane.

The basic structure of the system can be seen in FIG. The camera head 10 shown in FIG. 1 consists of three cameras 11, 12 and 13. These form in conjunction with the left and middle camera 11 and 12 and left and right camera 11 and 13, two stereo camera systems. Since the accuracy of depth information decreases with distance, two systems designed for different distances can be used to minimize the error (11 and 12 for near objects, 11 and 13 for distant objects).

The sensors of the cameras 11, 12 and 13 all have the same number of pixels; In the following, m denotes the number of pixel columns and n the number of pixel rows in each of the sensors. The sensors therefore have a resolution of m × n pixels. The signals of the sensors are output via a line 14.

The camera head 10 must be calibrated. During calibration, on the one hand, internal camera parameters (eg distortion) and external parameters (transformation between cameras) are determined. There is software for the calibration.

In real use, each camera is read out at the same time and the images are rectified (equalized and transformed) with the calibration information. The same optical information can now be seen in all camera pictures in different places, in the same line of pictures. The position difference is called disparity. The depth in front of the camera in three-dimensional space is given by the relation T = B-f / D (1). T describes the depth, B the length of the baseline (distance of the cameras), f the focal length and D the disparity.

Since B and f are constant, the same information is present in the T values (depth information values) and in the D values (disparity values), except for one constant, which is reciprocal values. If one were to determine the disparity at every possible point in the picture and then use formula (1) to calculate the depth, one would obtain a depth image. However, since the depth information does not add value, it will be convenient to save this calculation and use only a disparity image.

In the disparity picture are as already mentioned

Object information mixed with soil information. In order to be able to separate them, the ground information - i. a disparity image of an ideal soil - to be calculated.

If a disparity image is represented as a grayscale image (bright dots indicate high disparity, dark dots low disparity), then the ground level narrows as the distance increases (as the disparity decreases) until it turns black in the horizon (no disparity at infinity). If the baseline of the camera head 10 is parallel to the ground plane, it can be assumed that the disparity value of an ideal ground plane in the disparity image is independent of the horizontal pixel position and depends only on the vertical pixel position. Therefore, only a sample of the ground level has to be taken. If the camera head 10 is aimed directly at a long straight road, then this sample may be a line exactly in the middle of the image from bottom to top.

In FIG. 3, the measured values D (disparity) as a function of the line Z (lines of the image) of the random sample are represented by a thick line, in which case the lines are away from the horizon (if the camera head is not inclined, that is to say from the Middle of the picture). In line 0 (horizon), therefore, the disparity is 0. It can be seen that the dependence is approximately linear due to the assumption of an ideal bottom surface, but the extracted sample does not behave optimally due to noise. Therefore, this noisy line must be linearly interpolated (thin, solid line). This is done by fitting the model d = k * z + 0 where z represents the line of the image, d the associated disparity value, o an offset, and k the slope of the disparity between two lines.

Since only objects that are clearly above the ground level should be detected, the ground level must be raised slightly further (dashed line) so that the noise is completely below this dashed line. Since the sample does not cover the entire image, the calculated linear model of the soil must interpolate and calculate a probable soil level. The disparity image of the raised ground plane should have exactly the same number of lines as the actual image material, hence the following

Calculations can be done easily. In order to obtain a disparity image of the calculated ground plane, the raised ground information is considered to be the vector dj of which each component corresponds to the disparity value corresponding to the vertical pixel position (j = 1 ... n). (For pixels corresponding to objects above the horizon one sets dj = 0.) Since, as mentioned above, the disparity value is independent of the horizontal pixel position, the disparity image of the ideal ground plane is given as matrix D with Djk = dj, j = l ... n , k = l ... m.

Thus, the process of calculating the ground information is as follows: • taking a disparity image when the camera head 10 is aimed at a plane as flat as possible (e.g., a long straight road); this picture has to show a lot of ground information. • Parameters of the software define a line for extraction. This line in the picture is only to show ground information. (Sample) • The disparity values on this line are stored in a vector with the image dimensions (pixel positions). This information describes the disparity values of the soil. • The described vector is interpolated. • The disparity values are offset (ground level is raised mathematically). • A soil disparity image is generated from the new soil information. • Save the information as a matrix (disparity value for each camera pixel), for example as a csv file.

The matrix can now be loaded in different applications. By forming the difference between a real disparity image and the calculated soil disparity image, a disparity image with exclusively relevant information (obstacles) is created. If we calculate with disparity values and subtract the soil disparity image from the (real) disparity image (and not vice versa)

Obstacles positive sign. In this case, set all negative values to zero. Similarly one could also evaluate holes in the soil.

In summary, obstacle detection is thus based on a real disparity picture and an ideal ground disparity picture. A disparity image calculated by the stereo engine is sent to the obstacle detection. The obstacle recognition now calculates an obstacle image by subtracting and masking the soil disparity image from the real disparity image.

Now that the obstacle image is present, disparity clusters can be extracted and defined as obstacles.

The actual values of the pixels in the obstacle image Η (H is an n x m matrix) depend on the distance between the camera head and the object. The closer the object is to the camera, the higher the pixel value. With this interpretation, the obstacle image can be interpreted as a three-dimensional representation of the environment.

The obstacle image is now divided into slices in the z-direction (the distance to the front). If the "slices" e.g. three disparity values are "thick", all pixel values are determined where Hjk are 0 to 2; 3 to 5 are; 6 to 8 are; 9 to 11 are; etc. In Fig. 2 this is illustrated. (Fig. 2 shows disparities - no distances)

The camera head 10 is directed to three objects 21, 22, 23.

In FIG. 2, "slices" 31 to 34 are shown, which correspond to disparity ranges 36 to 39 to 46 to 48. Generally, each slice 30 includes a range of three disparity values 3n through 3n + 2.

By analyzing only the disparity values of the slices in the obstacle image, images for the individual slices can be calculated. By dividing the disparity information into slices, contours of objects can be recognized in each slice. For this purpose algorithms are used to extract contours. Each extracted contour is called a disparity cluster in the following process. Each disparity cluster is now described by:

Clusten = {disparity i, max r offseti, widthi} (2)

Where "offset" is the x coordinate of the obstacle (measured from the center of the image) and "width" is the real, maximum object width in the extracted disparity cluster. Furthermore, the maximum disparity value is stored. This is a measure of the camera point next point.

Since the parameters of the camera head (baseline, focal length) are known, the disparity according to formula (1) can be converted into meters at any time.

In order to perform a reliable localization, this method can be extended by a feature extraction. In this context, features are understood to be diverse information from the depth clusters. The cluster will be extended with this information:

Clusteri '= {disparityi, max, offseti, widthi, mi} (3)

The feature Mi is a vector, and can represent any amount of information. Mi contains representative information for the cluster. This information can be extracted deep features or the matrix of the cluster itself.

Self-localization is based on the fact that similar disparity clusters can be found in consecutive disparity images. By comparing the movement of the clusters in the image or the pixel values and the associated three-dimensional interpretation, the real movement of the vehicle can be calculated. A prerequisite for this is a stationary environment.

In order to calculate the motion robustly, to ignore moving objects and to minimize data noise, motion models are used. These models can significantly predict movements of clusters and thus significantly improve the motion calculation.

In the simplest case: If the vehicle is last equipped with e.g. 20 km / h, it can be assumed that the vehicle will move at a very similar speed in the next few seconds, and on this basis and the last stereo image calculate where the disparity clusters will approximate at the next stereo image. This allows the disparity clusters of different stereo images to be associated with each other much more easily. Of course, there are also much better models here; Allow for accelerations in each direction.

The self-localization itself is based on the analysis of extracted disparity clusters. In a first disparity image, disparity clusters are extracted. Since the camera pixels and thus the disparity information rush, a simple triangulation is excluded. It would create a drift, which is not compensated without external signals or optical landmarks. Therefore, a statistical framework must be used that takes into account the noise of the sensors and may be easily expandable. For this an algorithm, called extended Kalman filter, can be used.

Claims (3)

Claims: 1. A method for self-localization of vehicles based on stereo vision, in which images of at least two statically mounted on the camera cameras are read and read, in which disparity or depth information is calculated from the recorded images and in which the distance difference of the same objects is closed in its own motion in various images, characterized in that the disparity and depth information are interpreted as images, that it disparity or depth clusters are extracted and that this disparity or. Depth clusters can be found again in the following pictures. 2. The method according to claim 1, characterized in that a ground plane is calculated as Disparitäts- or depth value image and the difference between the ground level as Disparitäts- or depth value image and the disparity or. Depth information is formed before the disparity or depth information is compared. A method according to claim 2, characterized in that the ground plane is raised mathematically before the difference is formed and only values with the sign corresponding to obstacles are considered.