DE102013017621A1 - Method for evaluating stereoscopically captured frames, involves generating stereoscopically captured frames for evaluation of three-dimensional reconstruction of scene from captured frames as disparity image - Google Patents

Abstract

The method involves generating stereoscopically captured frames (B1,B2) for evaluation of three-dimensional reconstruction of scene from the captured frames as disparity image (DB). The three-dimensional reconstruction of scene knowledge (V) is used as geometric knowledge in response to prior knowledge similar to the scene knowledge.

Description

The invention relates to a method for evaluating stereoscopically recorded individual images, wherein a three-dimensional reconstruction of a scene from the stereoscopically captured individual images is generated as a disparity image during the evaluation, wherein scene prior knowledge is used in the three-dimensional reconstruction.

Methods for the evaluation of stereoscopically recorded images are known from the prior art, in which a scene prior knowledge is used for the evaluation.

Such a method describes, for example Gallup, D. et al.: "Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions"; Int. Conference on Computer Vision and Pattern Recognition 07, 2007 Using prior knowledge from a so-called structure-from-motion step and prefetching the information contained in the prior knowledge for a few points of a pose estimation from a structure-from-motion process in a subsequent dense stereo analysis.

The invention is based on the object to provide a comparison with the prior art improved method for the evaluation of stereoscopic captured images.

The object is achieved by a method having the features specified in claim 1.

Advantageous embodiments of the invention are the subject of the dependent claims.

In a method for evaluating stereoscopically recorded individual images, a three-dimensional reconstruction of a scene from the stereoscopically captured individual images is generated as a disparity image during the evaluation, wherein scene prior knowledge is used in the three-dimensional reconstruction.

According to the invention, prior knowledge of the scene is used specifically for specific scenes, with reconstruction results that are similar to the scene prior knowledge being preferred in the three-dimensional reconstruction as a function of the scene knowledge.

The evaluation is distinguished by a high level of noise-freeness and temporal robustness, as a result of which the stereoscopically recorded and processed image data have fewer errors even in poor visibility conditions, for example in rain, snow, fog and / or darkness and thus an error rate of using this image data Driver assistance systems with the same availability is lowered.

Embodiments of the invention will be explained in more detail below with reference to a drawing.

Showing:

1 schematically a block diagram of a sequence of a method according to the invention.

In the only one 1 is a block diagram of a sequence of a possible embodiment of a method according to the invention for the evaluation of stereoscopically captured frames B1, B2 shown.

In the evaluation of the individual images B1, B2, a three-dimensional reconstruction of a scene from the stereoscopically recorded individual images B1, B2 is generated as a disparity image DB, for example on the basis of a disparity estimation process DSV. The disparity estimation method DSV is, for example, a known global or semi-global estimation algorithm, which is also known as a global matching algorithm or a semi-global matching algorithm, such as Hirschmüller, H .: "Accurate and efficient stereo processing by semi-global matching and mutual information"; In: Proceedings of Int. Conference on Computer Vision and Pattern Recognition 05, San Diego, CA, volume 2, pp. 807-814, June 2005 is known.

In the case of the three-dimensional reconstruction, additional pre-knowledge of the geometry V pixel-specific, ie pixel-specific, is used as the scene prior knowledge. In the case of the three-dimensional reconstruction depending on the scene knowledge V, those reconstruction results which are similar to the scene prior knowledge V are preferred.

In this case, the scene prior knowledge V is modeled and / or learned from image data, the data determined by means of the modeling and / or learning being used to incorporate the scene prior knowledge V in the three-dimensional reconstruction in a pixel-specific manner.

In the modeling, it is expected that the sky in an upper image portion and a lane on which a vehicle including an image capturing unit for stereoscopically capturing the frames B1, B2 are located in a lower image portion. In this case, a disparity with the value "0" is preferred for the sky, in the case of the roadway, a disparity expected on the basis of an installation position of a sensor of image acquisition on the vehicle.

In learning scene awareness V, many image data acquired by the image capture unit are accumulated, and a point of most frequent disparity per pixel or pixel and an associated standard deviation are calculated.

In the following, a possible embodiment is described, which can be used in all stereoscopic evaluations in which disparity hypotheses in a so-called cost cube C (u, v, d) with an image width u = 1, image height v = 1 and a maximum disparities d = 0 are calculated.

cost_new(u, v, d)

neuer Kosten-Kubus,

aehnlichkeits-cost(u, v, d)

Ähnlichkeitskosten,

vorwissen-cost(u, v, d)

Szenenvorwissen-Datenterm,

min

Minimum,

maxcost

maximale Kosten und

d_erwartet(u, v)

erwartete Disparität für den Bildpunkt (u, v)

ergibt.In modeling scene awareness V, it is first assumed that the scene above the horizon is best described by the sky. Thus, a disparity with the value "0" is preferred in this area. This is done, for example, by introducing a scene pre-knowledge data term pre-knowledge-cost (u, v, d) in the cost cube C (u, v, d) of the stereo analysis according to cost_new (u, v, d) = similarity - cost (u, v, d) + prior knowledge - cost (u, v, d), (1) wherein the scene prior knowledge data term pre-knowledge cost (u, v, d) for a pixel or pixels (u, v) according to previous knowledge - cost (u, v, d) = min (maxcost, (d - d_expected (u, v)) ² (2) With:

cost_new (u, v, d)

new cost cube,

similarity cost (u, v, d)

Similarity costs

foreknowledge-cost (u, v, d)

Scene knowledge-data term,

min

Minimum,

maxcost

maximum costs and

d_expected (u, v)

expected disparity for the pixel (u, v)

results.

This results in the lowest cost in equation (2) if the determined disparity d at an expected disparity d_expected (u, v) with the value "0" also has the value "0". The maximum cost maxcost is described by a function depending on an outlier probability.

Equations (1) and (2) also apply to the second assumption that a roadway is expected below the horizon. Here, the expected disparity d_expected (u, v) is calculated from the mounting position of the sensor of the image acquisition unit and the value of the expected disparity d_expected (u, v) changes from line to line in the disparity image DB.

If there is more information about the three-dimensional structure of the scene from another data source, such as a digital map, they can be used analogously in accordance with a possible further development.

For learning scene knowledge V, the expected disparity d_expected (u, v) and a variance sigma_d ² (u, v) are learned from many image data. To be independent of current installation angles of the To achieve sensors of the image acquisition unit, the image data is transformed to a vehicle longitudinal direction.

This results in the scene prior knowledge data term prior knowledge cost (u, v, d) according to

This results in the least cost in the expected disparity d_expected (u, v) that occurs most frequently in the learning process.

When the scene knowledge V is embedded in a semi-global estimation algorithm, the estimation algorithm requires data costs for each possible correspondence hypothesis, which results from the cost cube C (u, v, d) of the size image width u · image height v · number of disparities d. The data costs are calculated for the cost cube c (u, v, d), for example with Hamming distances of censored images, for example a census of 9x7, and are present in the data cube c (u, v, d).

Due to the embedding, the calculation of the data costs for all hypotheses in the cost cube c (u, v, d) results:

The value d_expected is the expected disparity d_expected (u, v) from the modeling and / or the learning process. MAXCOST is calculated from an outlier probability (parameter) and SCALEFAC is calculated from the similarity criterion used and is 4.5 for a 9x7 census as a similarity criterion.

An additional effort to a classical semi-global estimation algorithm results here only by increasing the data costs according to equation (4).

The application is particularly advantageously extendible to all stereo algorithms that generate a cost cube c (u, v, d), including local stereo algorithms.

LIST OF REFERENCE NUMBERS

• B1

  frame

  B2

  frame

  DB

  disparity

  DSV

  Disparitätsschätzungsverfahren

  V

  scene knowledge

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited non-patent literature

• Gallup, D. et al.: "Real-Time Plane-Sweeping Stereo with Multiple Sweeping Directions"; Int. Conference on Computer Vision and Pattern Recognition 07, 2007 [0003]
• Hirschmüller, H .: "Accurate and efficient stereo processing by semi-global matching and mutual information"; In: Proceedings of Int. Conference on Computer Vision and Pattern Recognition 05, San Diego, CA, volume 2, pp. 807-814, June 2005 [0014]

Claims (3)

Method for evaluating stereoscopically recorded individual images (B1, B2), wherein a three-dimensional reconstruction of a scene from the stereoscopically captured individual images (B1, B2) is generated as a disparity image (DB) during the evaluation, wherein scene prior knowledge (V) is used in the three-dimensional reconstruction is characterized in that the scene prior knowledge (V) geometric prior knowledge is used pixel-specific, in the three-dimensional reconstruction depending on the scene knowledge (V) such reconstruction results are preferred, which are similar to the scene knowledge (V). A method according to claim 1, characterized in that the scene prior knowledge (V) is modeled. A method according to claim 1 or 2, characterized in that the scene prior knowledge (V) is learned from image data.

Images