DE102008015535B4 - Process for image processing of stereo images - Google Patents

Abstract

Method for image processing of stereo images (L, R), in which in a first step (S1) image data of an original image (O) are determined from two stereo images (L, R), on the basis of which a depth map (T1) of the original image (O) with a reduced resolution is determined and in a second step (S2) a depth map (T2) of a selectable section (AO) of the original image (O) is determined with a maximum resolution, characterized in that in the second step (S2) a maximum, the disparity (D) to be displayed in the depth map (T2) is specified.

Description

The invention relates to a method for image processing of stereo images according to the preamble of claim 1.

Depth calculation based on two images is a standard problem in image processing, for the solution of which various algorithms are known in the prior art.

These algorithms include, for example, global and semi-global stereo algorithms, such as those from “Hirschmüller, H.: Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information; in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 2, pp. 807-814”. However, these require a high level of computing effort, which prevents real-time implementation.

An obvious and widespread approach to reducing computing time is that from “Franke, U., Joss, A.: Real-time stereo vision for urban traffic scene understanding; in: Proceedings of the IEEE Intelligent Vehicles Symposium 2000, October 3-5, Dearborn, MI, pp. 273-278;” known reduction in image size of a depth image to be computed (hereinafter also called depth map). In the case of commonly used reductions by a factor of two in width and height, a number of the disparity values to be calculated is also halved, so that the computing time is reduced by a factor of eight. The disadvantage, however, is that this image size reduction leads to less precise disparity and/or depth values.

From the DE 10 310 849 A1 a method is known for photogrammetric determination of the distances or positions of objects relative to a reference point by evaluating a pair of stereo images of the objects, in which in a first step image data of an original image are determined from two stereo images, on the basis of which a depth map of the original image is determined with a reduced resolution becomes.

From "DAVIS, Robert L.: Screening of wide area search reconnaissance imagery. In Proceedings 2001 International Conference on Image Processing (Cat. No. 01CH37205), vol. 1, 2001, pp. 802-805” it is known, in the case of limited computing power or data transmission rate, to first provide image data in a lower resolution and then to use the data from a selectable section for the evaluation in a second step.

From the WO 97/21188 A1 a system and method for wide/narrow field of view detection is also known.

The invention is therefore based on the object of specifying an improved method for image processing of stereo images which, in particular, reduces the computing effort while the quality of the depth calculation remains the same.

The object is achieved according to the invention with a method which has the features specified in claim 1 .

Advantageous configurations of the invention are the subject matter of the dependent claims.

In the first step of the inventive method for image processing of stereo images to reduce the computing time of global stereo algorithms, image data of an original image are determined from two stereo images (= right and left input image), on the basis of which a depth map of the original image is determined with a reduced resolution. According to the invention, a depth map of a selectable section of the original image is determined with a maximum resolution in a second step.

According to the invention, a maximum disparity to be displayed in the depth map is specified in the second step.

According to a useful development of the invention, disparities with values greater than the maximum disparity are set to the value of the maximum disparity.

This results in the advantage that correct results can be achieved using the stereo algorithm.

Disparity is to be understood as meaning a horizontal distance between the same image elements on both partial images (= right and left input image).

In summary, the invention achieves a reduction in the computing time while increasing the quality or at least maintaining the same quality of the results of the stereo algorithm. This results in the possibility of executing a semi-global stereo algorithm in real time on FPGAs (=Field Programmable Gate Arrays) known in the prior art.

Exemplary embodiments of the invention are explained in more detail below with reference to drawings. However, the invention is not limited to the specifically cited exemplary embodiments, but advantageous reductions in the computing effort result from sub-combinations of the exemplary embodiments described.

• 1 schematisch ein rechtes und ein linkes Eingangsbild und in zwei Schritten berechnete Tiefenkarten,
• 2 schematisch mehrere Verteilungen von Disparitäten zur Ausführung einer Analyse eines Originalbildes in zwei Schritten,
• 3 schematisch mehrere Verteilungen von Disparitäten zur Ausführung einer Analyse eines Originalbildes in zwei Schritten mit veränderter dritter Verteilung, und
• 4 schematisch eine Zusammenfassung von jeweils vier Pfaden zu einem Scan, welcher ein Originalbild analysiert.

show:

• 1 schematically a right and a left input image and depth maps calculated in two steps,
• 2 schematically several distributions of disparities for performing an analysis of an original image in two steps,
• 3 schematically several distributions of disparities for carrying out an analysis of an original image in two steps with modified third distribution, and
• 4 a schematic of a combination of four paths to a scan, which analyzes an original image.

Corresponding parts are provided with the same reference symbols in all figures.

A method for image processing of stereo images to reduce computing time for globally optimizing stereo algorithms is described below. This class of methods also includes semi-global stereo algorithms (= semi-global matching).

1 shows a right input image R and a left input image L (=stereo images), for example of a street scene, which in particular represent a motif recorded from two different perspectives, for example by two cameras.

According to the invention, to reduce the computing time of global stereo algorithms, in a first step S1 image data of an original image O are determined from the right and left input images R, L, based on which a depth map T1 of the original image O is calculated with a reduced resolution, preferably half the resolution, with the entire original image O is covered with depth information.

In a second step S2, a further depth map T2 of a selectable section AO of the original image O is calculated with a maximum resolution, so that the section AO is covered with more precise depth information.

2 shows several distributions D1 to D3 of disparities D for carrying out the analysis of the original image O in two steps S1, S2. The first distribution D1 represents the disparities D with a value of up to "127" in the original image O.

In the analysis of the original image O performed in the first step S1 and the calculation of the depth map T1 with reduced, in particular half, resolution, a larger disparity range is automatically covered, but with reduced accuracy. Since the original image O according to the first distribution D1 has in particular disparities D with values up to “127”, examining the original image O with half the resolution results in a second distribution D2 with values from disparities D to “63”.

The results resulting from the first step S1, i. H. the depth information of the depth map T1 is used again in the second step S2 to analyze the freely selectable section AO of the original image O and to calculate the associated depth map T2 with full resolution, preferably original resolution.

In the second step S2, a maximum disparity Dmax to be displayed in the depth map T2 is additionally specified in the third distribution D3 of the disparities D. In the example shown, the section AO of the original image O is examined in the original resolution with a maximum disparity D _max of "63". The values of the disparities D of the first step S1 double due to the doubling of the resolution, so that a disparity D with the value “31” from the first step S1 corresponds to a disparity D with the value “62” in the second step S2.

As a result, areas of section AO which were recognized in first step S1 as objects with very high disparities D, in particular greater than “31”, are left out in second step S2 and mapped to the specified maximum disparity Dmax with the value “63”. This is achieved in particular by setting the similarity costs at the position of the maximum disparity Dmax to the value "zero", so that the globally optimizing stereo algorithm always chooses the maximum disparity Dmax as the solution.

In a post-processing of the second step S2 that is not shown in detail, all areas with the maximum disparity Dmax of “63” are then preferably replaced by the disparity D found from the first step S1.

mit: N = Anzahl der Operationen pro Pixel-Disparität-Kombination, D = Disparität.For a calculation of a depth map, not shown in detail, of the original image O with maximum resolution and a disparity Dmax of "128" (= values from "0" to "127"), the effort X ₁ is as follows: $${\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="DE102008015535B4\_0005.png"\; state="\{\{state\}\}">X</figure-callout>}}_1=N*Bil\mathrm{i.e}H\ddot{O}He*Bil\mathrm{i.e}b\mathrm{right}eite*D=N*Bil\mathrm{i.e}H\ddot{O}He*Bil\mathrm{i.e}b\mathrm{right}eite*128$$

where: N = number of operations per pixel-disparity combination, D = disparity.

so dass in vorteilhafter Weise gemäß: $$X_2=\frac{1}{4}{X}_1$$

der Aufwand X₂ dem um den Faktor „4“ reduzierten Aufwand X₁ bei gleicher oder verbesserter Auflösung der Tiefenmessung in großen Entfernungen entspricht.For the described calculation of the depth maps T1 and T2 of the original image O in two steps S1, S2 with a reduced disparity D of "64" (= values from "0" to "63"), the effort X ₂ is as follows: $${\mathrm{<figure-callout\; id="2"\; label="X"\; filenames="DE102008015535B4\_0005.png"\; state="\{\{state\}\}">X</figure-callout>}}_2=2*N*\left(\frac{Bil\mathrm{i.e}H\ddot{O}He}{2}\right)*\left(\frac{Bil\mathrm{i.e}b\mathrm{right}eite}{2}\right)*64$$

so that advantageously according to: $${\mathrm{<figure-callout\; id="2"\; label="X"\; filenames="DE102008015535B4\_0005.png"\; state="\{\{state\}\}">X</figure-callout>}}_2=\frac{1}{4}{\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="DE102008015535B4\_0005.png"\; state="\{\{state\}\}">X</figure-callout>}}_1$$

the effort X ₂ corresponds to the effort X ₁ reduced by a factor of "4" with the same or improved resolution of the depth measurement at large distances.

As an alternative to the calculation of the depth map T2 with disparities D up to a value of "63" in the third distribution D3, the second step S2 can also be carried out with disparities D in half disparity levels according to an in 3 illustrated third distribution D3 'are carried out.

For disparities D that are greater than the maximum disparity Dmax with the value "31", the roughly resolved disparities D, i. H. Disparities D with values greater than "16", used from the depth map T1 of the first step S1.

To generate the similarity costs, the gray values are additionally interpolated in order to generate D values for non-integer disparities. The disparities D with values from “31” to “63” are measured less precisely than in the first step S1, while the disparities D with values from “0” to “31” are measured more precisely.

As a result, it is advantageously possible to achieve an accuracy improved by a factor of “2” for the smallest disparities D with values of up to “31”.

From the prior art [“Hirschmüller, H.: Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information; in: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 2, pp. 807-814"] is a semi-global stereo algorithm (= Semi-global Mat ching) that reduces the global two-dimensional optimization problem of the stereo calculation to an optimization approximated by several one-dimensional paths A1 to A4, B1 to B4 (= directions). In this case, external read and write memory accesses are necessary in an original algorithm for each path A1 to A4, B1 to B4 in order to accumulate the path costs. Eight paths A1 to A4, B1 to B4 are a preferred choice as a compromise between low computational effort and high accuracy.

4 shows a combination of four paths A1 to A4 to a scan A and four paths B1 to B4 to a scan B, so that the four paths A1 to A4 in the scan A and the four paths B1 to B4 in the scan B are each processed simultaneously will. This results in a significant acceleration of the semi-global stereo algorithm.

The analysis (=scanning) of the original image O to acquire depth information and to calculate the depth maps T1, T2 is possible since only accumulated path costs for one line per path A1 to A4, B1 to B4 and all possible disparities D are necessary for the calculation . This means that all calculations for the four paths A1 to A4 can be processed internally in a computing unit RA or for the four paths B1 to B4 in a computing unit RB without accessing an external memory (not shown). The original image O is analyzed in particular from top left to bottom right on the basis of scan A and from bottom right to top left on the basis of scan B.

The simultaneous processing and internal buffering of several paths A1 to A4, B1 to B4 or directions, with only one line of buffering per path A1 to A4, B1 to B4 being necessary, reduces the required external memory bandwidth many times over and an integration of the semi-global stereo algorithm into an FPGA (=Field Programmable Gate Array) can be implemented in such a way that the stereo algorithm can be executed on it in real time.

Reference List

A

scan

oh

cutout

A1 to A4

path

B

scan

B1 to B4

path

D

disparity

Dmax

Maximum disparity

D1

First distribution

D2

Second distribution

D3

Third distribution

D3'

Third distribution

L

Left input image/stereo image

O

original image

R

Right input image/stereo image

RA

unit of account

RB

unit of account

S1

First step

S2

Second step

T1

depth map

T2

depth map

X1

expenditure

X2

expenditure

Claims (2)

Method for image processing of stereo images (L, R), in which in a first step (S1) image data of an original image (O) are determined from two stereo images (L, R), on the basis of which a depth map (T1) of the original image (O) with a reduced resolution is determined and in a second step (S2) a depth map (T2) of a selectable section (AO) of the original image (O) is determined with a maximum resolution, characterized in that in the second step (S2) a maximum, the disparity (D) to be displayed in the depth map (T2) is specified. procedure after claim 1 , characterized in that in the second step (S2) disparities (D) with greater values than the maximum disparity (D _max ) are set to the value of the maximum disparity (D _max ).

Images