DE102007027958A1 - Method for optimizing a stereoscopic image - Google Patents

Abstract

The invention relates to a method for optimizing a stereoscopic image, wherein at least two partial images (1.1, 1.2) of an object (O) are recorded by different recording points and the partial images (1.1, 1.2) are converted such that corresponding partial pixels (TP1.1, TP1 .2) of the two partial images (1.1, 1.2) coincide with respect to a coordinate in a spatial point, and a disparity corresponding to a deviation between coordinate values of the corresponding partial pixels (T1.1, T1.2) is determined on the basis of which a disparity image (DB) with corresponding Pixels (BP) is generated. According to the invention, disparity values (d_mess) of adjacent pixels (BP) of the ascertained disparity image (DB1) are processed on the basis of a smoothing term (E _G ) and a smoothed disparity image (DB2) is generated.

Description

The The invention relates to a method for optimizing a stereoscopic Image.

Stereoscopy method to the space-faithful illustration are usually based on the Comparison of at least two corresponding fields (also stereoscopic fields or image pairs), which consists of at least two different angles, for example two different ones Camera positions, are recorded, with corresponding subpixels the fields are assigned to each other by suitable methods. The invention and the prior art will be described below of two, by means of two in terms of orientation and focal length coordinated cameras recorded fields (= standard stereo with stereoscopic images); however, the invention is also related to systems usefully applicable with more than 2 cameras (= multi-stereo with Multi stereo images).

The half-images recorded from at least two different viewing angles can serve, for example, by simple triangulation of the reconstruction of depth values (also spatial values) according to: Distance = normalized focal length × base width / disparity.

Under Depth values thereby become the depth of a pixel (= space point the corresponding subpixels), which is the distance of the pixel from the projection center of the corresponding camera describes. By reconstruction of depth values for all Pixels of the stereoscopic image formed from the two fields Image is obtained a stereoscopic image with so-called dense depth estimate and thus a three-dimensional Image.

The Disparity of the stereoscopic image usually becomes pixel exact determined by a deviation (= distance) between Coordinate values of corresponding subpixels of the fields is determined. It represents a deviation between coordinate values the X coordinate of the corresponding subpixels a so-called Transverse disparity and a deviation between coordinate values the z-coordinate of the corresponding subpixels a so-called Depth or space depth disparity. When rectifying For example, the two fields are in a common Image plane projected such that all distance vectors corresponding Subpixels run parallel to the X coordinate of the image plane, where the Y coordinate is identical.

Most methods for determining disparities, such as. B. Correlation-based techniques or global optimizing approaches, such as the so-called semi-global matching, determine the disparities pixel-exact. Especially with small disparities, this usually leads to intolerable quantization effects when determining the distance or depth, which influences the quality of the data. Therefore, it is often tried to estimate the disparity more accurately, ie subpixel-exact. For this purpose, one uses a so-called "goodness-of-fit function", which has been evaluated in the context of stereo analysis for various disparities and fits (also called fit) a suitable function, usually a parabola, to the optimum and the functional values of the adjacent Disparities. The location of the minimum of this function is then considered an improved estimate of the disparity. Such a function is for example in D. Scharstein, R. Szeliski: A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms, Proceedings of the IEEE Workshop on Stereo and Multi-Baseline Vision, Kauai, HI, 12/2001 , described. This operation is performed on all pixels for which a disparity was previously determined. Even if this avoids the extreme quantization effects, this approach is not satisfactory because the independent evaluation of the considered pixels results in a high noise.

In addition, approaches are known which consider the estimation of a dense disparity image as a variational task, wherein a global energy term consisting of similarity measure of the stereo images and smoothness measure is minimized, as for example X. Huang, E. Dubois: "Disparity estimation for the intermediate view interpolation of stereoscopic images", ICASSP 2005 , is known. These methods are very computationally intensive and prone to undesirable smoothing of depths with little difference in gray scale. Thus, object detection becomes more difficult with increasing distance or depth.

Of the The invention is therefore based on the object, an improved method to specify a stereoscopic image.

The The object is achieved by the features specified in claim 1. Advantageous developments The invention are the subject of the dependent claims.

At the inventive method for optimizing a stereoscopic image will be disparity values from neighboring ones Pixels of a disparity image, where the disparity values be determined in a conventional method, based a smoothing term and a smoothed one Disparity image generated. This is the quality pixel correspondence by taking into account smoothness conditions be, for. B. such that adjacent pixels usually smooth surfaces having substantially the same distance. In other words: The difference in depth between adjacent pixels is used as an evaluation criterion taken for the smoothness of the object, being adjacent Pixels in particular without regard to local pixel contrasts (= Edges and thus depth differences), d. H. about disparity edges away, be smoothed piece by piece. hereby are unwanted smearing effects on object boundaries (= Disparity jumps) avoided, so that a in the stereo analysis determined depth map or a depth image extreme is noise-free. This allows the resolution limit of Stereosystemen be increased.

The underlying disparity picture becomes conventional for example, based on at least two partial images of an object formed, taken from different pickup points and are converted such that corresponding partial pixels of the two partial images with respect to a coordinate (eg the X-coordinate for the so-called transverse disparity or the Z-coordinate for the so-called depth deviation) to coincide in a point of space and a deviation between Coordinate values of the corresponding subpixels corresponding Disparity is determined based on a disparity image is generated with corresponding pixels.

in the Detail is done optimizing the local disparity Energy minimization, in particular so-called global minimization, in the case of a space model through a data term, which is the assumption of Image consistency is implemented, and by the smoothing term, the the piecewise smoothing of adjacent pixels implements, is given.

Preferably the smoothing term is weighted. In particular, the Weighted disparity values of neighboring pixels. The smoothing term may be a smoothness weight or an edge weight in the energy minimization function, wherein the weighting factor is a constant, in particular an integer positive constant is. With rising, especially a high Weighting factor is the reproduction of smooth surfaces improved, with an average weighting factor the exact edge replica improved, which should be free of gaps. It can by varying the weighting factor a correspondingly optimal Weighting factor for both the replica of smooth Surfaces and edges are determined. Prefers becomes a smooth surface for the reproduction given optimal weighting factor, in this embodiment then the smoothing term representing object or disparity edges Pixels is disabled or reduced.

In considered a further embodiment the smoothing term depth differences and / or cross differences. In this case, by means of the predeterminable weighting factor, the smoothing for example, depending on the distance (= depth difference) set that objects at great distance or depth of space stronger be smoothed as objects near the camera. Alternatively, the weighting factor can also be dependent from the side distance of the object to the camera specified accordingly become (= cross differences).

In Another embodiment may additionally to the partial images of the object a contrast term, in particular a Gray value image are taken into account. This can also be done the smoothing at object boundaries is reduced accordingly or be disabled if necessary.

When a possible embodiment is the smoothed Disparity image based on one on a global energy minimization feature based on the estimation of Subpixel-exact disparities of the smoothing term and the estimation problem as local Energy minimization is formulated. This is the energy minimization function formed from a data term and the smoothing term, wherein as data term a similarity measure of similarity of disparity values of the pixels is determined. there due to the image consistency of the room their associated Space points (= pixels) approximately the same subpixels with the same values, in particular color values, brightness values.

In one possible embodiment, the similarity measure is formed on the basis of a weighted squared disparity difference, in particular based on a so-called SSD function (SSD = sum of squared differences or sum of squared differences). As a result, linear differences in brightness and contrast of adjacent pixels are taken into account. The advantage of the SSD function is simplicity and efficient implementation.

About that In addition, the energy minimization process is performed iteratively, whereby in each iteration step a so-called "least squares problem" is solved. This is a gradual optimization of stereoscopic image based on the disparity images.

embodiments The invention will be explained in more detail with reference to a drawing. Showing:

1 schematically two fields recorded by different camera positions with an object identified in the spatial depth,

2 schematically two depth maps with an associated depth distribution for the identified in the spatial depth object according to 1 , wherein the left depth map was generated by means of a conventional interpolation method and the right depth map was generated by means of the optimization method according to the invention, and

3 schematically an embodiment of an iterative optimization method of stereoscopic images based on disparity images in the form of a block diagram.

each other corresponding parts are in all figures with the same reference numerals Mistake.

1 shows schematically two, taken from different camera positions fields 1.1 . 1.2 with an object O identified in the spatial depth, z. B. a vertical tail section of a motor vehicle.

Based on the recorded fields 1.1 . 1.2 is by means of a data processing unit 2 (please refer 3 ), z. B. a control unit of a driver assistance system, a Disparitätsbild determined in a conventional manner. In the following example, the determined disparity image DB represents a depth image (also called a depth map) representing the distance of the object O, wherein a depth deviation (= depth disparity) of corresponding partial image points TP1.1 coinciding in a spatial or image point BP of the disparity image DB , TP1.2 of the fields 1.1 . 1.2 due to the independent processing of the corresponding subpixels TP1.1, TP1.2 of the pixel BP lead to a strong noise. Ie. the usual estimation of the disparity, in particular a subpixel-precise disparity, for example by local parabola fitting, ignores the fact that adjacent pixels BP of the disparity image DB usually belong to smooth surfaces with the same distance. Especially with small disparities results in this fact contradicting strong sound noise, which makes an object detection with increasing distance difficult. Such noise (also called "stereo cloud") in a disparity image DB1 determined in a conventional manner is described, for example, in US Pat 2 shown in the left image and often leads to erroneous object detections. In the example, the object O is positioned at a distance of 40 m. The estimated spatial pixels BP of the disparity image DB1 (= depth map) in this example rush in their distance or depth by +/- 1 m.

• – Objekte und somit deren Bilder oder Bildbereiche weisen näherungsweise glatte Oberflächen auf, so dass daraus resultierend
• – benachbarte Bildpunkte von Disparitätsbildern, welche dasselbe detektierte Objekt O repräsentieren, sehr ähnliche oder gleiche Disparitätswerte aufweisen sollten.

In order to reduce the noise in determined disparity images DB1, an optimization problem is formulated in the method according to the invention, which is formed from a data term E _D given by correlation and a smoothing term E _G. In this case, the following object properties or object parameters are taken into account by means of the new smoothing term E _G compared to conventional methods:

• - Objects and thus their images or image areas have approximately smooth surfaces, resulting in it
• Adjacent pixels of disparity images representing the same detected object O should have very similar or equal disparity values.

there the optimization problem is formulated in such a way that undesired Smear effects (= noise) at object boundaries (= jumps in dispersion) be reduced or avoided. This can be the optimization problem be solved for example by simple relaxation.

• Mit E_ges = Fehlerterm, E_D = Datenterm, E_G = Glättungsterm, d₀ = lokaler Disparitätswert.

In the optimization of the disparity image DB1, the local disparity d _{0 is given} as an energy minimization problem according to:

• With E _ges = error term, E _D = data term, E _G = smoothing term, d ₀ = local disparity value.

there is taken into account that a reasonably accurate estimate the disparity of the disparity images DB1 a certain smoothness in the estimated disparities or distances should have. That means the estimate the disparity at a pixel BP (x, y) (BP also position x) from the other pixels BP (x + 1, y + 1) and BP (x - 1, respectively) y - 1) depend on a local neighborhood by a factor x got to. It should in the optimization of the disparity picture DB1 be considered that at object boundaries Disparitätssprünge and thus disparate disparities and / or pixels BP depending on the local contrast different disparities occur.

Depending on the specification, the number of neighboring pixels BP to be taken into account is predetermined. For example, a 3 × 3 mask with adjacent numbering of the pixels BP is specified: 1 2 3 4 0 5 6 7 8th

mit N = 9 für das 3×3 System, d_i = Pixeldisparität (= Bildpunktdisparität), d = mittlerer Pixeldisparitätswert, d₀ = lokale Disparität, σ = geschätzte Disparitätsvarianz.The smoothness of the solution is defined for example by a disparity variance σ, other energy formulations are conceivable. Thus, the smoothing term E _G for determining the smoothness results:

with N = 9 for the 3 × 3 system, d _i = pixel disparity (= pixel disparity), d = average pixel parity value, d ₀ = local disparity, σ = estimated disparity variance.

In the so-called SPG estimation, a parabola is filtered to the values of a so-called SSD function {d_, d _o , d ₊ } (with SSD = sum of squared disparity differences), where d _{o is} the minimum of the error function. The resulting parabola opening gives a measure of the increase in error energy when leaving the optimum. Thus, the data term E _D can be formulated as follows: e D ~ a O (d O - d mess ) 2 [3] with a ₀ = constant, d ₀ = minimum of the error function, d _mess = local disparity.

The error term E _ges (= total energy) then results according to: e ges = a O (d O - d mess ) 2 + λ · σ 2 (d O ) [4] with λ = weighting factor for a distance-dependent weighting.

mit folgenden Variationen: ao → 0: do = d, [9] ao → ∞: do = dmess oder [10] λ → ∞: do = d. [11] An increase in the smoothing and thus the smoothness of the object surface by varying d _o is thus offset by an increase in the energy in the data term E _D. The optimum results to:

with the following variations: a O → 0: d O = d, [9] a O → ∞: d O = d mess or [10] λ → ∞: d O = d , [11]

Equation [8] also allows for smoothing and thus smoothness across disparity edges. It is desirable to have a formulation which excludes or greatly reduces the influence of the smoothing term E _G on disparity edges. For this purpose, the smoothing term E _G is modified, for example, as follows:

mit K = Verstärkungsfaktor (Verbesserung der Homogenität durch Iteration des Optimierungsverfahrens).The larger the disparity variance σ, the less effective a change in the local disparity d _o . Do you write in addition λ = λ ( d ) results for the error term E _tot :

with K = gain factor (improvement of homogeneity by iteration of the optimization method).

The equation [16] for σ _n = 1 and small variances of the equation [8], but reduces the influence of the mean disparity variance with increasing variance d ,

glättet Objekte O in großer Entfernung stärker, was in vielen Fällen erwünscht ist.If one chooses a constant weighting factor with λ (d) = constant, all disparities are treated the same, whereby by means of the predeterminable weighting factor λ, the smoothing can be set as a function of the distance. The choice of

Smooths out objects O at a great distance, which is desirable in many cases.

In 2 , right image is a smoothed disparity image DB2 (= depth map) generated by means of the described optimization method, in which the depth noise is largely reduced and thus the smooth object surface of the detected object O is clearly recognizable.

Since the local variance changes as the local disparity d _o varies, the method must be applied in the known iterative form, as described in US Pat 3 is shown by way of example on a block diagram.

In this case, based on measured disparities d _mess in a first iteration step n according to the method described above, a smoothed disparity image DB2 (n) with a homogeneity H is determined which corresponds to a function of the disparity d and the variance of the disparity d_var in this region, e.g. B. in the 3 × 3 Matrix, corresponds. In a further iteration stage n + 1, the next smoothed disparity image DB2 (n + 1) for the respectively considered area with, for example, 3 × 3 pixels BP is then determined on the basis of the mean disparity d_mean and the constant a ₀ and the homogeneity H, where for all Examples considered a convergence for a ₀ > 0 is assumed. The case a ₀ = 0 can occur only in areas with constant gray values and is excluded as "disparity d not measurable".

In addition, if, for example, not all adjacent disparities d of neighboring pixels BP are known, the variances are σ and N, respectively d to calculate accordingly.

1.1 1.2

fields

2

Data processing unit

BP

pixel

d_mess

disparity

DB1

disparity

DB2

geglättes disparity

H

homogeneity

O

object

TP1.1

Subpixel

TP1.2

Subpixel

QUOTES INCLUDE IN THE DESCRIPTION

This list The documents listed by the applicant have been automated generated and is solely for better information recorded by the reader. The list is not part of the German Patent or utility model application. The DPMA takes over no liability for any errors or omissions.

Cited non-patent literature

• D. Scharstein, R. Szeliski: "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms," Proceedings of the IEEE Workshop on Stereo and Multi-Baseline Vision, Kauai, HI, 12/2001 [0006]
• - X. Huang, E. Dubois: "Disparity estimation for the intermediate view of interpolation of stereoscopic images", ICASSP 2005 [0007]

Claims (12)

Method for optimizing a stereoscopic image, wherein at least two partial images ( 1.1 . 1.2 ) of an object (O) from different recording points and the partial images ( 1.1 . 1.2 ) are converted in such a way that corresponding partial image points (TP1.1, TP1.2) of the two partial images ( 1.1 . 1.2 ) coincide with respect to a coordinate in a spatial point, and a disparity corresponding to a deviation between coordinate values of the corresponding partial pixels (T1.1, T1.2) is determined on the basis of which a disparity image (DB) with corresponding pixels (BP) is generated, characterized that disparity values (d_mess) of adjacent pixels (BP) of the determined disparity image (DB1) are processed on the basis of a smoothing term (E _G ) and a smoothed disparity image (DB2) is generated. A method according to claim 1, characterized in that the smoothing term (E _G ) is weighted. A method according to claim 1 or 2, characterized in that the smoothing term (E _G ) takes into account differences in depth and / or cross differences. Method according to one of claims 1 to 3, characterized in that the smoothing term (E _G ) for object edges representing pixels (BP) is deactivated or reduced. Method according to one of claims 1 to 4, characterized in that in addition a contrast term, in particular a gray scale image in the determination of the smoothed Disparity image (DB2). Method according to one of Claims 1 to 5, characterized in that the smoothed disparity image (DB2) is determined on the basis of an optimization method based on a global energy minimization function (E _ges ). Method according to Claim 6, characterized in that the energy minimization function (E _ges ) is formed from a data term (E _D ) and the smoothing term (E _G ). A method according to claim 7, characterized in that a similarity measure of the similarity of disparity values (d_mess) of the pixels (BP) is determined as data term (E _D ). Method according to claim 8, characterized in that that the similarity measure is based on a weighted squared disparity difference is formed. Method according to one of claims 6 to 9, characterized in that the energy minimization method iteratively, wherein in each iteration step a least-square problem is solved. System for carrying out the method according to one of claims 1 to 10, characterized in that at least one data processing unit ( 2 ), which processes disparity values (d_mess) of adjacent pixels (BP) of at least one determined disparity image (DB1) on the basis of a smoothing term (E _G ) and generates a smoothed disparity image (DB2). Computer program product for processing disparity images for carrying out the method according to one of Claims 1 to 10 on a data processing unit ( 2 ).