EP2656613A1

EP2656613A1 - Apparatus and method for determining a disparity estimate

Info

Publication number: EP2656613A1
Application number: EP11728636.9A
Authority: EP
Inventors: Markus Schlosser; Jörn Jachalsky; Ralf Ostermann
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2010-12-22
Filing date: 2011-06-30
Publication date: 2013-10-30
Also published as: WO2012084277A1; US9591281B2; US20130272582A1

Abstract

The invention relates to a method and an apparatus for determining a disparity value for an object located in or to be placed into a stereoscopic image pair having an associated disparity map. First an area to be analyzed in one of the stereoscopic images is determined (1) Then a histogram is built (2) from disparity estimates of the associated disparity map that fall within the determined area. Subsequently a contiguous range of bins is searched (3) in the histogram that also contains a sufficient number (Nm±n) of pixels. Finally, a disparity estimate for the determined area is selected (4) from the contiguous range of histogram bins.

Description

Apparatus and Method for Determining a Disparity Estimate

FIELD OF THE INVENTION The invention relates to a method and an apparatus for

determining a disparity value for an object located in or to be placed into a stereoscopic image pair, and more specifically a stereoscopic image pair having an associated disparity map. BACKGROUND

In 3D-TV, 3D-video and 3D-cinema, the insertion of graphics elements needs to follow some rules concerning their depth positioning in order to avoid visual discomfort. Most

importantly, a superimposed element should not be

stereoscopically positioned behind an object in the video, as this would violate real world physical constraints. However, graphics elements may also not keep a too large safety margin in front of the closest object as a too strong "pop-out effect" may also lead to visual fatigue, caused by the accommodation- vergence conflict. Especially subtitles should be placed just in front of the closest object, as reading them is equivalent to a frequent refocusing between the video and the text. A human observer needs significantly more time to switch his or her focus of attention if the associated jump in depth is larger .

As a consequence, the stereoscopic positioning of text and graphics for 3D menus or 3D subtitles requires few, but highly reliable and accurate depth estimates to avoid these elements to be placed too far in front of the screen or, even worse, behind a video object. To compute depth information from a set of two (or more) images, stereo matching is applied to find point correspondences between the input images. The displacement between two corresponding points is referred to as disparity. The 3D structure of a scene can be reconstructed from these disparities through triangulation if the camera parameters are known.

Using calibration and rectification, it can be approximated reasonably well as if the images were captured with perfectly aligned, ideal pinhole cameras, which do not show any lens distortions. Although this allows the search to be restricted to horizontal lines, stereo matching still remains an ill- defined estimation problem for several reasons, like

occlusions, perspective deformations, specular reflections, depth discontinuities, as well as missing or quasi-periodic texture .

For the above reasons the performance of the stereo matching process inherently depends on the underlying image content. For some parts of an image it is inherently more difficult to determine accurate values for the disparity. This leads to varying levels of accuracy and reliability for the disparity estimates .

For this reason, in addition to the actual disparity value itself the reliability of a disparity estimate represents valuable information. A confidence map reflecting the estimated reliability is preferably provided along with the disparity map, wherein a confidence value is determined for every

disparity value. SUMMARY

It is an object of the invention to provide a solution for determining a highly reliable and accurate disparity value for an object located in or to be placed into a stereoscopic image pair .

According to one aspect of the invention, this object is achieved by a method for determining a disparity value for an object located in or to be placed into a stereoscopic image pair, the stereoscopic image pair having an associated

disparity map, which comprises the steps of:

- determining an area to be analyzed in one of the stereoscopic images ;

- building a histogram from disparity estimates of the

associated disparity map that fall within the determined area;

- searching a contiguous range of bins in the histogram that also contains a sufficient number of pixels; and

- selecting a disparity estimate for the determined area from the contiguous range of histogram bins.

According to a further aspect of the invention, an apparatus for determining a disparity value for an object located in or to be placed into a stereoscopic image pair, the stereoscopic image pair having an associated disparity map, is adapted to perform a method according to the invention. For this purpose the apparatus comprises a graphics analyzing block for

determining the area that is to be analyzed. A histogram building block is provided for building a histogram from all disparity estimates that fall within the area determined by the graphics analyzing block. A searching block is provided for searching for the closest or farthest contiguous range of bins in the histogram is searched that also contains a sufficient number of pixels in total. Finally, a selecting block is provided for selecting a robust estimate of the minimum

disparity. Of course, one or more of the different processing blocks may likewise be combined into a single multi-purpose processing block. If an application requires only few, but highly reliable and accurate depth/disparity estimates, the described invention robustly removes false estimates, which are inherent in stereo matching. A cluster of similar estimates in the depth/disparity map needs to be sufficiently large to be considered a robust detection of an object or even part of an object, which can be checked by histogram analysis. Advantageously, only those disparity estimates falling within the determined area are used to build the histogram for which an associated confidence measure exceeds a defined threshold. Alternatively, instead of accumulating the number of disparity estimates exceeding a defined threshold, the histogram is built by accumulating the confidence values of the disparity

estimates, or a value derived from the confidence values of the disparity estimates. The confidence measure is favorably derived from the similarity function employed during stereo matching or from information about at a match quality between a pixel or a group of pixels in the first stereo image and a corresponding pixel or a corresponding group of pixels in the second stereo image. Alternatively, a more elaborate confidence measure is used, as described, for example, in J. Jachalsky et al . : "Confidence evaluation for robust, fast-converging

disparity map refinement", IEEE International Conference on Multimedia and Expo (ICME), 2010, pp .1399-1404.

Based on this additional confidence evaluation, a cluster of similar estimates in the depth/disparity map needs to pass the confidence evaluation in addition to being sufficiently large to be considered a robust detection of an object or even part of an object. As these two strategies are independent of each other, by combining the two strategies false estimates are detected and removed even more robustly than with each one of these strategies alone.

In order to determine the full (reliable) disparity range of the stereoscopic images, favorably the area to be analyzed in one of the stereoscopic images is the complete stereoscopic image. For this purpose the contiguous range farthest from and closest to a viewer are searched in the histogram. The

determined disparity estimates may then be used to adapt the stereoscopic image pair to a display. For example, the

determined disparity range may be compared with some predefined limits to verify that the content will not cause visual fatigue. Additionally, it may also be used to guide

synthesizing appropriate new views in order to respect these limits. This maximizes the 3D effect while minimizing visual discomfort. Examples for an adaptation of the stereoscopic image pair to a display are found, for example, in L. Chauvier et al . : "Does size matter? The impact of screen size on

stereoscopic 3DTV", IBC 2010 Conference Paper.

Advantageously, the area to be analyzed in one of the

stereoscopic images is determined by the area of a graphics object to be placed into the stereoscopic image pair, e.g. a menu item or a subtitle. As the stereoscopic positioning of text and graphics for 3D menus or 3D subtitles requires highly reliable and accurate depth/disparity estimates, the invention is well suited for such applications. In order to avoid these elements to be placed too far in front of the screen or behind a video object, the contiguous range closest to a viewer within the area of the graphics object is searched in the histogram.

Favorably, the disparity estimate for the determined area is selected as the disparity for which the collected number of pixels surpasses the threshold or a fraction of the threshold, as a function of the absolute minimum, mean or maximum of the found contiguous range, or as a function of the maximum or median of a sub-histogram for this contiguous range. All these approaches allow to determine the disparity value with a reasonable computational effort.

Advantageously, the determined area to be analyzed is

subdivided into smaller areas. This allows to ensure at least a certain spatial proximity of the pixels associated with the found disparity interval.

For a better understanding the invention shall now be explained in more detail in the following description with reference to the figures. It is understood that the invention is not limited to this exemplary embodiment and that specified features can also expediently be combined and/or modified without departing from the scope of the present invention as defined in the appended claims. In the figures:

Fig. 1 shows a flowchart of a method according to the

invention, and

Fig. 2 depicts an apparatus adapted to perform the method according to the invention.

DETAILED DESCRIPTION OF PREFERED EMBODIMENTS

In Fig. 1 a simplified flowchart of a method according to the invention is depicted. In a first step 1 the area that is to be analyzed is determined. This may be the complete stereoscopic image or the area covered by a graphics element to be placed into the stereoscopic image or an area determined by some other means. For simplicity a bounding box of the graphics element may be used during this step instead of an accurate 2D projection of the graphics element. In a second step 2 a histogram is built from all disparity estimates that fall within the area determined in the first step and whose assigned confidence measure is above a defined threshold CM_m±_n.

Alternatively, instead of accumulating the number of disparity estimates exceeding a defined threshold CM_mi_nr the histogram can be built by accumulating the confidence values of the disparity estimates, or a value derived from the confidence values of the disparity estimates. As disparities are typically estimated with integer or fractional accuracy, the histogram may easily be built by allocating each potential disparity value its specific bin in the histogram. In a third step 3 the closest (to the viewer) contiguous range of histogram bins is searched that also contains a sufficient number N_mi_n of pixels in total. Advantageously, the number of disparity values in the

contiguous range of bins also has surpassed a minimum threshold in one histogram bin N_mi_{n b}±_n. A new range starts whenever the number of disparity values drops below a defined threshold d_m±_n for a certain disparity interval A_d. Finally, in a fourth step 4 a robust estimate of the minimum disparity is obtained. The estimate of the minimum disparity may be obtained in several ways, for example as a function of the disparity for which the collected number of pixels surpasses the threshold N_m±_n or a fraction of the threshold N_m±_nr as a function of the absolute minimum, mean or maximum of the found disparity interval, or as a function of the maximum or median of the sub-histogram for this disparity interval.

Apparently the above procedure may not only be used to robustly determine the closest object in the scene, but also the

farthest one. In this case the farthest (to the viewer)

contiguous range of histogram bins is searched that also contains a sufficient number N_mi_n of pixels in total. In addition, in the fourth step 4 a robust estimate of the maximum disparity is obtained.

Alternatively, it is likewise possible to determine disparity values for specified objects within the stereoscopic image. For this purpose for each object a histogram is built for those pixels that belong to the object. In this case contiguous ranges are searched in the histograms associated to the

objects. Several approaches for object segmentation are known from prior art. Either fully automatic or semi-automatic segmentation approaches can be used. The segmentation does not need to be perfect, due to the inherent robustness of the proposed histogram analysis. Due to the combination of the confidence evaluation with the histogram analysis, adjusting the parameters is uncritical. In an actual implementation, the parameter values CM_m±_n=0.5 , , and Delta_d=l pixel lead to good results. In Fig. 2 an apparatus 10 adapted to perform the method of Fig. 1 is shown schematically. The apparatus 10 comprises a graphics analyzing block 11 for determining the area that is to be analyzed. A histogram building block 12 builds a histogram from all disparity estimates that fall within the area

determined by the graphics analyzing block 11. A searching block 13 is provided for searching for the closest or farthest contiguous range in the histogram is searched that also

contains a sufficient number N_mi_n of pixels in total. Finally, a disparity estimation block 14 is provided for selecting a robust estimate of the minimum disparity. Of course, one or more of the different processing blocks 11, 12, 13, 14 may likewise be combined into a single multi-purpose processing block . In the above description, contiguousness is defined in terms of disparity and not real-world coordinates, so that dispersed outliers of similar depth could happen to pass the test. To ensure at least a certain spatial proximity of the pixels associated with the found disparity interval, the examined area may also be subdivided into smaller ranges.

Claims

1. A method for determining a disparity value for an object located in or to be placed into a stereoscopic image pair, the stereoscopic image pair having an associated disparity map, the method comprising the steps of:

- determining (1) an area to be analyzed in one of the stereoscopic images;

- building (2) a histogram from disparity estimates of the associated disparity map that fall within the determined area;

- searching (3) a contiguous range of histogram bins that also contains a sufficient number (N_m±_n) of pixels; and

- selecting (4) a disparity estimate for the determined area from the contiguous range of histogram bins.

2. The method according to claim 1, wherein only those

disparity estimates falling within the determined area are used to build (2) the histogram for which an associated confidence measure exceeds a defined threshold (CM_mi_n) .

3. The method according to claim 1, wherein the histogram is built by accumulating associated confidence measures of the disparity estimates, or a value derived from the associated confidence measures of the disparity estimates.

4. The method according to one of claims 1 to 3, wherein the determined disparity estimate is used to adapt the

stereoscopic image pair to a display.

5. The method according to one of claims 1 to 4, wherein the area to be analyzed in one of the stereoscopic images is the complete stereoscopic image.

6. The method according to one of claims 1 to 5, wherein the contiguous range farthest from a viewer is searched (3) in the histogram.

7. The method according to one of claims 1 to 5, wherein the contiguous range closest to a viewer is searched (3) in the histogram.

8. The method according to claim 7, wherein the area to be

analyzed in one of the stereoscopic images is determined (1) by the area of one or more graphics objects to be placed into the stereoscopic image pair.

9. The method according to claim 8, wherein the graphics object is a menu item, a score board, a logo or a subtitle.

10. The method according to one of claims 1 to 9, wherein the disparity estimate for the determined area is selected (4) as the disparity for which the collected number of pixels surpasses the threshold (N_m±_n) or a fraction of the threshold (N_m±_n) , as a function of the absolute minimum, mean or maximum of the found contiguous range, or as a function of the maximum or median of a sub-histogram for this contiguous range .

11. The method according to one of claims 1 to 10, further

comprising the step of subdividing the determined area to be analyzed .

12. An apparatus (10) for determining a disparity value for an object located in or to be placed into a stereoscopic image pair, the stereoscopic image pair having an associated disparity map, characterized in that the apparatus (10) is adapted to perform a method according to one of claims 1 11.