EP2761876A1

EP2761876A1 - Method and device for filtering a disparity map

Info

Publication number: EP2761876A1
Application number: EP12775782.1A
Authority: EP
Inventors: Cedric Thebault; Philippe Robert; Sylvain Thiebaud
Original assignee: Thomson Licensing SAS
Current assignee: InterDigital CE Patent Holdings SAS
Priority date: 2011-09-29
Filing date: 2012-09-27
Publication date: 2014-08-06
Also published as: WO2013045853A1; US20140226899A1; JP2014534665A; US9299154B2; CN103828355B; KR20140069266A; BR112014007263A2; CN103828355A

Abstract

The invention relates to a method of filtering a map of disparity associated with one of the views of a stereoscopic image comprising the steps of identifying the pixels (x_n) with which is associated an erroneous disparity value (d(x_n)) by comparison with the disparity values of the neighbouring pixels (x_n-1,x_n+1) and of correcting the disparity values of the identified pixels.

Description

Method and device for filtering a disparity map

The present invention relates to a stereoscopic image and more particularly to a method and a device for filtering the disparity map associated with such an image.

3D view synthesis or 3D view interpolation is to create a 3D rendering other than that represented by the source stereoscopic image. The 3D rendering then adjusts to the chosen application or the user. The views needed to address multi-view screens are mostly recreated from a single stereoscopic image.

Indeed, this synthesis of view is based on the source stereoscopic image and on a map of disparity. The disparity map is generally associated with a right or left source image in the case of a stereoscopic image and provides for each pixel a datum representing the distance (in number of pixels) between the position of this image element in the right image and in the left image.

This view synthesis can also be based on a depth map because it is possible to return to a depth value from the disparity value, or vice versa, by introducing certain data, such as the size of the screen or the distance between the observer and the screen using an algorithm known to those skilled in the art.

To create a new interpolated view from a right or left source image and the associated disparity map, this disparity map is projected at the level of the view you want to generate. After this projection, we therefore have a modified disparity map corresponding to the desired view, each pixel of this disparity map modified indicates a disparity value which will then be used to adapt the information corresponding to the right or left image to regenerate.

The disparity vector of each pixel to be interpolated must point to the same points of objects in the right and left images.

However, management of the obscuration zones is necessary because there are pixels in the image to interpolate that support points of visible objects in one image and absent in the other. Indeed these objects that are in the field of a first sight, are either outside the field of a scene from a second view, or obscured by a foreground object. These particular pixels must be interpolated only from the image that contains the corresponding point. It is therefore necessary to identify these particular pixels and to assign them a disparity.

Document US2010 / 0215251 describes a method and a device for processing a depth map. The method consists in obtaining a depth map based on a compressed depth map including the depth information of a captured scene. The scene has an object. Occlusion information including information occluded by the object in the depth map is provided. At least a portion of the occlusion information is then used to reduce the distortions of the artifacts in the depth map.

This document therefore relates to an information processing method for the correction of a depth map related to an area obscured by an object.

Our invention aims to overcome a different problem which concerns so-called transition pixels located on the edges of objects. If, for example, a scene has two flat objects located at fixed and distinct depths, the disparity map associated with the elements of the right view or the left view should contain only two values, the value of the disparity of the first object. and that distinct from the disparity of the second object since there are only two possibilities in the scene with two different disparities.

The disparity map containing only two disparity values, it is in this case easy to project the disparity map on an intermediate view between the left view and the right view and thus to interpolate one of the two images.

But this is not the case because in the so-called transition zones, corresponding to the zones bordering the two objects, the disparity value associated with the pixels, referred to as transitions, results from a mixture between the disparity value of the pixel of the first object and that of the second object. Unfortunately, it is impossible to correctly define a disparity value for such pixels.

We will therefore end up with an intermediate disparity value between the disparity values of one object and the other object. Unfortunately this disparity value is not correct because it does not point to similar pixels in both images.

During the projection of the disparity map, such values will be found on the one hand discordant, and on the other hand do not point to the same objects in the two images which can cause visible artifacts on the interpolated views.

This problem of erroneous or outlier disparity value is present in the disparity maps obtained by a disparity map estimator and in disparity maps of the CGI contents.

A possible solution to the above-mentioned problem would be to use unfiltered disparity maps (that is, crenellated at the contours of the objects). Unfortunately if it is possible to consider this kind of map for CGI content, disparity estimators will most often naturally generate intermediate values between different objects. The present invention provides a method for overcoming the disadvantages mentioned.

The invention consists of a filtering method of a disparity map associated with one of the views of a stereoscopic image comprising at least one object partially covering an area obscured by this object. The method comprises the steps of determining at least one transition zone around the object, identifying the pixels of this transition zone whose disparity value is considered as erroneous compared with the disparity values of the neighboring pixels.

corresponding to the object or to the obscured zone and the correction of the disparity values of the pixels identified by allocation of a new disparity value

dependent on the object or the obscured area according to criteria relating to the membership of the pixel to the object or the hidden area. This method therefore allows intelligent filtering that only filters out doubtful disparity values and thus preserves the majority of the disparity map.

Preferably, the step of identifying pixels having an erroneous disparity value consists in identifying as erroneous any pixel whose disparity value is not between the disparity values of the neighboring pixels (x _n- i, x _n + i) ■

According to a variant of the invention, the step of identifying pixels having an erroneous disparity value consists in identifying as erroneous any pixel whose disparity value is different from the disparity values of the neighboring pixels (x _n- i, x _n + i) of a determined threshold value.

Preferably, the step of correcting the erroneous disparity values of the pixels consists in replacing the disparity value of the pixels identified by a value calculated as a function of the disparities of the neighboring pixels.

According to a variant of the invention, the step of correcting the erroneous pixel disparity values consists in eliminating the disparity value of the identified pixels.

Preferably, the step of correcting the disparity values of the pixels consists in determining which disparity value of the neighboring pixels (χ _η- ι, χ _η + ι), the disparity value of the identified pixel is the closest, then to replace the pixel disparity value identified by a replacement value corresponding to the value of one of the neighboring pixels (x i _n, x _n + i), the disparity value is closest.

Preferably, the method comprises the additional step, after the

determining the pixel of greatest proximity among the two neighboring pixels (x _n- ι, χ _η + ι), to determine the disparity value difference between that of the neighboring pixel and that of the next neighboring pixel (x _n-2 , Xn + ₂ ) and to determine the replacement value as a function of this difference.

The invention also consists in a device for generating three-dimensional image display signals comprising at least one object partially covering a zone obscured by this object comprising a device for transmitting a first flow of data corresponding to one of the right or left images to be transmitted, and a second data stream corresponding to the disparity map associated with one of the right or left images to be transmitted and a device for receiving data streams for displaying three-dimensional images. It comprises a device for filtering the disparity map comprising means for determining at least one transition zone around the object, means for identifying the pixels (x _n ) of this transition zone whose value of disparity (d (x _n )) is considered erroneous by comparison with disparity values of neighboring pixels (x _n- i, x _n + i)

corresponding to the object or to the hidden area and means for correcting erroneous disparity values of the identified pixels.

The features and advantages of the invention mentioned above, as well as others, will appear more clearly on reading the following description, taken in conjunction with the accompanying drawings, in which:

FIG. 1 represents a system for delivering 3D content and containing a device for filtering the elements of a disparity card according to the invention, either at the level of the signal generation device or at the level of the receiver.

FIG. 2 represents a block diagram explaining a filtering method according to the invention.

Figure 3 shows the disparity values of a disparity map of a system without a filtering device.

FIG. 4 represents the disparity values of a disparity card of a system with filtering device according to the invention.

The system as represented by FIG. 1 is a device for generating display signals for at least one three-dimensional image (3D). It comprises in the transmission device a generation means 2 of two data streams representing the pixels, or picture elements (in English "picture elements" or "pixels") of the images. left and right respectively, a means 3 for generating a data stream corresponding to the difference information of the pixels of the right or left image represented by a disparity map for each image, a selection means 4 of one of the streams of data of the right or left image to be transmitted, a transmission means 5 of one of the data streams corresponding to the selected right or left image and the data flow corresponding to the disparity map. It also comprises and according to the invention a filtering device 6 disparity values of the various elements of the disparity map for the detection of erroneous values and the replacement of these values by other disparity values depending on the disparities of neighboring elements of the disparity map associated with the means of generating the data of the disparity map.

A data receiving device receives the two data streams corresponding to one of the right or left images and the disparity map respectively and provides an interface 7 with the 3D image display signals to a screen or display device 8 According to a variant of the invention, the filtering device 6 of the disparity card is situated at the level of the signal receiver device and not at the level of the transmission device.

The method according to the invention consists in filtering the disparity map 6 so as to determine, eliminate or replace the "dubious" disparity values. The term "dubious" disparity value is defined as the pixel disparity values of the transition zones that do not correspond to one of the disparity values of the neighboring pixels.

This method allows intelligent filtering that only filters out questionable disparity values and thus preserves the majority of the disparity map.

The disparity map represents sampled information. We must therefore solve the sampling problems. This is why the method according to the invention consists in filtering the disparity map to improve the contours of objects that suffer from this lack of sampling. This filtering method must first correctly identify the erroneous disparity values, and secondly correct these values.

The erroneous disparity values are intermediate values between actual disparity values in that these disparity values correspond to depths where an object is located. But the disparity values can be locally increasing or decreasing in the presence of a surface not parallel to the plane of the screen. The definition of the detection algorithm must take into account these different cases.

It is also preferable to have a solution whose results are continuous to avoid artefacts due to neighboring and similar pixels and having different corrections, or to an object edge corrected differently from one frame to another. For this reason it is therefore advisable to replace disparity values rather than delete them. However, it is possible to delete these values if artifacts are to be deleted.

The method proposed here is associated with a study of pixels belonging to a horizontal line along the x-axis, but it would be possible to use vertical information as well.

The disparity values d (x _n ) of pixels x _n are determined as erroneous if these values differ from the disparity values d (x _n- i) and d (x _n + i) of the two neighboring pixels horizontally direct x _n- i and x _{n +} i of a value greater than the difference | d (x _n + 2) -d (x _n + i) | between this last value d (x _n + i) and that d (x _n + ₂ ) of its other neighboring pixel horizontally (x _{n +} 2) or of a value greater than the difference | d (x _n- 2) -d (x _n- i) | between this last value d (x _n- i) and that d (x _n- 2) of its other neighboring pixel horizontally (x _n- 2).

According to a variant of the invention, other criteria can be used to define what will be considered as erroneous disparity value. One method consists in comparing the disparity value of a pixel with respect to a determined threshold value as a function of the disparities of the neighboring pixels. Threshold values can thus be added. For example, it is possible to determine as erroneous pixels whose disparity values (expressed in pixels) different from more than 2 pixels with its two neighbors.

In the case where these erroneous values should be removed, it is preferable to use a less strict criterion in order to remove as little as possible.

In a method according to the invention, the determination of the erroneous disparity values is therefore done by comparing these values d (x _n ) with the disparity values d (x _n- 2), d (x _n- i), d ( x _n + i) or d (x _n- i), d (Xn + i), d (x _n + 2) of four other neighboring pixels x _n- 2 x _n -i n + ix _n + 2 ■

A mimic of detection and correction of pixels having erroneous disparity values is represented in FIG.

The first step 100 corresponds to identifying the pixels N pixels having erroneous disparity values by comparison with the disparity values of the neighboring pixels N-1 and N + 1:

Steps 101 and 102 then make it possible to determine whether the disparity d (x _n ) of the pixel x _n lies between those d (x _n- i) and d (x _n + 1) of the pixels x _n- i and x _{n +} i and, in step 101 and step 102, comparing the disparities of pixels x _n- i and x _{n +} i _: d (x _n- i) <d (Xn) <d (x _{n +} i) or (x _n- i)> d (Xn)> d (x _{n +} i)

Step 103 corresponds to the case where the value of the disparity is not between those of the neighboring pixels. A correction of this value is necessary. This value can be replaced by the nearest disparity value or deleted.

Then in steps 104 and 105, the disparity gap between the pixels x _n and x _n- i is compared with that between x _n and x _{n +} i. These steps correspond to a comparison study determining if the disparity value of x _n is closer to that of x _{n +} i or x _n- i. In the case of a pixel located on the border between two objects, this amounts to determining which object this pixel is closest to:

| d (x _n- i) -d (xn) | <| d (x _{n +} i) -d (x _n ) |,

Step 106 makes it possible to take into account a variable disparity (increasing or

decreasing) for the object corresponding to the pixels x _n- i and x _n . ₂ . This step therefore makes it possible to determine a replacement disparity value of (x _n ) corresponding to the maximum value of the sum between the disparity d (x _n- i) of the pixel x _n- i and the difference between the disparities d (x _n- 2) and d (x _n- i) pixesl x _n- 2 and x _n- i.

Similarly in steps 107, 108 and 109, the disparity of replacement values (x _n) are determined taking into account a variable disparity for objects corresponding to the adjacent pixels _n x 2 and x _n i or x _n + i and x _n + 2- Maximum values are determined if d (x _n ) is closer to the larger value of the two disparity values. And minimal values are determined if d (x _n ) is closer to the smaller value of the two disparity values.

During step 1 10, if the erroneous disparity value of the pixel is greater than this maximum value determined in step 106, the erroneous disparity value of the pixel x _n is replaced by this determined maximum value: x _n ) = max value.

During this step 1 10, if the erroneous disparity value of the pixel is not greater than this maximum value determined in step 106, the erroneous disparity value of the pixel x _n is replaced by the disparity value of the pixel. x _n-i: d '(x _n) = d _(n x i)

Similarly, during step 1 12,1a erroneous disparity value of the pixel x _n is replaced by the minimum value determined during the preceding step: "d (x _n ) = min value" or by the disparity value of the next pixel x _{n +} i: "d (x _n ) = d (x _{n +} i)".

Similarly, during steps 1 14 and 1 16, the erroneous disparity value of the pixel x _n is replaced by the minimum or maximum value determined during the preceding step or by the disparity value of the next pixel x _{n +} 1 or preceding x _n i. According to a variant of the invention, the process described by this mimic can be refined by taking into account a large change of disparity or particular cases.

The last steps 1 10 - 1 16 therefore preferably assign a new value to the disparity value according to the preceding criteria. It is alternatively possible to replace this step with a deletion of the disparity value.

In all untreated cases, the disparity value is considered correct and not replaced.

The filtering method proposed by the invention is symmetrical and therefore treats the right and left edges of the objects in the same way, although the problems related to these edges are different.

Indeed, the examples concern the interpolation of the right edge of objects in the foreground from the disparity map associated with the left view or the interpolation of the left edge of the objects in the foreground from the map of disparity associated with the right view because artifact problems are the most important.

The other two configurations, the interpolation of the left edge of the objects in the foreground from the disparity map associated with the left view, or the interpolation of the right edge of the objects in the foreground from the map of disparity associated with the right view, pose much less problems insofar as the projections of erroneous values are most often obscured by the object in the foreground. However pre-filtering is interesting in that we can in some cases recover part of the edge of the object in the foreground.

According to a variant of the invention, the method provides processing on the right edge of the objects in the foreground for the interpolation from the disparity map associated with the left view. Similarly and according to another variant of the invention, the method provides processing on the left edge of the objects in the foreground for the interpolation from the disparity map associated with the right view. This dissociation taking place in the first step (d (xn-1) <d (xn) <d (xn + 1) or d (xn-1)> d (xn)> d (xn + 1)), It would suffice to retain only one of these conditions to pre-filter only one side.

FIG. 3 represents the disparity values of a disparity map of an image comprising two elements of disparities different from a system without a filtering device. These two values are represented by a light gray and a dark gray

respectively. At the transition between the two elements, there are different disparity values represented by gray shades intermediate between light gray and dark gray

FIG. 4 represents the disparity values of a disparity card of a system with filtering device according to the invention. At the transition level, the erroneous disparity values have been corrected and replaced by one of the two existing values according to the method according to the invention. After filtering, the disparity map has only two values.

Claims

claims

A method of filtering a disparity map associated with one of the views of a stereoscopic image comprising at least one object partially covering an area obscured by this object and comprising the steps of:

determining at least one transition zone around the object,

identification of the pixels (x _n ) of this transition zone whose disparity value (d (x _n )) is considered to be erroneous by comparison with the disparity values of the neighboring pixels (x _n- i, x _n + i)

corresponding to either the object or the hidden area;

correction of the disparity values of the pixels identified by allocation of a new disparity value depending on the object or the obscured zone according to criteria relating to the membership of the pixel in the object or in the obscured zone.

2. Method according to claim 1 characterized in that the step of identifying the pixels whose disparity value is considered as erroneous consists in identifying as erroneous any pixel whose disparity value is not equal to one of the disparity values of the neighboring pixels (x _n- i, x _n + i).

3. Method according to claim 1 characterized in that the step of identifying the pixels whose disparity value is considered as erroneous consists in identifying as erroneous any pixel whose disparity value is different from the disparity values of the neighboring pixels ( x _n- i, x _n + i) of a determined threshold value.

4. Method according to claim 1 characterized in that the step of correcting the erroneous pixel disparity values consists in replacing the disparity value of the pixels identified by a value calculated as a function of the disparities of the neighboring pixels.

5. Method according to claim 1 characterized in that the step of correcting the erroneous pixel disparity values consists in eliminating the disparity value of the identified pixels.

6. Method according to claim 4, characterized in that the step of correcting the disparity values of the pixels consists in determining which disparity value of the neighboring pixels (x _n -i, x _n + i), the disparity value of the identified pixel is nearest and to replace the disparity value of the pixel identified by a replacement value corresponding to the disparity value of one of the neighboring pixels (x _n- i, x _n + i) whose disparity value is the closest.

7. Method according to claim 6, characterized in that the method comprises the additional step, after determining the pixel of greatest proximity among the two neighboring pixels (x _n- i, x _n + i), to determine the difference values of disparity between that of the neighboring pixel and that of the next neighboring pixel (x _n- 2, x _n + 2) and to determine the replacement value taking into account this difference.

8. Device for generating three-dimensional image display signals

comprising at least one object partially covering an area obscured by this object comprising a device for transmitting a first data stream corresponding to one of the right or left images to be transmitted, and a second data stream corresponding to the disparity map associated with one of the right or left images to be transmitted and a device for receiving data streams for displaying three-dimensional images, characterized in that it comprises a filtering device of the disparity map comprising

means for determining at least one transition zone around the object;

means for identifying the pixels (x _n ) of this transition zone whose disparity value (d (x _n )) is considered to be erroneous by comparison with the disparity values of the neighboring pixels (x _n- i, x _n + i) corresponding to either the object or the obscured area and; means for correcting the erroneous disparity values of the identified pixels.

9. Signal generating device according to claim 8 characterized in that the filtering device associated with the disparity map is located at the transmitting device.

10. Signal generation device according to claim 8 characterized in that the filtering device associated with the disparity map is located at the receiving device.