Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
  • G06T7/55—Depth or shape recovery from multiple images
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/50—Depth or shape recovery
  • G06T7/55—Depth or shape recovery from multiple images
  • G06T7/593—Depth or shape recovery from multiple images from stereo images
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/128—Adjusting depth or disparity
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2200/00—Indexing scheme for image data processing or generation, in general
  • G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data

Definitions

• the present invention relates to a method for post-processing a digital video signal, said digital video signal having a plurality of views with associated disparity maps.
• Such a method may be used in, for example, a video communication system for 3D video applications within MPEG standards.
• 3D scenes rely on depth maps and disparity maps.
• the depth map of a point of view is a grayscale image, each pixel of which contains the distance to a camera filming the scene.
• Reconstruction the new point of view created being called a Reconstructed view (or reconstructed image). If a point P of the scene is visible from both points of view, a translation vector will give its pixel coordinate in the new point of view from its pixel coordinate in the original one. These vectors are called disparity vectors. Projective geometry results, as disclosed in the document “three dimensional computer vision - O.Faugeras MIT Press 1993", establish a simple relation between disparity vectors of the disparity map and depth values of the depth map.
• multi-view or stereo coding schemes During transmission of a video signal, multi-view or stereo coding schemes, well known to those skilled in the art, generally encode by compression the textures and the depth maps needed to cover a certain range of points of view.
• texture can be encoded by using standard methods, potentially leading to well-known artifacts in the case of lossy encoding
• depth (or disparity) encoding is a little more tricky: for an encoded depth map to have a visually similar aspect as the original one does not necessarily mean that it has the same reconstruction properties.
• points or areas could be translated to the wrong place (because of wrong disparity vectors). This would create texture discontinuities that may be more noticeable than what the "visual" aspect of the encoded map suggested.
• depth map can be encoded using the Multiple Auxiliary Component tools (MAC) (as described in « Amendment 1 : Visual extensions, ISO/IEC JTC 1/SC 29/WG 11 N 3056, December 1999 »), in which they are DCT encoded on a block basis, similarly to a classic luminance image encoding well known to those skilled in the art.
• MAC Multiple Auxiliary Component tools
• No specific treatments of the underlying artifacts are proposed but traditional MPEG tools that, as previously stated, are good for texture but not necessarily for depth maps.
• this can lead to a fuzzy edge along with isolated texture pixels, two effects that, moreover, are time-inconsistent in the course of following points of view.
• a method as defined in claim 1 for post-processing a digital video signal, said digital video signal having a plurality of views with associated disparity maps, which method corrects depth map (or disparity map) coding artifacts.
• An embodiment of the method according to the invention comprises the steps defined in claim 2.
• a further embodiment of the method according to the invention comprises the steps defined in claim 3.
• a further embodiment of the method according to the invention comprises the steps defined in claim 4.
• a further embodiment of the method according to the invention comprises the steps defined in claim 5.
• a further embodiment of the method according to the invention comprises the steps defined in claim 6.
• a further embodiment of the method according to the invention comprises the steps defined in claim 7.
• FIG. 1 is a schematic diagram of the method for post-processing a video signal according to the invention
• Fig. 2 depicts some holes in a projected disparity map, holes generated by the first step of the method for post-processing a video signal of Fig. 1,
• - Fig. 3 depicts the filtering of holes in the projected disparity map during a first step of the method for post-processing a video signal of Fig. 1
• - Fig. 4 illustrates the filling of holes in the projected disparity map during a second step of the method for post-processing a video signal of Fig. 1
• - Fig. 5 illustrates the filtering of undefined pixel values during the third step of the method for post-processing a video signal of Fig. 1.
• the present invention relates to a method for post-processing a digital video signal. Such a method may be used within a video communication system for 3D video applications in MPEG4.
• a 3D video signal comprises a plurality of points of view with different associated characteristics such as shape, texture, motion vectors, disparity map, depth map, colors, etc.
• a video signal When a video signal is transmitted, it is encoded. During the encoding process, its different characteristics are encoded and especially the disparity and depth maps with a compression algorithm. This compression may lead to impaired disparity and depth maps with artifacts.
• the post-processing method is based on the fact that, during the reconstruction of a new view from another view and its impaired depth map, the depth map, as described further, undergoes some transformation (for example projection), making it easier to detect problematic values than when using the original decoded map.
• some transformation for example projection
• disparity vectors are defined by a single value. Therefore, we will further refer to them as "disparity values”.
• Io will be the original texture image and I N the new reconstructed view
• d will be the disparity map
• d(x,y) will be the disparity value at pixel ( j>).
• the reconstruction process can be done in several ways, a common one of which (as described in the document "Disparity field and depth map coding for multi-view 3D image generation, by D.Tzovaras, N.Grammalidis, G.Strinzis, Signal Processing: Image Communication 11 (1998) pages 205-230)), can be divided into the following main steps that will be detailed hereinafter:
• a projected disparity map is generated from the original disparity map of a point of view. N pixels of a view are projected on one pixel of the future view.
• N can take any value:
• a list of disparity values corresponds to this pixel of the future view.
• the set of the lists corresponding to every pixel of the reconstructed view is the projected disparity map.
• the projected disparity map should have more or less the same regularity as the original one. It also means that, if the original disparity map has some irregularities, the projected disparity map will have the same irregularities. If we consider the number of points projected at each pixel, there should also be some kind of regularity. In particular, in the middle of holes, there should not be isolated projected pixels. However, due to the DCT quantization process used to encode the disparity map, some pixel values are changed, and once projected, they are no longer close to their originally surrounding pixels. Such pixels are called isolated projected pixels. As we can see in Fig. 2, there is an original disparity map Do, which has two areas of pixels with the same values dol and do2.
• the projected disparity map Dp has two corresponding areas of pixels dpi, dp2, but with areas Hpl, Hp2 of "hole". Due to some compression artefacts, some pixels could be erroneously projected on these holes areas. They are called isolated pixels.
• the method for post-processing removes those isolated projected pixels by using a first filtering FI.
• the filtering is performed as follows. For each pixel Pp of the projected disparity map Dp, the number of surrounding pixels Pps defined as holes is counted (the notion of "surrounding pixel" is application dependent: it may be a square neighborhood centered on the pixel, a circular, rectangular one). If this number goes beyond a certain threshold Tl, for example more than half the number of pixels contained in the neighborhood, the considered isolated pixel is set as "hole” and any pixel of the original disparity map leading to this pixel is marked as "wrong disparity value".
• a first wrong pixel Po2 of the original disparity map Do has a corresponding isolated projected pixel Pp2 on the area Dp2 of the projected disparity map Dp.
• a second Pol 1 and a third Pol 2 pixel of the original disparity map Do has the same corresponding isolated projected pixel Ppl on the area Dpi of the projected disparity map Dp.
• the window in dotted line defines the surrounding pixels Pps of Pp2, here defined as holes, which makes Pp2 an isolated pixel. The same holds for Ppl.
• pixels of the original disparity map Do corresponding to Ppl and Pp2 and Pol 1 and Pol 2, and Po2, respectively, are set as "wrong value".
• the filtering is done as follows. It is also possible to detect erroneous values by checking other isolated pixels of the projected map, as far as the number of projected pixel values is concerned.
• the isolated projected pixels are those that have no coherent characteristics with the ones of their surrounding pixels.
• the number of values of a pixel of the projected disparity map ranges between 0 and n (n depends on the original disparity map).
• the required precision in isolated pixels detection will determine which kind of isolated pixel one wants to process: the isolated pixels surrounded by holes, then the ones having two or more values, and which are surrounded by pixels containing only one value, and so on.
• a second step 2) there is a filling of the holes of the projected disparity map.
• the second step following projected disparity map generation, consists in filling the holes of this disparity map through interpolation.
• the interpolation type is selected by considering that, in the final reconstructed view, the holes correspond to something belonging to the background rather than to the foreground object (which is the most common case in "real life"). An example is given below.
• the projected disparity domain we repeat the hole-boundary disparity value over the entire line of the hole, through a padding process.
• the padding process consists in taking the boundary disparity value, which is the nearest of the hole and pads the entire line of said hole with it.
• interpolation strongly depends on hole boundary disparity values. The problem when using compressed (i.e.
• the hole boundary values used for this padding process might be significantly changed by the compression algorithm and therefore might be erroneous, moreover in a way that varies from one line to the other: then, the padding process is performed on the basis of these erroneous values, which are propagated over the entire line in the hole. Visually, it can create what we call "worn edges" in the reconstructed view.
• a new specific post-processing filter F2 for hole boundary values is applied, which avoids the problems stated above. It applies to the boundaries of a hole and not the hole itself.
• filtering is performed only at boundaries of holes that are larger than a given (application-dependent) threshold T2, for example, 3 pixels. Then, for these selected holes H, median filtering is performed on boundary pixel values, taking into account the surrounding projected pixels values (excluding holes of course), as shown in Fig. 4.
• the projected disparity map Dp comprises two areas; one filed with pixels P and the other with holes H.
• the hole HI is selected in the projected disparity map Dp.
• a window W of pixel values (which are not holes) is taken around the boundary pixel value Vkl of the considered hole HI. Once all the pixel values within the window W are sorted, the median value is taken and replaces the boundary pixel value Vkl.
• the regular padding process is performed on the hole HI itself with the new pixel value Vkl .
• a boundary pixel of a selected hole in the projected disparity map Dp is the pixel number 11 with a value 10.
• the corresponding pixel in the original disparity map DO is a pixel number 1 with the value 10.
• After median filtering of the boundary pixel number 11, its new value is 5.
• the corresponding pixel number 1 in the original disparity map Do takes the value "wrong".
• the pixel number 6 in the original disparity map Do is assigned the value 5.
• final regularization is preferably performed. It consists in regularization of the projected disparity map by a median filtering F3 over the lines (this enables the projected disparity to be more regular and to smooth the irregularities that could not be detected by the previous filtering, for example because they were not located in the vicinity of big disparity discontinuities). As before, possible changes are translated to the original disparity map.
• a fourth step 4 final I N view generation based on the hole-filled projected disparity map is performed.
• the final reconstructed view R_VIEW can be generated from this filtered projected disparity map as is well known to the person skilled in the art as described in the document "Disparity field and depth map coding for multi-view 3D image generation, by D.Tzovaras, N.Grammalidis, G.Strinzis, Signal Processing: Image Communication 11 (1998) pages 205-230".
• the reconstructed view is generated from a modified version of the original disparity map as described below. Note that during the various filtering processes in steps 1 and 2, whenever a pixel is modified or labeled as "Wrong Disparity Value" in the projected disparity map, the original disparity map is marked and modified as well as described in the foregoing. These modified or labeled pixels are thus translated to the original disparity map.
• the original decoded disparity map with the undefined values i.e. the modified original disparity map
• one of the boundary pixel values of the undefined values is copied: in general, there are two possible boundary pixel value candidates to choose from, one on the left and one on the right of the pixel containing the undefined value.
• One choice would be to use a bilinear or nearest neighbor interpolation, but the variable size of the holes could make the interpolation inconsistent in space.
• the boundary pixel that has the closest value to the original pixel in the original decoded disparity map is preferably chosen.
• disparity map Dcco one can see one undefined value WV1 that has to be corrected. It corresponds to a pixel Pcol in the original decoded disparity map Deo.
• Psl is the pixel that has the nearest value from the corresponding original pixel Pcol of the original decoded disparity map Deo. Said pixel Psl replaces the undefined value WV1.
• the present invention is not limited to the aforementioned video application. It can be used within any application using a system for processing a signal taking compressed disparity maps into account.
• the invention applies to video compression algorithms of the other MPEG standard family (MPEG-1, MPEG-2) and to the ITU H26X family (H261, H263 and extensions, H261 being the latest today, reference number Q15-K-59) if applied to disparity map coding.
• the method according to the present invention is not limited to the aforementioned implementation.
• Said hardware or software items can be implemented in several manners, such as by means of wired electronic circuits or by means of an integrated circuit that is suitably programmed.
• the integrated circuit can be contained in a computer or in a decoder.
• the decoder comprises means for performing steps 1, 2, 3 and 4 of the method for post-processing a video signal, as described previously, said means being hardware or software items as stated above.
• the integrated circuit comprises a set of instructions.
• said set of instructions contained, for example, in a computer programming memory or in a decoder memory may cause the computer or the decoder to carry out the different steps of the post-processing method.
• the set of instructions may be loaded into the programming memory by reading a data carrier such as, for example, a disk.
• a service provider can also make the set of instructions available via a communication network such as, for example, the Internet.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Image Processing (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (2)

Family

ID=32116329

Family Applications (1)

Country Status (7)

Cited By (6)

Families Citing this family (34)

Family Cites Families (8)

• 2003
  • 2003-10-01 US US10/531,935 patent/US7224355B2/en not_active Expired - Fee Related
  • 2003-10-01 JP JP2004546237A patent/JP4485951B2/ja not_active Expired - Fee Related
  • 2003-10-01 KR KR1020057006878A patent/KR100960294B1/ko not_active Expired - Fee Related
  • 2003-10-01 EP EP03809395A patent/EP1574079B1/en not_active Expired - Lifetime
  • 2003-10-01 WO PCT/IB2003/004363 patent/WO2004039086A2/en not_active Ceased
  • 2003-10-01 CN CN200380101874A patent/CN100584039C/zh not_active Expired - Fee Related
  • 2003-10-01 AU AU2003263557A patent/AU2003263557A1/en not_active Abandoned

Also Published As

Similar Documents

Legal Events