WO2013113134A1 - Appareil, procédé et programme informatique de codage et décodage pour vidéo - Google Patents

Appareil, procédé et programme informatique de codage et décodage pour vidéo Download PDF

Info

Publication number
WO2013113134A1
WO2013113134A1 PCT/CN2012/000139 CN2012000139W WO2013113134A1 WO 2013113134 A1 WO2013113134 A1 WO 2013113134A1 CN 2012000139 W CN2012000139 W CN 2012000139W WO 2013113134 A1 WO2013113134 A1 WO 2013113134A1
Authority
WO
WIPO (PCT)
Prior art keywords
block
depth
texture
adjacent
disparity
Prior art date
Application number
PCT/CN2012/000139
Other languages
English (en)
Inventor
Dmytro Rusanovskyy
Miska Hannuksela
Wenyi SU
Original Assignee
Nokia Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Corporation filed Critical Nokia Corporation
Priority to PCT/CN2012/000139 priority Critical patent/WO2013113134A1/fr
Publication of WO2013113134A1 publication Critical patent/WO2013113134A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8451—Structuring of content, e.g. decomposing content into time segments using Advanced Video Coding [AVC]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
    • H04N19/139—Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/513—Processing of motion vectors
    • H04N19/517—Processing of motion vectors by encoding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/593—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial prediction techniques
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234327—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into layers, e.g. base layer and one or more enhancement layers

Definitions

  • the present invention relates to an apparatus, a method and a computer program for video coding and decoding.
  • DIBR depth image-based rendering
  • a typical implementation of DIBR takes stereoscopic video and corresponding depth information with stereoscopic baseline as input and synthesizes a number of virtual views between the two input views.
  • DIBR algorithms may also enable extrapolation of views that are outside the two input views and not in between them.
  • DIBR algorithms may enable view synthesis from a single view of texture and the respective depth view.
  • video compression systems such as Advanced Video Coding standard H.264/AVC or the Multiview Video Coding MVC extension of H.264/AVC can be used.
  • the motion vector prediction specified in H.264/AVC/MVC may be not optimal for video coding systems utilizing inter-view and/or view synthesis prediction (VSP) along with inter prediction.
  • VSP view synthesis prediction
  • MVP motion vector prediction
  • This invention proceeds from the consideration that the depth or disparity information (Di) for a current block (cb) of texture data is available through decoding of coded depth or disparity information or can be estimated at the decoder side prior to decoding of the current texture block cb, thus making it possible to utilize depth or disparity information in MVP process.
  • the utilization of depth or disparity information (Di) in MVP improves compression in multi-view, multi-view + depth, and MVC-VSP coding systems.
  • the cb term is used to note the current block of texture data, the depth or disparity information associated with cb is named as d(cb).
  • the current block of texture data is defined as a texture block being coded by an encoder or encoding method or decoded by a decoder or decoding method.
  • encoder/decoder may use 2D blocks of texture data (A,B,C and so on). These blocks are called adjacent blocks and they are spatially adjacent to cb image area (2D fragment of image adjacent to or surrounding the cb) and which is assumed to be available prior to coding/decoding of cb. See Figure 15, where 2D image fragment adjacent to cb and utilized in MVP is shown in grey color.
  • MVP process for cb may use adjacent 2D blocks of texture data (A,B,C and so on), which are located in 2D fragments of other images within the same video (video fragment) adjacent to the cb block, see figure 16. This video fragment is assumed to be available (coded/decoded) prior to coding/decoding of the cb.
  • MVP process for cb may use adjacent 2D blocks of texture data (A,B,C and so on), which are located in 2D fragments of other images which are located in different views of the same multiview video data (multiview video fragment) adjacent to the cb block, see figure 16.
  • Such multiview video fragment is assumed to be available (coded/decoded) prior to coding/decoding of the cb.
  • adjacent blocks A, B,C and so on may be located in spatio/temporal/inter-view proximity of the cb block in several 2D images which are available (coded/decoded) prior to the current image coding process.
  • Encoder/decoder can take in use motion information (such as horizontal motion vector component mv_x, vertical motion vector component mv_y, and reference frames which may be identified for example using reference frame indexes refldx to one or more reference picture lists) associated with these blocks MV(A),MV(B),MV(C) as well as the depth/disparity information associated with these blocks, d(A), d(B), d(C), as they assumed to be available prior to coding/decoding cb.
  • motion information such as horizontal motion vector component mv_x, vertical motion vector component mv_y, and reference frames which may be identified for example using reference frame indexes refldx to one or more reference picture lists
  • Spatial resolution of an image is defined as the number of pixels (image samples) representing an image in horizontal and vertical direction.
  • images at different resolution can be interpreted as two images have different number of pixels either in horizontal direction, or in vertical direction, or in both directions.
  • Motion information can be of certain accuracy or precision MV(A) corresponding to a certain resolution.
  • the H.264/AVC coding standard uses 1 ⁇ 4-pixel location motion vector precision, which in many implementations requires that the reference image is up-sampled by factor of 4x from the original image resolution along both coordinate axes.
  • motion vector resolution refers to the reference image resolution at which this motion vector is obtained during motion estimation procedure.
  • motion vector resolution is 4x relative to the original image resolution along both coordinate axes when 1 ⁇ 4-pixel motion vector precision is in use.
  • Many coding schemes pre-define the motion vector precision, although coding schemes of adaptive motion vector precision, where the motion vector precision is selected by the encoder have also been described.
  • Encoder may choose motion vector accuracy, i.e. how accurately motion estimation is actually performed, to be lower than or equal to the precision, while many times encoders use motion vector accuracy equal to the precision.
  • motion vectors are represented at 1 ⁇ 4-pixel precision in the bitstream, but the encoder may choose to perform motion estimation at 1 ⁇ 2-pixel accuracy, i.e. search only full- pixel and 1 ⁇ 2-pixel positions.
  • motion vector precision and accuracy may be used interchangeably as synonyms.
  • a method comprising: decoding, from a bitstream, a first coded texture block of a first coded texture picture into a first texture block cb, wherein the decoding the first coded texture block comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co-located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block A and the second adjacent texture block B on the basis a similarity value resulting from said comparison; deriving one or more prediction parameters for the decoding of the first coded texture block cb from values associated with the selected one or
  • said prediction parameters may include one or more of the following: a number of prediction blocks, such as uni- or bi-prediction; a type of one or more prediction references used, such as inter, inter-view and view synthesis; one or more reference pictures used; motion vector predictors or motion vectors to be applied a method of motion vector prediction to be applied; a zero-valued prediction error signal to be inferred.
  • the method of the first embodiment may further comprise conditioning the selection of one or both of the first adjacent texture block A and the second adjacent texture block B on the basis the similarity value resulting from said comparison on one or more threshold values obtained from the bitstream.
  • the method of the first embodiment may further comprise obtaining the first depth/disparity block d(cb) by decoding from the bitstream or by estimating.
  • the method may further comprise setting the motion vector predictor to a value derived from the depth/disparity information of the current block of texture data d(cb).
  • decoding a first coded texture block cb may comprise processing of more than two adjacent blocks.
  • said selecting the first adjacent texture block A and the second adjacent texture block B for the first texture block cb may comprise one of the following: selecting the first adjacent texture block A and the second adjacent texture block B which are located in 2D fragment of image (set of pixels which belong to the same image) adjacent to or surrounding the cb; selecting the first adjacent texture block A and the second adjacent texture block B which are located in video fragment (set of pixels that belong to different images of the same video data) or multiview video fragment (set of pixels that belong to different images of the same multiview video data) adjacent to the cb block.
  • said obtaining a depth/disparity block associated with adjacent texture block may comprise one or more of the following: selecting a first adjacent texture block Z and a second adjacent texture block Y; obtaining motion information MV(Z) utilized for decoding of the texture block Z and motion information MV(Y) utilized for decoding of the second adjacent texture block Y; obtaining one or more motion information candidates MV(X) obtained from the motion information MV(Z) and/or the motion information MV(Y); obtaining one or more motion information candidates MV(d(cb)) obtained from depth/disparity information d(cb) associated with first texture block cb; obtaining a texture block A through applying the motion information MV(Z) and counted from location of the first texture block cb; obtaining a texture block B through applying the motion information MV(Y) and counted from location of the first texture block cb; obtaining one or more texture blocks through applying the motion information MV(X) and counted from location of the first texture
  • said selecting the first adjacent texture block A and the second adjacent texture block B for the first texture block cb may comprise one of the following: selecting a co-located block in a first reference picture as the first adjacent texture block A and selecting the second adjacent texture block B based on motion information or depth information associated with the first adjacent texture block, MV(A) and/or d(A); using the first depth/disparity block d(cb) for selecting a block in a second reference picture as the first adjacent texture block A and selecting the second adjacent texture block B based on values associated with the first adjacent texture block A; using the first depth/disparity block d(cb) for selecting a block in a third reference picture as the first adjacent texture block A and selecting the second adjacent texture block B which is located in 2D fragment of image or in video fragment or multiview video fragment adjacent to the first adjacent texture block A.
  • the method may comprise: spatial resolutions of texture and depth images are normalized by re-sampling of either one of component (texture or depth) to the resolution of another component (depth or texture), or both components are resampled to a single spatial resolution.
  • the method may comprise: motion information MV(A) and MV(B) of adjacent texture blocks A and B respectively are rescaled to meet spatial resolution of the depth image instead of resampling of depth image.
  • Said motion information may include motion vector components motion vector components, motion partitions sizes and so on.
  • the method may comprise: said comparison which produces similarity metric from application of motion information of adjacent texture blocks MV(A), MV(B) to depth image is adjusted to reflect the difference in resolution between texture and depth images. Said adjustment may be include decimation, subsampling, interpolation or upsampling depth information d(MV(A), d(cb)) and d(MV(B), d(cb)) to match with resolution of motion information obtained from adjacent texture blocks A, B and so on.
  • the method may comprise: said comparison which produces similarity metric from application of motion information of adjacent texture blocks MV(A), MV(B) to non-regularly sampled depth information is adjusted to reflect difference in representation or sampling method utilized for texture and depth data.
  • Said adjustment may include resampling (downsampling, upsampling, rescaling) of depth, texture or motion information, as well as linear and non-linear operations of merging or aggregating of depth information represented with non-uniform sampling method.
  • the method may comprise: information required for decoding operation is signaled to decoder through the bitstream.
  • Said information may include signaling of method utilized for spatial resolution normalization of texture and/or depth images, or signaling of methods utilized for rescaling of motion information (e.g. rounding, downscaling/upscaling factor and so on), or signaling of method utilized for comparison adjustment (rescaling, resampling or non-linear operations of merging or aggregation).
  • an apparatus comprising a video decoder configured for decoding, from a bitstream, a first coded texture block of a first coded texture picture into a first texture block cb, wherein the decoding the first coded texture block comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co- located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block (A or B) and the second adjacent texture block on the basis a similarity value resulting from said comparison between the first depth/disparity block d(cb) and the first adjacent depth/dispar
  • the method may comprise: spatial resolutions of texture and depth images are normalized by in-loop re-sampling of either one of component (texture or depth) to the resolution of another component (depth or texture), or both components are resampled to a single spatial resolution.
  • the method may comprise: motion information MV(A) and MV(B) of adjacent texture blocks A and B respectively are rescaled to meet spatial resolution of the depth image instead of resampling of depth image.
  • Said motion information may include motion vector components motion vector components, motion partitions sizes and so on.
  • the method may comprise: said comparison which produces similarity metric from application of motion information of adjacent texture blocks MV(A), MV(B) to depth image is adjusted to reflect the difference in resolution between texture and depth images. Said adjustment may be include decimation, subsampling, interpolation or upsampling depth information d(MV(A), d(cb)) and d(MV(B), d(cb)) to match with resolution of motion information obtained from adjacent texture blocks A, B and so on.
  • the method may comprise: said comparison which produces similarity metric from application of motion information of adjacent texture blocks MV(A), MV(B) to non-regularly sampled depth information is adjusted to reflect difference in representation or sampling method utilized for texture and depth data.
  • Said adjustment may include resampling (downsampling, upsampling, rescaling) of depth, texture or motion information, as well as linear and non-linear operations of merging or aggregating of depth information represented with non-uniform sampling method.
  • the method may comprise: information required for decoding operation is decoded from the bitstream.
  • Said information may include decoding indexes for method utilized for spatial resolution normalization of texture and/or depth images, or decoding indexes for methods utilized for rescaling of motion information (e.g. rounding, downscaling/upscaling factor and so on), or decoding indexes for method utilized for comparison adjustment (rescaling, resampling or non-linear operations of merging or aggregation).
  • a computer readable storage medium stored with code thereon for use by an apparatus, which when executed by a processor, causes the apparatus to perform: decoding, from a bitstream, a first coded texture block of a first coded texture picture into a first texture block cb, wherein the decoding the first coded texture block comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co-located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block (A or B) and the second adjacent texture block on the basis a similarity value resulting from said comparison between the
  • an apparatus comprising at least one processor and at least one memory, said at least one memory stored with code thereon, which when executed by said at least one processor, causes an apparatus to perform: decoding, from a bitstream, a first coded texture block of a first coded texture picture into a first texture block cb, wherein the decoding the first coded texture block comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co- located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block (A or B) and the second adjacent texture block on the basis a similarity value
  • a video decoder configured for decoding, from a bitstream, a first coded texture block of a first coded texture picture into a first texture block cb, wherein the decoding the first coded texture block comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co-located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block A and the second adjacent texture block B on the basis a similarity value resulting from said comparison between the first depth/disparity block d(cb) and the first adjacent depth/disparity block d(A
  • a method comprising: encoding a first uncompressed texture block cb of a first uncompressed texture picture into a first coded texture block of a first coded texture picture of a bitstream, wherein the encoding the first uncompressed texture block cb comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co-located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block (A or B) and the second adjacent texture block on the basis a similarity value resulting from said comparison between the first depth/disparity block d(
  • an apparatus comprising a video encoder configured for encoding a first uncompressed texture block cb of a first uncompressed texture picture into a first coded texture block of a first coded texture picture of a bitstream, wherein the encoding the first uncompressed texture block cb comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co- located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block A and the second adjacent texture block B on the basis a similarity value resulting from said comparison between the first depth/disparity block d
  • a computer readable storage medium stored with code thereon for use by an apparatus, which when executed by a processor, causes the apparatus to perform: encoding a first uncompressed texture block cb of a first uncompressed texture picture into a first coded texture block of a first coded texture picture of a bitstream, wherein the encoding the first uncompressed texture block cb comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co-located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block A and the second adjacent texture block B on the basis
  • At least one processor and at least one memory said at least one memory stored with code thereon, which when executed by said at least one processor, causes an apparatus to perform: encoding a first uncompressed texture block cb of a first uncompressed texture picture into a first coded texture block of a first coded texture picture of a bitstream, wherein the encoding the first uncompressed texture block cb comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co-located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block A and the second adjacent texture block
  • a video encoder configured for encoding a first uncompressed texture block cb of a first uncompressed texture picture into a first coded texture block of a first coded texture picture of a bitstream, wherein the encoding the first uncompressed texture block cb comprises: selecting a first adjacent texture block A and a second adjacent texture block B; obtaining a first adjacent depth/disparity block d(A) and a second adjacent depth/disparity d(B); obtaining a first depth/disparity block d(cb) spatially co-located with the first texture block cb; comparing the first depth/disparity block d(cb) with the first adjacent depth/disparity block d(A) and the second adjacent depth/disparity block d(B); selecting one or both of the first adjacent texture block A and the second adjacent texture block B on the basis a similarity value resulting from said comparison between the first depth/disparity block d(
  • the method comprises encoding a first data element with a first method being the method of the first embodiment; calculating a first cost metric Costl for the coded first data element; encoding a second data element with a second method being a motion vector prediction method alternative to the first method; calculating a second cost metric Cost2 for coded second data element; selecting between the first and the second method a method which is determined to be optimal with respect to first and second cost metrics Costl and Cost2; encoding the first and second data elements with the selected method; signalling an index indicating the selected method through the bitstream.
  • a first data element may include either a single texture block Cb or a set of coded blocks (slice, image, group of images).
  • the second data element may include a set of texture block (A, B).
  • said cost metric can be compound of rate-distortion metric, or other cost metrics, such as rate- distortion-complexity metric.
  • said signaling is performed at the various level of the coded data representation.
  • the index is signaled in at least one of the following: at the sequence parameters set, at picture parameters set, at the slice header or along with motion information for a particular block partition.
  • the method comprises decoding, from a bitstream, an index indicating a method being utilized for decoding of a data set: decoding, from a bitstream, decoded data set specification where said method being indicated by the index is applicable; applying a first method being the method of claim 1 if decoding of bitstream specified so; applying a second method being a motion vector prediction method alternative to the first method if decoding of bitstream specified so.
  • Figure 1 shows a simplified 2D model of a stereoscopic camera setup
  • Figure 2 shows a simplified model of a multiview camera setup
  • Figure 3 shows a simplified model of a multiview autostereoscopic display (ASD);
  • Figure 4 shows a simplified model of a DIBR-based 3DV system
  • Figures 5 and 6 show an example of a TOF-based depth estimation system
  • Figures 7a and 7b show the spatial and temporal neighbourhood of the currently coded block serving as the candidates for MVP in H.264/AVC;
  • Figure 8 shows a flow chart of a depth/disparity information based
  • Figure 9 shows a flow chart of a depth/disparity information based
  • FIG. 10 shows schematically an electronic device suitable for employing some embodiments of the invention.
  • Figure 1 1 shows schematically a user equipment suitable for employing some embodiments of the invention
  • Figure 12 further shows schematically electronic devices employ embodiments of the invention connected using wireless and wired network connections
  • Figure 13 shows an example of blocks layout cb, A,B,C, D;
  • Figure 14 shows blocks of texture data (cb, S, T,U) and depth/disparity data associated with these blocks, d(cb), d(S), d(T) and d(U) respectively;
  • Figure 15 shows conception of spatially adjacent blocks of texture data
  • Figure 16 shows conception of adjacent blocks in 2D texture or multiview texture data
  • Figure 17 shows flowchart of an example of implementation for depth-based motion vector competition in Skip mode for
  • FIG. 18 shows flowchart of an example of implementation for depth-based motion vector competition in Direct mode for B-slices
  • Figure 19 shows a flowchart of possible motion vector prediction (MVP) process
  • Figure 20 shows reference non-uniformed sampled depth image and reference uniformly sampled texture image
  • Figure 21 shows an example of mapping of a depth map into another view
  • Figure 22 shows an example of generation of an initial depth map estimate after coding a first dependent view of a random access unit
  • Figure 23 shows an example of derivation of a depth map estimate for the current picture using motion parameters of an already coded view of the same access unit.
  • Figure 24 shows an example of updating of a depth map estimate for a dependent view based on coded motion and disparity vectors
  • H.264/AVC Some key definitions, bitstream and coding structures, and concepts of H.264/AVC are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented.
  • the aspects of the invention are not limited to H.264/AVC, but rather the description is given for one possible basis on top of which the invention may be partly or fully realized.
  • the H.264/AVC standard was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunications Standardisation Sector of International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of International Standardisation Organisation (ISO) / International Electrotechnical Commission (I EC).
  • H.264/AVC The H.264/AVC standard is published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.264 and ISO/IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC).
  • AVC MPEG-4 Part 10 Advanced Video Coding
  • SVC Scalable Video Coding
  • MVC Multiview Video Coding
  • High Efficiency Video Coding is another and more recent development of video coding technology by the Joint Collaborative Team - Video Coding (JCT-VC) of VCEG and MPEG.
  • JCT-VC Joint Collaborative Team - Video Coding
  • H.264/AVC and HEVC Some key definitions, bitstream and coding structures, and concepts of H.264/AVC and HEVC are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented. Some of the key definitions, bitstream and coding structures, and concepts of H.264/AVC are the same as in the current working draft of HEVC - hence, they are described below jointly. The aspects of the invention are not limited to H.264/AVC or HEVC, but rather the description is given for one possible basis on top of which the invention may be partly or fully realized.
  • bitstream syntax and semantics as well as the decoding process for error-free bitstreams are specified in H.264/AVC and HEVC.
  • the encoding process is not specified, but encoders must generate conforming bitstreams.
  • Bitstream and decoder conformance can be verified with the Hypothetical Reference Decoder (HRD), which is specified in Annex C of H.264/AVC.
  • HRD Hypothetical Reference Decoder
  • the standard contains coding tools that help in coping with transmission errors and losses, but the use of the tools in encoding is optional and no decoding process has been specified for erroneous bitstreams.
  • the elementary unit for the input to an H.264/AVC or HEVC encoder and the output of an H.264/AVC or HEVC decoder is a picture.
  • a picture may either be a frame or a field.
  • a frame comprises a matrix of luma samples and corresponding chroma samples.
  • a field is a set of alternate sample rows of a frame and may be used as encoder input, when the source signal is interlaced.
  • Chroma pictures may be subsampled when compared to luma pictures. For example, in the 4:2:0 sampling pattern the spatial resolution of chroma pictures is half of that of the luma picture along both coordinate axes and consequently a macroblock contains one 8x8 block of chroma samples per each chroma component.
  • a picture is partitioned to one or more slice groups, and a slice group contains one or more slices.
  • a slice consists of an integer number of macroblocks ordered consecutively in the raster scan within a particular slice group
  • a macroblock is a 16x16 block of luma samples and the corresponding blocks of chroma samples. For example, in the 4:2:0 sampling pattern, a macroblock contains one 8x8 block of chroma samples per each chroma component.
  • a picture is partitioned to one or more slice groups, and a slice group contains one or more slices.
  • a slice consists of an integer number of macroblocks ordered consecutively in the raster scan within a particular slice group.
  • a CU coding units
  • a CU consists of one or more prediction units (PU) defining the prediction process for the samples within the CU and one or more transform units (TU) defining the prediction error coding process for the samples in the said CU.
  • PU prediction units
  • TU transform units
  • a CU consists of a square block of samples with a size selectable from a predefined set of possible CU sizes.
  • a CU with the maximum allowed size is typically named as LCU (largest coding unit) and the video picture is divided into non-overlapping LCUs.
  • An LCU can be further split into a combination of smaller CUs, e.g. by recursively splitting the LCU and resultant CUs.
  • Each resulting CU typically has at least one PU and at least one TU associated with it.
  • Each PU and TU can be further split into smaller PUs and TUs in order to increase granularity of the prediction and prediction error coding processes, respectively.
  • the PU splitting can be realized by splitting the CU into four equal size square PUs or splitting the CU into two rectangle PUs vertically or horizontally in a symmetric or asymmetric way.
  • the division of the image into CUs, and division of CUs into PUs and TUs is typically signalled in the bitstream allowing the decoder to reproduce the intended structure of these units.
  • a picture can be partitioned in tiles, which are rectangular and contain an integer number of LCUs.
  • the partitioning to tiles forms a regular grid, where heights and widths of tiles differ from each other by one LCU at the maximum.
  • a slice consists of an integer number of CUs. The CUs are scanned in the raster scan order of LCUs within tiles or within a picture, if tiles are not in use. Within an LCU, the CUs have a specific scan order.
  • WD Working Draft
  • a basic coding unit in a HEVC WD5 is a treeblock.
  • a treeblock is an NxN block of luma samples and two corresponding blocks of chroma samples of a picture that has three sample arrays, or an NxN block of samples of a monochrome picture or a picture that is coded using three separate colour planes.
  • a treeblock may be partitioned for different coding and decoding processes.
  • a treeblock partition is a block of luma samples and two corresponding blocks of chroma samples resulting from a partitioning of a treeblock for a picture that has three sample arrays or a block of luma samples resulting from a partitioning of a treeblock for a monochrome picture or a picture that is coded using three separate colour planes.
  • Each treeblock is assigned a partition signalling to identify the block sizes for intra or inter prediction and for transform coding.
  • the partitioning is a recursive quadtree partitioning.
  • the root of the quadtree is associated with the treeblock.
  • the quadtree is split until a leaf is reached, which is referred to as the coding node.
  • the coding node is the root node of two trees, the prediction tree and the transform tree.
  • the prediction tree specifies the position and size of prediction blocks.
  • the prediction tree and associated prediction data are referred to as a prediction unit.
  • the transform tree specifies the position and size of transform blocks.
  • the transform tree and associated transform data are referred to as a transform unit.
  • the splitting information for luma and chroma is identical for the prediction tree and may or may not be identical for the transform tree.
  • the coding node and the associated prediction and transform units form together a coding unit.
  • a slice may be a sequence of treeblocks but (when referring to a so-called fine granular slice) may also have its boundary within a treeblock at a location where a transform unit and prediction unit coincide. Treeblocks within a slice are coded and decoded in a raster scan order. For the primary coded picture, the division of each picture into slices is a partitioning.
  • a tile is defined as an integer number of treeblocks co- occurring in one column and one row, ordered consecutively in the raster scan within the tile.
  • the division of each picture into tiles is a partitioning. Tiles are ordered consecutively in the raster scan within the picture.
  • a slice contains treeblocks that are consecutive in the raster scan within a tile, these treeblocks are not necessarily consecutive in the raster scan within the picture.
  • Slices and tiles need not contain the same sequence of treeblocks.
  • a tile may comprise treeblocks contained in more than one slice.
  • a slice may comprise treeblocks contained in several tiles.
  • in-picture prediction may be disabled across slice boundaries.
  • slices can be regarded as a way to split a coded picture into independently decodable pieces, and slices are therefore often regarded as elementary units for transmission.
  • encoders may indicate in the bitstream which types of in-picture prediction are turned off across slice boundaries, and the decoder operation takes this information into account for example when concluding which prediction sources are available. For example, samples from a neighboring macroblock or CU may be regarded as unavailable for intra prediction, if the neighboring macroblock or CU resides in a different slice.
  • NAL Network Abstraction Layer
  • Decoding of partially lost or corrupted NAL units is typically difficult.
  • NAL units are typically encapsulated into packets or similar structures.
  • a bytestream format has been specified in H.264/AVC or HEVC for transmission or storage environments that do not provide framing structures. The bytestream format separates NAL units from each other by attaching a start code in front of each NAL unit.
  • encoders run a byte-oriented start code emulation prevention algorithm, which adds an emulation prevention byte to the NAL unit payload if a start code would have occurred otherwise.
  • start code emulation prevention is performed always regardless of whether the bytestream format is in use or not.
  • the picture parameter set contains data based on which each slice group map unit of a picture is associated with a particular slice group.
  • a slice group can contain any slice group map units, including non-adjacent map units.
  • the flexible macroblock ordering (FMO) feature of the standard is used.
  • a slice consists of one or more consecutive macroblocks (or macroblock pairs, when MBAFF is in use) within a particular slice group in raster scan order. If only one slice group is in use, H.264/AVC slices contain consecutive macroblocks in raster scan order and are therefore similar to the slices in many previous coding standards. In some profiles of H.264/AVC slices of a coded picture may appear in any order relative to each other in the bitstream, which is referred to as the arbitrary slice ordering (ASO) feature. Otherwise, slices must be in raster scan order in the bitstream.
  • ASO arbitrary slice ordering
  • NAL units consist of a header and payload.
  • the NAL unit header indicates the type of the NAL unit and whether a coded slice contained in the NAL unit is a part of a reference picture or a non-reference picture.
  • the NAL unit header indicates the type of the NAL unit and whether a coded slice contained in the NAL unit is a part of a reference picture or a non-reference picture.
  • H.264/AVC includes a 2-bit nal_ref_idc syntax element, which when equal to 0 indicates that a coded slice contained in the NAL unit is a part of a non-reference picture and when greater than 0 indicates that a coded slice contained in the NAL unit is a part of a reference picture.
  • a draft HEVC includes a 1-bit nal_ref_idc syntax element, also known as nal_ref_flag, which when equal to 0 indicates that a coded slice contained in the NAL unit is a part of a non-reference picture and when equal to 1 indicates that a coded slice contained in the NAL unit is a part of a reference picture.
  • the header for SVC and MVC NAL units additionally contains various indications related to the scalability and multiview hierarchy.
  • NAL units can be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units.
  • VCL Video Coding Layer
  • VCL NAL units are either coded slice NAL units, coded slice data partition NAL units, or VCL prefix NAL units.
  • Coded slice NAL units contain syntax elements representing one or more coded macroblocks, each of which corresponds to a block of samples in the uncompressed picture.
  • IDR Instantaneous Decoding Refresh
  • auxiliary coded picture such as an alpha plane
  • coded slice extension for SVC slices not in the base layer or MVC slices not in the base view.
  • a set of three coded slice data partition NAL units contains the same syntax elements as a coded slice.
  • Coded slice data partition A comprises macroblock headers and motion vectors of a slice
  • coded slice data partition B and C include the coded residual data for intra macroblocks and inter macroblocks, respectively. It is noted that the support for slice data partitions is only included in some profiles of H.264/AVC.
  • a VCL prefix NAL unit precedes a coded slice of the base layer in SVC and MVC bitstreams and contains indications of the scalability hierarchy of the associated coded slice.
  • coded slice NAL units contain syntax elements representing one or more CU.
  • a coded slice NAL unit can be indicated to be a coded slice in an Instantaneous Decoding Refresh (IDR) picture or coded slice in a non-IDR picture.
  • IDR Instantaneous Decoding Refresh
  • a coded slice NAL unit can be indicated to be a coded slice in a Clean Decoding Refresh (CDR) picture (which may also be referred to as a Clean Random Access picture).
  • CDR Clean Decoding Refresh
  • a non-VCL NAL unit may be for example of one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of stream NAL unit, or a filler data NAL unit.
  • SEI Supplemental Enhancement Information
  • Parameter sets are essential for the reconstruction of decoded pictures, whereas the other non-VCL NAL units are not necessary for the reconstruction of decoded sample values and serve other purposes presented below.
  • sequence parameter set Parameters that remain unchanged through a coded video sequence are included in a sequence parameter set.
  • the sequence parameter set may optionally contain video usability information (VUI), which includes parameters that are important for buffering, picture output timing, rendering, and resource reservation.
  • VUI video usability information
  • a picture parameter set contains such parameters that are likely to be unchanged in several coded pictures. No picture header is present in H.264/AVC bitstreams but the frequently changing picture-level data is repeated in each slice header and picture parameter sets carry the remaining picture-level parameters.
  • H.264/AVC syntax allows many instances of sequence and picture parameter sets, and each instance is identified with a unique identifier.
  • Each slice header includes the identifier of the picture parameter set that is active for the decoding of the picture that contains the slice, and each picture parameter set contains the identifier of the active sequence parameter set. Consequently, the transmission of picture and sequence parameter sets does not have to be accurately synchronized with the transmission of slices. Instead, it is sufficient that the active sequence and picture parameter sets are received at any moment before they are referenced, which allows transmission of parameter sets using a more reliable transmission mechanism compared to the protocols used for the slice data.
  • parameter sets can be included as a parameter in the session description for H.264/AVC Real-time Transport Protocol (RTP) sessions. If parameter sets are transmitted in-band, they can be repeated to improve error robustness.
  • RTP Real-time Transport Protocol
  • Adaptation Parameter Set includes parameters that are likely to be unchanged in several coded slices.
  • the APS syntax structure includes parameters or syntax elements related to context- based adaptive binary arithmetic coding (CABAC), adaptive sample offset, adaptive loop filtering, and deblocking filtering.
  • CABAC context- based adaptive binary arithmetic coding
  • an APS is a NAL unit and coded without reference or prediction from any other NAL unit.
  • An identifier, referred to as aps_id syntax element is included in APS NAL unit, and included and used in the slice header to refer to a particular APS.
  • An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but assist in related processes, such as picture output timing, rendering, error detection, error concealment, and resource reservation.
  • SEI messages are specified in H.264/AVC and HEVC, and the user data SEI messages enable organizations and companies to specify SEI messages for their own use.
  • H.264/AVC and HEVC contains the syntax and semantics for the specified SEI messages but no process for handling the messages in the recipient is defined. Consequently, encoders are required to follow the H.264/AVC or HEVC standard when they create SEI messages, and decoders conforming to the H.264/AVC or HEVC standard are not required to process SEI messages for output order conformance.
  • a coded picture in H.264/AVC consists of the VCL NAL units that are required for the decoding of the picture.
  • a coded picture can be a primary coded picture or a redundant coded picture.
  • a primary coded picture is used in the decoding process of valid bitstreams.
  • a redundant coded picture is a redundant representation that should only be decoded when the primary coded picture cannot be successfully decoded.
  • an access unit consists of a primary coded picture and those NAL units that are associated with it.
  • the appearance order of NAL units within an access unit is constrained as follows.
  • An optional access unit delimiter NAL unit may indicate the start of an access unit. It is followed by zero or more SEI NAL units.
  • the coded slices or slice data partitions of the primary coded picture appear next, followed by coded slices for zero or more redundant coded pictures.
  • An access unit in MVC is defined to be a set of NAL units that are consecutive in decoding order and contain exactly one primary coded picture consisting of one or more view components.
  • an access unit may also contain one or more redundant coded pictures, one auxiliary coded picture, or other NAL units not containing slices or slice data partitions of a coded picture.
  • the decoding of an access unit always results in one decoded picture consisting of one or more decoded view components.
  • an access unit in MVC contains the view components of the views for one output time instance.
  • a view component in MVC is referred to as a coded representation of a view in a single access unit.
  • An anchor picture is a coded picture in which all slices may reference only slices within the same access unit, i.e., inter-view prediction may be used, but no inter prediction is used, and all following coded pictures in output order do not use inter prediction from any picture prior to the coded picture in decoding order.
  • Inter-view prediction may be used for IDR view components that are part of a non-base view.
  • a base view in MVC is a view that has the minimum value of view order index in a coded video sequence. The base view can be decoded independently of other views and does not use inter-view prediction.
  • the base view can be decoded by H.264/AVC decoders supporting only the single-view profiles, such as the Baseline Profile or the High Profile of H.264/AVC.
  • a coded video sequence is defined to be a sequence of consecutive access units in decoding order from an IDR access unit, inclusive, to the next IDR access unit, exclusive, or to the end of the bitstream, whichever appears earlier.
  • a group of pictures is and its characteristics may be defined as follows.
  • a GOP can be decoded regardless of whether any previous pictures were decoded.
  • An open GOP is such a group of pictures in which pictures preceding the initial intra picture in output order might not be correctly decodable when the decoding starts from the initial intra picture of the open GOP.
  • pictures of an open GOP may refer (in inter prediction) to pictures belonging to a previous GOP.
  • An H.264/AVC decoder can recognize an intra picture starting an open GOP from the recovery point SEI message in an H.264/AVC bitstream.
  • a closed GOP is such a group of pictures in which all pictures can be correctly decoded when the decoding starts from the initial intra picture of the closed GOP.
  • no picture in a closed GOP refers to any pictures in previous GOPs.
  • a closed GOP starts from an IDR access unit.
  • closed GOP structure has more error resilience potential in comparison to the open GOP structure, however at the cost of possible reduction in the compression efficiency.
  • Open GOP coding structure is potentially more efficient in the compression, due to a larger flexibility in selection of reference pictures.
  • the bitstream syntax of H.264/AVC indicates whether a particular picture is a reference picture for inter prediction of any other picture.
  • Pictures of any coding type (I, P, B) can be reference pictures or non-reference pictures in H.264/AVC.
  • the NAL unit header indicates the type of the NAL unit and whether a coded slice contained in the NAL unit is a part of a reference picture or a non-reference picture.
  • pixel or sample values in a certain picture area or "block" are predicted. These pixel or sample values can be predicted, for example, by motion compensation mechanisms, which involve finding and indicating an area in one of the previously encoded video frames that corresponds closely to the block being coded. Additionally, pixel or sample values can be predicted by spatial mechanisms which involve finding and indicating a spatial region relationship.
  • Prediction approaches using image information from a previously coded image can also be called as inter prediction methods which may be also referred to as temporal prediction and motion compensation.
  • Prediction approaches using image information within the same image can also be called as intra prediction methods.
  • the second phase is one of coding the error between the predicted block of pixels or samples and the original block of pixels or samples. This may be accomplished by transforming the difference in pixel or sample values using a specified transform. This transform may be a Discrete Cosine Transform (DCT) or a variant thereof. After transforming the difference, the transformed difference is quantized and entropy encoded. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel or sample representation (i.e. the visual quality of the picture) and the size of the resulting encoded video representation (i.e. the file size or transmission bit rate).
  • DCT Discrete Cosine Transform
  • the decoder reconstructs the output video by applying a prediction mechanism similar to that used by the encoder in order to form a predicted representation of the pixel or sample blocks (using the motion or spatial information created by the encoder and stored in the compressed representation of the image) and prediction error decoding (the inverse operation of the prediction error coding to recover the quantized prediction error signal in the spatial domain).
  • the decoder After applying pixel or sample prediction and error decoding processes the decoder combines the prediction and the prediction error signals (the pixel or sample values) to form the output video frame.
  • the decoder (and encoder) may also apply additional filtering processes in order to improve the quality of the output video before passing it for display and/or storing as a prediction reference for the forthcoming pictures in the video sequence.
  • motion information is indicated by motion vectors associated with each motion compensated image block.
  • Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder) or decoded (at the decoder) and the prediction source block in one of the previously coded or decoded images (or pictures).
  • H.264/AVC and HEVC as many other video compression standards, divide a picture into a mesh of rectangles, for each of which a similar block in one of the reference pictures is indicated for inter prediction. The location of the prediction block is coded as motion vector that indicates the position of the prediction block compared to the block being coded. Inter prediction process may be characterized using one or more of the following factors.
  • motion vectors may be of quarter-pixel accuracy, and sample values in fractional-pixel positions are obtained using a finite impulse response (FIR) filter.
  • FIR finite impulse response
  • Block partitioning for inter prediction Many coding standards, including H.264/AVC and HEVC, allow selection of the size and shape of the block for which a motion vector is applied for motion-compensated in the encoder, and indicating the selected size and shape in the bitstream so that decoders can reproduce the motion-compensated prediction done in the encoder.
  • H.264/AVC and HEVC allow selection of the size and shape of the block for which a motion vector is applied for motion-compensated in the encoder, and indicating the selected size and shape in the bitstream so that decoders can reproduce the motion-compensated prediction done in the encoder.
  • a basic unit for inter prediction in many coding standards including H.264/AVC is a macroblock, corresponding to a 16x16 block of luma samples and corresponding chroma samples.
  • a macroblock can be further divided to 16x8, 8x16, or 8x8 macroblock partitions, and the 8x8 partition can be further divided to 4x4, 4x8, or 8x4 sub-macroblock partitions, and a motion vector is coded for each partition.
  • a block is used to refer to a unit for inter prediction, which may be of a different level in the partitioning structure.
  • a block in the following may refer to a macroblock, a macroblock partition or a sub- macroblock partition, whichever is used as a unit for inter prediction.
  • the sources of inter prediction are previously decoded pictures.
  • Many coding standards including H.264/AVC and HEVC, enable storage of multiple reference pictures for inter prediction and selection of the used reference picture on macroblock or macroblock partition basis.
  • Motion vector prediction In order to represent motion vectors efficiently in bitstreams, motion vectors may be coded differentially with respect to a block-specific predicted motion vector. In many video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Differential coding of motion vectors is typically disabled across slice boundaries. Multi-hypothesis motion-compensated prediction. H.264/AVC and HEVC enable the use of a single prediction block in P and SP slices (herein referred to as uni-predictive slices) or a linear combination of two motion- compensated prediction blocks for bi-predictive slices, which are also referred to as B slices.
  • Individual blocks in B slices may be bi-predicted, uni- predicted, or intra-predicted, and individual blocks in P or SP slices may be uni-predicted or intra-predicted.
  • the reference pictures for a bi-predictive picture are not limited to be the subsequent picture and the previous picture in output order, but rather any reference pictures can be used.
  • one reference picture list referred to as reference picture list 0 is constructed for P slices, and two reference picture lists, list 0 and list 1 , are constructed for B slices.
  • prediction in forward direction may refer to predicting from a reference picture in reference picture list 0
  • prediction in backward direction may refer to predicting from a reference picture in reference picture list 1
  • the reference pictures for prediction may have any decoding or output order relation to each other or to the current picture.
  • Weighted prediction Many coding standards use a prediction weight of 1 for prediction blocks of inter (P) pictures and 0.5 for each prediction block of a B picture (resulting into averaging). Many coding standards, such as H.264/AVC and HEVC, allow weighted prediction for both P and B slices. In implicit weighted prediction, the weights are proportional to picture order counts, while in explicit weighted prediction, prediction weights are explicitly indicated. In many video codecs, the prediction residual after motion compensation is first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding.
  • a transform kernel like DCT
  • each PU has prediction information associated with it defining what kind of a prediction is to be applied for the pixels within that PU (e.g. motion vector information for inter predicted PUs and intra prediction directionality information for intra predicted PUs).
  • each TU is associated with information describing the prediction error decoding process for the samples within the said TU (including e.g. DCT coefficient information). It is typically signaled at CU level whether prediction error coding is applied or not for each CU. In the case there is no prediction error residual associated with the CU, it can be considered there are no TUs for the said CU.
  • the prediction weight may be scaled according to the POC difference between the POC of the current picture and the POC of the reference picture.
  • the prediction weight may be scaled according to the POC difference between the POC of the current picture and the POC of the reference picture.
  • the a default prediction weight may be used, such as 0.5 in implicit weighted prediction for bi-predicted blocks.
  • H.264/AVC specifies the process for decoded reference picture marking in order to control the memory consumption in the decoder.
  • the maximum number of reference pictures used for inter prediction referred to as M, is determined in the sequence parameter set.
  • M the maximum number of reference pictures used for inter prediction
  • a reference picture is decoded, it is marked as "used for reference”. If the decoding of the reference picture caused more than M pictures marked as "used for reference”, at least one picture is marked as "unused for reference”.
  • the operation mode for decoded reference picture marking is selected on picture basis.
  • the adaptive memory control enables explicit signaling which pictures are marked as "unused for reference” and may also assign long-term indices to short-term reference pictures.
  • the adaptive memory control requires the presence of memory management control operation (MMCO) parameters in the bitstream. If the sliding window operation mode is in use and there are M pictures marked as "used for reference", the short-term reference picture that was the first decoded picture among those short-term reference pictures that are marked as "used for reference” is marked as "unused for reference”. In other words, the sliding window operation mode results into first-in-first-out buffering operation among short-term reference pictures.
  • MMCO memory management control operation
  • IDR instantaneous decoding refresh
  • a reference picture set valid or active for a picture includes all the reference pictures used as reference for the picture and all the reference pictures that are kept marked as "used for reference” for any subsequent pictures in decoding order.
  • “Curr” refers to the reference pictures that are included in the reference picture lists of the current picture and hence may be used as inter prediction reference for the current picture.
  • “Foil” refers to reference pictures that are not included in the reference picture lists of the current picture but may be used in subsequent pictures in decoding order as reference pictures.
  • “St” refers to short-term reference pictures, which may generally be identified through a certain number of least significant bits of their POC value.
  • “Lt” refers to long-term reference pictures, which are specifically identified and generally have a greater difference of POC values relative to the current picture than what can be represented by the mentioned certain number of least significant bits.
  • RefPicSetStCurrO, RefPicSetStCurrl , RefPicSetStFollO and RefPicSetStFolM are collectively referred to as the short-term subset of the reference picture set.
  • RefPicSetLtCurr and RefPicSetLtFoll are collectively referred to as the long- term subset of the reference picture set.
  • a reference picture set may be specified in a picture parameter set and taken into use in the slice header through an index to the reference picture set.
  • a reference picture set may also be specified in a slice header.
  • a long-term subset of a reference picture set is generally specified only in a slice header, while the short-term subsets of the same reference picture set may be specified in the picture parameter set or slice header.
  • Pictures that are included in the reference picture set used by the current slice are marked as "used for reference", and pictures that are not in the reference picture set used by the current slice are marked as "unused for reference”.
  • a Decoded Picture Buffer may be used in the encoder and/or in the decoder. There are two reasons to buffer decoded pictures, for references in inter prediction and for reordering decoded pictures into output order. As H.264/AVC provides a great deal of flexibility for both reference picture marking and output reordering, separate buffers for reference picture buffering and output picture buffering may waste memory resources. Hence, the DPB may include a unified decoded picture buffering process for reference pictures and output reordering. A decoded picture may be removed from the DPB when it is no longer used as reference and needed for output.
  • the reference picture for inter prediction is indicated with an index to a reference picture list.
  • the index is coded with variable length coding, i.e., the smaller the index is, the shorter the corresponding syntax element becomes.
  • Typical high efficiency video codecs such as a draft HEVC codec employ an additional motion information coding/decoding mechanism, often called merging/merge mode/process/mechanism, where all the motion information of a block/PU is predicted and used without any modification/correction.
  • the aforementioned motion information for a PU comprises 1 ) The information whether 'the PU is uni-predicted using only reference picture listO' or 'the PU is uni-predicted using only reference picture Iist1 ' or 'the PU is bi-predicted using both reference picture listO and Iist1 ' 2) Motion vector value corresponding to the reference picture listO 3) Reference picture index in the reference picture listO 4) Motion vector value corresponding to the reference picture list 1 5) Reference picture index in the reference picture list 1 .
  • predicting the motion information is carried out using the motion information of adjacent blocks and/or co-located blocks in temporal reference pictures.
  • a list is constructed by including motion prediction candidates associated with available adjacent/co-located blocks and the index of selected motion prediction candidate in the list is signalled. Then the motion information of the selected candidate is copied to the motion information of the current PU.
  • this type of coding/decoding the CU is typically named as skip mode or merge based skip mode.
  • the merge mechanism is also employed for individual PUs (not necessarily the whole CU as in skip mode) and in this case, prediction residual may be utilized to improve prediction quality.
  • This type of prediction mode is typically named as inter-merge mode.
  • a reference picture list is constructed in two steps: first, an initial reference picture list is generated.
  • the initial reference picture list may be generated for example on the basis of frame_num, POC, temporaljd, and/or reference picture set.
  • the initial reference picture list may be reordered by reference picture list reordering (RPLR) commands contained in slice headers.
  • RPLR reference picture list reordering
  • the RPLR commands indicate the pictures that are ordered to the beginning of the respective reference picture list.
  • the reference picture list 0 may be initialized to contain RefPicSetStCurrO first, followed by RefPicSetStCurrl , followed by RefPicSetLtCurr.
  • Reference picture list 1 may be initialized to contain RefPicSetStCurrl first, followed by RefPicSetStCurrO.
  • the initial reference picture lists may be modified through the reference picture list modification syntax structure, where pictures in the initial reference picture lists may be identified through an entry index to the list.
  • the merge list may be generated on the basis of reference picture list 0 and/or reference picture list 1 for example using the reference picture lists combination syntax structure included in the slice header syntax.
  • There may be a reference picture lists combination syntax structure, created into the bitstream by an encoder and decoded from the bitstream by a decoder, which indicates the contents of the merge list.
  • the syntax structure may indicate that the reference picture list 0 and the reference picture list 1 are combined to be an additional reference picture lists combination used for the prediction units being uni-directional predicted.
  • the syntax structure may include a flag which, when equal to a certain value, indicates that the reference picture list 0 and reference picture list 1 are identical thus reference picture list 0 is used as the reference picture lists combination.
  • the syntax structure may include a list of entries, each specifying a reference picture list (list 0 or list 1 ) and a reference index to the specified list, where an entry specifies a reference picture to be included in the merge list.
  • the frame_num syntax element is used for various decoding processes related to multiple reference pictures.
  • the value of frame_num for IDR pictures is 0.
  • the value of frame_num for non-IDR pictures is equal to the frame_num of the previous reference picture in decoding order incremented by 1 (in modulo arithmetic, i.e., the value of frame_num wrap over to 0 after a maximum value of frame_num).
  • a value of picture order count is derived for each picture and is non-decreasing with increasing picture position in output order relative to the previous IDR picture or a picture containing a memory management control operation marking all pictures as "unused for reference".
  • POC therefore indicates the output order of pictures. It is also used in the decoding process for implicit scaling of motion vectors in the temporal direct mode of bi-predictive slices, for implicitly derived weights in weighted prediction, and for reference picture list initialization of B slices. Furthermore, POC is used in the verification of output order conformance.
  • view dependencies are specified in the sequence parameter set (SPS) MVC extension.
  • the dependencies for anchor pictures and non- anchor pictures are independently specified. Therefore anchor pictures and non-anchor pictures can have different view dependencies.
  • all the anchor pictures have the same view dependency, and all the non-anchor pictures have the same view dependency.
  • dependent views are signaled separately for the views used as reference pictures in reference picture list 0 and for the views used as reference pictures in reference picture list 1 .
  • inter_view_flag in the network abstraction layer (NAL) unit header which indicates whether the current picture is not used or is allowed to be used for inter-view prediction for the pictures in other views.
  • NAL network abstraction layer
  • inter-view prediction is supported by texture prediction (i.e., the reconstructed sample values may be used for inter-view prediction), and only the decoded view components of the same output time instance (i.e., the same access unit) as the current view component are used for inter-view prediction.
  • texture prediction i.e., the reconstructed sample values may be used for inter-view prediction
  • decoded view components of the same output time instance i.e., the same access unit
  • MVC utilizes multi-loop decoding.
  • motion compensation and decoded view component reconstruction are performed for each view.
  • an initial reference picture list is generated in two steps: i) An initial reference picture list is constructed including all the short-term and long-term reference pictures that are marked as "used for reference” and belong to the same view as the current slice as done in H.264/AVC. Those short-term and long-term reference pictures are named intra-view references for simplicity, ii) Then, inter-view reference pictures and inter-view only reference pictures are appended after the intra-view references, according to the view dependency order indicated in the active SPS and the "inter_view_flag" to form an initial reference picture list.
  • the initial reference picture list may be reordered by reference picture list reordering (RPLR) commands which may be included in a slice header.
  • RPLR reference picture list reordering
  • the RPLR process may reorder the intra-view reference pictures, inter-view reference pictures and inter-view only reference pictures into a different order than the order in the initial list.
  • Both the initial list and final list after reordering must contain only a certain number of entries indicated by a syntax element in the slice header or the picture parameter set referred by the slice.
  • a texture view refers to a view that represents ordinary video content, for example has been captured using an ordinary camera, and is usually suitable for rendering on a display.
  • Depth-enhanced video refers to texture video having one or more views associated with depth video having one or more depth views.
  • a number of approaches may be used for representing of depth-enhanced video, including the use of video plus depth (V+D), multiview video plus depth (MVD), and layered depth video (LDV).
  • V+D video plus depth
  • MVD multiview video plus depth
  • LDV layered depth video
  • V+D video plus depth
  • V+D a single view of texture and the respective view of depth are represented as sequences of texture picture and depth pictures, respectively.
  • the MVD representation contains a number of texture views and respective depth views.
  • the texture and depth of the central view are represented conventionally, while the texture and depth of the other views are partially represented and cover only the dis-occluded areas required for correct view synthesis of intermediate views.
  • Depth-enhanced video may be coded in a manner where texture and depth are coded independently of each other.
  • texture views may be coded as one MVC bitstream and depth views may be coded as another MVC bitstream.
  • depth-enhanced video may be coded in a manner where texture and depth are jointly coded.
  • some decoded samples of a texture picture or data elements for decoding of a texture picture are predicted or derived from some decoded samples of a depth picture or data elements obtained in the decoding process of a depth picture.
  • some decoded samples of a depth picture or data elements for decoding of a depth picture are predicted or derived from some decoded samples of a texture picture or data elements obtained in the decoding process of a texture picture.
  • VSP view synthesis prediction
  • a prediction signal such as a VSP reference picture
  • DIBR view synthesis prediction
  • a synthesized picture i.e., VSP reference picture
  • a specific VSP prediction mode for certain prediction blocks may be determined by the encoder, indicated in the bitstream by the encoder, and used as concluded from the bitstream by the decoder.
  • inter prediction and inter-view prediction use essentially the same motion-compensated prediction process.
  • Inter-view reference pictures and inter-view only reference pictures are essentially treated as long-term reference pictures in the different prediction processes.
  • view synthesis prediction may be realized such a manner that it uses the essentially the same motion-compensated prediction process as inter prediction and inter-view prediction.
  • motion-compensated prediction that includes and is capable of flexibly selecting mixing inter prediction, inter-prediction, and/or view synthesis prediction is herein referred to as mixed-direction motion-compensated prediction.
  • reference picture lists in MVC and MVD may contain more than one type of reference pictures, i.e. inter reference pictures (also known as intra-view reference pictures), inter-view reference pictures, inter-view only reference pictures, and VSP reference pictures, a term prediction direction is defined to indicate the use of intra-view reference pictures (temporal prediction), interview prediction, or VSP.
  • inter reference pictures also known as intra-view reference pictures
  • inter-view reference pictures also known as intra-view reference pictures
  • inter-view only reference pictures inter-view only reference pictures
  • VSP reference pictures a term prediction direction is defined to indicate the use of intra-view reference pictures (temporal prediction), interview prediction, or VSP.
  • an encoder may choose for a specific block a reference index that points to an inter-view reference picture, thus the prediction direction of the block is inter-view.
  • Motion vector (MV) prediction specified in H.264/AVC/MVC utilizes correlation which is present in neighboring blocks of the same image (spatial correlation) or in the previously coded image (temporal correlation).
  • motion vector components MVd(x) and MVd(y) for P macroblocks or macroblock partitions are differentially coded using either median or directional prediction from spatially neighboring blocks, wherein a neighboring block may be a macroblock, macroblock partition or a sub-macroblock partition.
  • Figure 13 illustrates labeling of spatially neighboring blocks for the current block cb.
  • blocks A to D may be used as motion vector prediction sources for various prediction modes.
  • other neighboring blocks, such as block E may be used as as motion vector prediction source.
  • Directional motion vector prediction is used for certain shapes of macroblock partitions, namely 8x16 and 16x8, when the reference index of cb is the same as in certain spatially neighboring block determined by the shape of the current macroblock partition and its location within the current macroblock.
  • the motion vector of certain block is taken as the motion vector predictor for the current macroblock partition.
  • Figure 13 Please refer to Figure 13 for an illustration of the location of neighboring blocks A to C relative to the current block cb.
  • the reference indices of the neighboring blocks immediately above (block B), diagonally above and to the right (block C), and immediately left (block A) of the current block cb are first compared to the reference index of cb. If one and only one of the reference indices of blocks A, B, and C is equal to the reference index of cb, then the motion vector predictor for cb is equal to the motion vector of the block A, B, or C for which the reference index is equal to the reference index of cb.
  • the motion vector predictor for cb is derived as a median value of the motion vectors of blocks A, B, and C regardless of their reference index value.
  • Figure 7a shows how blocks A, B, and C are spatially related to the currently coded block (cb).
  • the median motion vector prediction process for deriving MVp is specified in H.264/AVC as follows:
  • MVp median ⁇ mvA,mvB,mvC ⁇ , where mvA, mvB, mvC are motion vectors (without reference index) of the spatially neighboring blocks.
  • the motion vectors and reference indexes for the neighboring blocks A, B, C of the current block cb are determined based on their availability as follows. If the top-right (C) block is not available, for example when it is in a different slice than cb or outside picture boundaries, the location on the top-right (C) is replaced with the top-left block (D).
  • the motion vector mvA, mvB, or mvC, respectively is equal to 0 and the reference index for block A, B, or C, respectively is -1 for motion vector prediction and mixed-direction motion- compensated prediction.
  • a P macroblock may also be coded in the so-called P_Skip type in H.264/AVC.
  • P_Skip type in H.264/AVC.
  • no differential motion vector, reference index, or quantized prediction error signal is coded into the bitstream.
  • the reference picture of a macroblock coded with the P_Skip type has index 0 in reference picture list 0.
  • the motion vector used for reconstructing the P_Skip macroblock is obtained using median motion vector prediction for the macroblock without any differential motion vector being added.
  • P_Skip may be beneficial for compression efficiency particularly in areas where the motion field is smooth.
  • B slices of H.264/AVC four different types of inter prediction are supported: uni-predictive from reference picture list 0, uni-directional from reference picture list 1 , bi-predictive, direct prediction, and B_skip.
  • the type of inter prediction can be selected separately for each macroblock partition.
  • B slices utilize a similar macroblock partitioning as P slices.
  • the prediction signal is formed by a weighted average of motion-compensated list 0 and list 1 prediction signals.
  • Reference indices, motion vector differences, as well as quantized prediction error signal may be coded for uni-predictive and bi-predictive B macroblock partitions.
  • Two direct modes are included in H.264/AVC, temporal direct and spatial direct, and one of them can be selected into use for a slice in a slice header.
  • the reference index for reference picture list 1 is set to 0 and the reference index for reference picture list 0 is set to point to the reference picture that is used in the co-located block (compared to cb) of the reference picture having index 0 in the reference picture list 1 if that reference picture is available, or set to 0 if that reference picture is not available.
  • the motion vector predictor for cb is essentially derived by considering the motion information within a co-located block of the reference picture having index 0 in reference picture list 1.
  • Motion vector predictors for a temporal direct block are derived by scaling a motion vector from the co- located block where the scaling weight is proportional to picture order count differences between the current picture and the reference pictures associated with the inferred reference indexes in list 0 and list 1 , and by selecting the sign for the motion vector predictor depending on which reference picture list it is using.
  • Figure 7b shows an example illustration of co-located blocks of the currently coded block (cb) for MVP of the temporal direct mode of H.264/AVC.
  • the use of temporal direct mode is not supported in any present coding profile for MVC, even though the syntax of the standard supports also the temporal direct mode.
  • motion vector prediction in spatial direct mode can be divided into three steps: reference index determination, determination of uni- or bi-prediction, and motion vector prediction.
  • the reference picture with the minimum non- negative reference index i.e., non-intra block
  • reference index 0 is selected for both reference picture lists.
  • uni- or bi-prediction for H.264/AVC spatial direct mode is determined as follows: If a minimum non-negative reference index for both reference picture lists was found in the reference index determination step, bi-prediction is used. If a minimum non-negative reference index for either but not both of reference picture list 0 or reference picture list 1 was found in the reference index determination step, uni-prediction from either reference picture list 0 or reference picture list 1 , respectively, is used.
  • the motion vector predictor is derived similarly to the motion vector predictor of P blocks using the motion vectors of spatially adjacent blocks A, B, and C.
  • bitstream for a direct mode block No motion vector differences or reference indices are present in the bitstream for a direct mode block, while quantized prediction error signal may be coded and present therefore present in the bitstream.
  • a B_skip macroblock mode is similar to the direct mode but no prediction error signal is coded and included in the bitstream.
  • Lagrangian cost function uses a weighting factor or ⁇ to tie together the exact or estimated image distortion due to lossy coding methods and the exact or estimated amount of information required to represent the pixel/sample values in an image area.
  • the Lagrangian cost function may be represented by the equation:
  • C D+AR
  • D the image distortion (for example, the mean-squared error between the pixel/sample values in original image block and in coded image block) with the mode and motion vectors currently considered
  • ⁇ is a Lagrangian coefficient
  • R is the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).
  • the Multiview Video Coding (MVC) extension of H.264 referred above enables to implement a multiview functionality at the decoder, thereby allowing the development of three-dimensional (3D) multiview applications.
  • MVC Multiview Video Coding
  • Stereoscopic video content consists of pairs of offset images that are shown separately to the left and right eye of the viewer. These offset images are captured with a specific stereoscopic camera setup and it assumes a particular stereo baseline distance between cameras.
  • FIG. 1 shows a simplified 2D model of such stereoscopic camera setup.
  • C1 and C2 refer to cameras of the stereoscopic camera setup, more particularly to the center locations of the cameras, b is the distance between the centers of the two cameras (i.e. the stereo baseline), f is the focal length of cameras and X is an object in the real 3D scene that is being captured.
  • the real world object X is projected to different locations in images captured by the cameras C1 and C2, these locations being x1 and x2 respectively.
  • the horizontal distance between x1 and x2 in absolute coordinates of the image is called disparity.
  • the images that are captured by the camera setup are called stereoscopic images, and the disparity presented in these images creates or enhances the illusion of depth.
  • Adaptation of the disparity is a key feature for adjusting the stereoscopic video content to be comfortably viewable on various displays.
  • FIG. 2 shows a simplified model of such multiview camera setup that suits to this solution. This setup is able to provide stereoscopic video content captured with several discrete values for stereoscopic baseline and thus allow stereoscopic display to select a pair of cameras that suits to the viewing conditions.
  • a more advanced approach for 3D vision is having a multiview autostereoscopic display (ASD) that does not require glasses.
  • the ASD emits more than one view at a time but the emitting is localized in the space in such a way that a viewer sees only a stereo pair from a specific viewpoint, as illustrated in Figure 3, wherein the boat is seen in the middle of the view when looked at the right-most viewpoint. Moreover, the viewer is able see another stereo pair from a different viewpoint, e.g. in Fig. 3 the boat is seen at the right border of the view when looked at the left-most viewpoint.
  • the ASD technologies may be capable of showing for example 52 or more different images at the same time, of which only a stereo pair is visible from a specific viewpoint. This supports multiuser 3D vision without glasses, for example in a living room environment.
  • the above-described stereoscopic and ASD applications require multiview video to be available at the display.
  • the MVC extension of H.264/AVC video coding standard allows the multiview functionality at the decoder side.
  • the base view of MVC bitstreams can be decoded by any H.264/AVC decoder, which facilitates introduction of stereoscopic and multiview content into existing services.
  • MVC allows inter-view prediction, which can result into significant bitrate saving compared to independent coding of all views, depending on how correlated the adjacent views are.
  • the rate of MVC coded video is proportional to the number of views. Considering that ASD may require 52 views, for example, as input, the total bitrate for such number of views will challenge the constraints of the available bandwidth.
  • DIBR depth image-based rendering
  • DIBR algorithms may also enable extrapolation of views that are outside the two input views and not in between them. Similarly, DIBR algorithms may enable view synthesis from a single view of texture and the respective depth view.
  • texture data should be available at the decoder side along with the corresponding depth data.
  • depth information is produced at the encoder side in a form of depth pictures (also known as depth maps) for each video frame.
  • a depth map is an image with per-pixel depth information.
  • Each sample in a depth map represents the distance of the respective texture sample from the plane on which the camera lies. In other words, if the z axis is along the shooting axis of the cameras (and hence orthogonal to the plane on which the cameras lie), a sample in a depth map represents the value on the z axis.
  • Depth information can be obtained by various means. For example, depth of the 3D scene may be computed from the disparity registered by capturing cameras.
  • a depth estimation algorithm takes a stereoscopic view as an input and computes local disparities between the two offset images of the view. Each image is processed pixel by pixel in overlapping blocks, and for each block of pixels a horizontally localized search for a matching block in the offset image is performed. Once a pixel-wise disparity is computed, the corresponding depth value z is calculated by equation (1 ):
  • f is the focal length of the camera and b is the baseline distance between cameras, as shown in Figure 1 .
  • D refers to the disparity observed between the two cameras
  • Ad reflects a possible horizontal misplacement of the optical centers of the two cameras.
  • the algorithm is based on block matching, the quality of a depth-through-disparity estimation is content dependent and very often not accurate. For example, no straightforward solution for depth estimation is possible for image fragments that are featuring very smooth areas with no textures or large level of noise.
  • the depth value may be obtained using the time-of-flight (TOF) principle.
  • FIGs 5 and 6 show an example of a TOF-based depth estimation system.
  • the camera is provided with a light source, for example an infrared emitter, for illuminating the scene.
  • a light source for example an infrared emitter
  • Such an illuminator is arranged to produce an intensity modulated electromagnetic emission for a frequency between 10-100 MHz, which typically requires LEDs or laser diodes to be used.
  • Infrared light is typically used to make the illumination unobtrusive.
  • the light reflected from objects in the scene is detected by an image sensor, which is modulated synchronously at the same frequency as the illuminator.
  • the image sensor is provided with optics; a lens gathering the reflected light and an optical bandpass filter for passing only the light with the same wavelength as the illuminator, thus helping to suppress background light.
  • the image sensor measures for each pixel the time the light has taken to travel from the illuminator to the object and back.
  • the distance to the object is represented as a phase shift in the illumination modulation, which can be determined from the sampled data simultaneously for each pixel in the scene.
  • the accuracy of the TOF- based depth estimation is mostly content independent. For example, it is not suffering from the lack of textural appearance in the content.
  • currently available TOF cameras have low pixel resolution sensors and the depth estimation is heavily influenced by random and systematic noise.
  • Disparity or parallax maps such as parallax maps specified in ISO/IEC International Standard 23002-3, may be processed similarly to depth maps. Depth and disparity have a straightforward correspondence and they can be computed from each other through mathematical equation.
  • the 3DV systems described above typically use mixed-direction motion-compensated prediction including interview and/or view synthesis prediction.
  • the motion vector prediction specified in H.264/AVC/MVC may be suboptimal for video coding systems utilizing inter-view and/or view synthesis prediction (VSP).
  • VSP inter-view and/or view synthesis prediction
  • MVP motion vector prediction
  • MVC multi-view coding
  • MVD multiview+depth
  • MVP-VSP multi-view with in-loop view synthesis
  • Figure 14 illustrates the association of depth/disparity blocks d(cb), d(S), d(T), and d(U) with the texture blocks cb, S, T, and U respectively.
  • a depth/disparity block for or associated with a texture block co-locates with the texture block.
  • the spatial location of the depth/disparity block in relation to the depth/disparity frame it resides is the same as the spatial location of the respective texture block in relation to the depth/disparity frame it resides.
  • the spatial extents i.e. the number of samples, may differ in the depth/disparity frame compared to those of the luma texture frame.
  • the location of the blocks may be normalized as if both frames had the same spatial extents for example through scaling the coordinates of the block and/or resampling of one or both of the frames.
  • the association of a depth/disparity block and a respective texture block may also be derived through other criterion than spatial co-location.
  • the depth or disparity information (Di) for a current block (cb) or for previously coded/decoded texture data is available through decoding of coded depth or disparity information or can be estimated at the decoder side prior to decoding of the current texture block, and this information can be utilized in MVP.
  • An image fragment or a 2D fragment or a 2D fragment of an image is defined as a subset of an image.
  • An image fragment is typically, but needs not be, rectangular in shape.
  • a video fragment is defined as a subset of a video.
  • a video fragment may be a temporal video fragment, consisting of a subset of the frames or pictures of a single-view video sequence.
  • a video fragment may be a spatio-temporal video fragment consisting of image fragments in a subset of the frames or picture of a single-view video sequence.
  • a video fragment may be a multiview video fragment consisting of frames/pictures or image fragments from different views of a single access unit (i.e. sampling instant) of a multiview video sequence. Other combinations are also possible to form a video fragment, such as spatio-temporal multiview video fragment.
  • Figure 15 shows a currently coded/decoded image of texture data, currently coded block cb of texture data, and a fragment of the image (shown in grey) of texture data which is spatially adjacent or located in proximity of cb and which is assumed to be available (decoded) prior to coding/decoding of cb block.
  • 2D blocks of texture data called “adjacent blocks" A,B,C, D, and F shown as white blocks, and which are assumed to be available (decoded) prior to coding/decoding of cb can utilized by MVP process for coding/decoding of texture block cb. It is noted that blocks A,B,C, D, and F in Figure 15 are merely examples of selection of adjacent blocks and other blocks may have been selected as well and/or alternatively.
  • a Euclidean vector may be represented by a line segment having a definite direction and connecting an initial point with a terminal point.
  • Motion information e.g. horizontal and vertical motion vector components mv_x and mv_y and a reference index
  • MV(cb) motion information for block cb, MV(cb)
  • the prediction block is denoted (cb,MV(cb)). If motion information for block M is applied for motion- compensated prediction at the location of block N, the prediction block is denoted (N,MV(M)).
  • Figure 16 shows a concept of adjacent blocks of texture data which is extended to video and multiview video applications.
  • a currently coded image of texture data cb is shown in dark grey block, adjacent image fragments (shown in light grey of several images of texture data) are assumed to be available (coded/decoded) prior to coding of cb block.
  • adjacent image fragments may belong to previously coded/decoded images of the same view (2D video coding), or may belong to previously coded images of another view (multiview video coding).
  • Correspondence between cb and adjacent blocks may be established in different ways, below are just few examples of such.
  • Texture blocks A and B are spatial neighbors of the cb in currently coded image
  • texture block D is located in the previously coded/decoded image at the same spatial coordinates (x,y) as cb
  • texture blocks E, F and G which are located at different images are associated with blocks A,B, C through their motion information MV(A), MV(B) and MV(C) respectively.
  • block E may be equal to (cb,MV(C))
  • block F may be equal to (cb,MV(B)
  • block G may be equal to (cb,MV(A)).
  • block E may be equal to (C,MV(C))
  • block F may be equal to mcp(B,MV(B)
  • block G may be equal to (A,MV(A)).
  • the coding order of texture and depth view components with an access unit is such that the texture and depth view components of the base view are coded first in any order. Then, the depth view component of a non-base view is coded before the texture view component of the same non-base view.
  • the depth view components are coded in their inter-view dependency order, and the texture view components are likewise coded in their inter-view dependency order.
  • These three texture and depth views may be coded for example in order TO DO D1 T1 D2 T2 or DO D1 D2 TO T1 T2 or DO TO D1 T1 D2 T2.
  • the bitstream order of the coded view components may be the same as their coding order, and likewise the decoding order of the coded view components may be the same as the bitstream order.
  • the inter-view dependency order of depth is typically, but needs not be, the same as the inter-view dependency order for texture.
  • the number of depth views may differ from that of texture views. For example, no depth view may be coded for a texture view from which no other texture view is predicted. Slices of depth view components may be differentiated from slices of texture view components for example using a different NAL unit type value.
  • the coding/decoding order of texture and depth may be interleaved using smaller units than view components, such as on block or slice basis.
  • the respective coding/decoding order of coded texture and depth units, such as blocks, may follow the ordering rules described in the previous paragraph. For example, there may be two spatially adjacent texture blocks, ta and tb, where tb follows ta in coding/decoding order, and two depth/disparity blocks, da and db, spatially co-located with ta and tb, respectively.
  • the coding/decoding order of the blocks may be (da, ta, db, tb) or (da, db, ta, tb).
  • the bitstream order of the blocks may be the same as their coding order.
  • the coding/decoding order of texture and depth may be interleaved using greater units than view components, such as group of pictures or coded video sequences.
  • a coded block of texture data (cb_t) can be pixel-wise associated with a block of depth/disparity data (cb_d). The latter can be utilized in the proposed MVP chains without modifications.
  • the actual disparity information instead of the depth map data the actual disparity information or at least an estimate of it may be required.
  • a conversion from the depth map data to disparity information may be required. The conversion may be performed as following:
  • d is a depth map value
  • z is the actual depth value
  • D is the resulting disparity.
  • the parameters f, b, Z ne ar and Z far derived from the camera setup; i.e. the used focal length (f), camera separation (b) and depth range (Z ne ar,Zfar) respectively.
  • the disparity information may be estimated from the available textures at the encoder/decoder sides through a block matching procedure or any other means. Depth and/or disparity parameters of a block
  • the pixels of a coded block of texture can be associated with a block of depth information (cb_d) for each of said pixels.
  • the depth/disparity information can be aggregatively presented through average depth/disparity values for cb_d and deviation (e.g. variance) of cb_d.
  • the average Av(cb_d) depth/disparity value for a block of depth information cb_d is computed as:
  • Av(cb_d) sum(cb_d(x,y))/num_pixels, (3) where x and y are coordinates of the pixels in cb_d, and num_pixels is number of pixels within cb_d, and function sum adds up all the sample/pixel values in the given block, i.e. function sum(block(x,y)) computes a sum of samples values within the given block for all values of x and y corresponding to the horizontal and vertical extents of the block.
  • the deviation Dev(cb_d) of the depth/disparity values within a block of depth information cb_d can be computed as:
  • Dev(cb_d) sum(abs(cb_d(x,y) - Av(cb_d)))/num_pixels (4) where function abs returns the absolute value of the value given as input.
  • an application-specific predefined threshold T1 can be defined such that:
  • cb_d if the deviation of the depth/disparity values within a block of depth information cb_d is less than or equal than the threshold T1 , such cb_d block can be considered as homogenous.
  • the coded block of texture data cb can be compared to its neighboring blocks nb through their depth/disparity information.
  • the selection of the neighboring block nb may be determined for example based on the coding mode of cb.
  • Equation (6) where x and y are coordinates of the pixels in cb_d, and in its neighboring depth/disparity block (nb_d), num_pixels is number of pixels within cb_d and functions sum and abs are defined above. Equation (6) may also be regarded as a sum of absolute differences (SAD) between the given blocks normalized by the number of pixels in the block.
  • SAD sum of absolute differences
  • the similarity of two depth/disparity blocks is compared.
  • the similarity may be compared for example using equation (6) but any other similarity or distortion metric may also be used.
  • a sum of squared differences (SSD) normalized by the number of pixels may be used as computed in equation (7):
  • nsse(cb_d, nb_d) sum( (cb_d(x,y) - nb_d(x,y)) A 2 ) / num_pixels (7)
  • x and y are coordinates of the pixels in cb_d and in its neighboring depth/disparity block (nb_d)
  • num_pixels is number of pixels within cb_d
  • notation ⁇ 2 indicates a power of two
  • function sum is defined above.
  • a sum of transformed differences may be used as a similarity or distortion metric.
  • Both the current depth/disparity block cb_d and a neighboring depth/disparity block nb_d are transformed using for example DCT or a variant thereof, herein marked as function T().
  • T() a difference between the current depth/disparity block cb_d and a neighboring depth/disparity block nb_d
  • T() a sum of transformed differences
  • nsatd(cb_d, nb_d) sum( abs(tcb_d(x,y) - tnb_d(x,y)) ) / num_pixels (8)
  • any similarity/distortion metric could be added to the definition of function diff in equation (9).
  • the used similarity/distortion metric is pre-defined and therefore stays the same in both the encoder and the decoder.
  • the used similarity/distortion metric is determined by the encoder, for example using rate-distortion optimization, and encoded in the bitstream as an indication.
  • the indication of the used similarity/distortion metric may be included for example in a sequence parameter set, a picture parameter set, a slice parameter set, a picture header, a slice header, within a macroblock syntax structure, or anything alike.
  • the indicated similarity/distortion metric may be used in pre-determined operations in both the encoding and the decoding loop, such as depth/disparity based motion vector prediction.
  • the decoding processes for which the indicated similarity/distortion metric is indicated are also indicated in the bitstream for example in a sequence parameter set, a picture parameter set, a slice parameter set, a picture header, a slice header, within a macroblock syntax structure, or anything alike.
  • the encoder may select which similarity/distortion metric is used for each particular decoding process where a similarity/distortion based selection or other processing is used, such as depth/disparity based motion vector prediction, and encode respective indications of the selected disparity/distortion metrics and to which decoding processes they apply to into the bitstream.
  • a similarity/distortion based selection or other processing such as depth/disparity based motion vector prediction
  • both a video encoder and a video decoder typically apply a prediction mechanism, hence the followings elements may apply similarly to both a video encoder and a video decoder.
  • the reference picture of a macroblock coded with the P_Skip type has index 0 in reference picture list 0.
  • Such reference index selection equal to 0 is herein referred to as a "zero-value" reference index prediction.
  • the reference index(es) for a direct mode block and for a B_skip block in H.264/AVC are selected based on the minimal available non-negative reference index in the neighboring blocks.
  • the reference index(es) for skip and/or direct modes and /or alike may be selected based on the similarity of the depth/disparity of the current block and the depth/disparity of neighboring blocks. For example, the reference index of the neighboring block that has the smallest depth/disparity deviation compared to the current block may be selected as the reference index of the current block.
  • the directional and/or median motion vector prediction and/or alike is applied only among the motion vectors of those neighboring blocks that have the same reference index as the current block or additionally have sufficient depth/disparity similarity compared to the depth/disparity of the current block.
  • the motion vector predictor For each cb block that utilizes inter-view prediction, i.e. uses a reference index that points to an inter-view (only) reference picture and for which no motion vector for block A, B, and C is available for MVP, the motion vector predictor is set according to the disparity information of the current block of texture data Di(cb). In other words, whenever no motion vector predictor is available and the reference index points to an inter-view reference picture or an inter-view only reference picture, the motion vector predictor is set to a value derived from Di(cb), such as the average disparity derived over all samples of Di(cb).
  • a co-located block in reference index 0 in reference picture list 1 is selected as the source for motion vector predictor for a current block cb using temporal direct mode.
  • a disparity-compensated co-located block is selected as a source for motion vector prediction.
  • the disparity information of the current block of texture data Di(cb) may be averaged and quantized to the block grid for motion-compensated prediction to form a disparity motion vector.
  • the disparity motion vector is then used to find a disparity-compensated co-located block, whose motion vector is used as the source for motion vector prediction.
  • the depth/disparity information of the disparity- compensated co-located block is compared to the depth/disparity information of the current block. If the depth/disparity information of these two blocks is sufficiently similar, for example if their deviation is below a certain threshold, the motion vector of the disparity-compensated co-located block is used in motion vector prediction of the current block. Otherwise, the motion vector of the disparity-compensated co-located block is not used in motion vector prediction of the current block.
  • the disparity/depth information of these two blocks may differ, for example, when an object covered by current block is occluded by another object in disparity-compensated co-located block.
  • a similarity search of depth/disparity information is performed between the depth/disparity of the current block Di(cb) and a selected area in the reference picture.
  • the disparity motion vector may be used as a start position for the similarity search, and only the blocks adjacent to the block where the disparity motion vector may be within the area being searched.
  • the block that results in a smallest value of a distortion or distance measure, such as deviation, compared to Di(cb) is selected as the disparity- compensated co-located block, whose motion vector is used as the source for motion vector prediction of the current block cb.
  • a temporal motion vector prediction may be used in a uni-prediction mode, and a disparity-compensated co-located block is selected as a source for motion vector prediction.
  • the MVP classifies neighboring blocks of the current block (cb) with respect to its prediction directions (intra-view , inter-view, VSP), and processes each prediction direction (group of neighboring blocks) independently.
  • the MVP involves no mixing of temporal, inter-view, or VSP directions in a single decision.
  • the directional and/or median motion vector prediction is computed according from the neighboring blocks with identical prediction direction (again, no mixing of temporal, inter-view, or VSP directions).
  • neighboring blocks A, B, and C may be available for motion vector prediction and/or mixed-direction motion-compensated predictiononly if they have the same prediction direction as the current block (cb).
  • a motion vector predictor from the neighboring blocks can be selected based on depth/disparity similarity, such as the minimal (absolute) difference in average (per pixel) depth/disparity, between currently coded block of texture data, and neighboring blocks of texture data.
  • depth/disparity similarity such as the minimal (absolute) difference in average (per pixel) depth/disparity, between currently coded block of texture data, and neighboring blocks of texture data.
  • the motion vector of that neighboring block that has the minimal (absolute) difference in average (per pixel) depth/disparity compared to the depth/disparity of cb can be selected as the motion vector predictor for the currently coded or decoded texture block (cb).
  • all those neighboring blocks nb for which the similarity measure of the disparity/depth block of nb compared to the disparity/depth block of cb is below a threshold value may be selected for the directional and/or median MVP or alike. 6. Selection of uni-prediction/bi-prediction and the reference picture list(s) used for prediction
  • a number of prediction parameters for the decoding of the currently coded or decoded block (cb) can be determined based on the values associated with the neighboring blocks. For example, the number of reference picture lists used, i.e. the number of prediction blocks used for the currently coded block cb, can be determined based on the number of prediction blocks used in the neighboring block having the greatest depth/disparity similarity, such as the smallest average (absolute) difference in (per pixel) depth/disparity compared to current depth block.
  • the depth/disparity similarity of the current block cb and the prediction block may be used to determine whether uni-prediction or bi-prediction is used, and which reference picture list is used for uni-prediction.
  • bi-prediction there are two prediction blocks, which are herein referred to as pbO and pb1 , which originate from a picture in reference picture list 0 and reference picture list 1 , respectively.
  • pbO and pb1 there are two prediction blocks, which are herein referred to as pbO and pb1 , which originate from a picture in reference picture list 0 and reference picture list 1 , respectively.
  • Let the depth/disparity information of pbO and pb1 be Di(pbO) and Di(pb1 ), respectively.
  • Di(pbO) is compared to the depth/disparity information of the current block Di(cb).
  • Di(pbO) is similar to Di(cb)
  • pbO is used as a prediction block; otherwise, pbO is not used as a prediction block for the current block.
  • Di(pb1 ) is similar to Di(cb)
  • pb1 is used as a prediction block; otherwise pb1 is not used as a prediction block for the current block.
  • both pbO and pb1 remain as prediction blocks, bi-prediction is used. If pbO is a prediction block but pb1 is not, uni-prediction from pbO (i.e. reference picture list 0) is used.
  • uni-prediction from pb1 i.e. reference picture list 1
  • another coding mode may be inferred, such as bi-predictive mode with motion vector difference coding, or another pair of reference pictures may be selected, for example.
  • uni-prediction one reference picture list
  • bi-prediction two reference picture lists
  • the above-mentioned embodiments may be generalized to more than two prediction hypotheses. 7. Coding mode and prediction direction prediction
  • the coding mode e.g. temporal direct, spatial direct, skip
  • the coding mode can be determined to be equal to the coding mode of the neighboring block having the greatest depth/disparity similarity, such as the smallest average (absolute) difference in (per pixel) depth/disparity, compared to current depth block.
  • the prediction direction (intra-view, inter-view, VSP) can be selected based on the greatest depth/disparity similarity, such as the minimal difference in average (per pixel) depth/disparity between currently coded block (cb) of texture, and neighboring blocks of texture.
  • the reference index can be selected based on the greatest depth/disparity similarity, such as the minimal difference in average (per pixel) depth/disparity between currently coded blocks, and neighboring blocks of texture.
  • the encoder/decoder selects one or more adjacent blocks S, T, U, ... to the current block cb 2.
  • the encoder/decoder obtains the depth/disparity blocks associated with the one more adjacent blocks d(S), d(T), d(U), ...
  • the encoder/decoder compares the current depth/disparity block d(cb) to the depth/disparity blocks associated with the one more adjacent blocks d(S), d(T), d(U), ... using a similarity/distortion metric such as the one presented in equation (9).
  • the encoder/decoder selects one or more of the depth/disparity blocks associated with the one more adjacent blocks d(S), d(T), d(U), ... on the basis of comparison (e.g. minimizing distortion or maximizing similarity compared to d(cb)).
  • the encoder/decoder derives motion information MVP for cb from the selected depth/disparity blocks d(S), d(T), d(U) associated with the one more adjacent blocksS, T, U,, ... or from the selected adjacent blocks S, T, U, ... or from the information coded for the selected blocks d(S), d(T), d(U), ... or the selected blocks S, T, U, ...
  • the encoder/decoder encodes/decodes block cb using the motion information MVP.
  • the encoder/decoder may for example use MVP as the reference index and motion vector predictor and encode/decode motion vector difference relative to the MVP; or the encoder/decoder may for example use MVP as the motion vector for cb.
  • the methods for selecting adjacent blocks to be used in MVP may include but are not limited to the following or any combination of the following:
  • adjacent blocks that share a boundary with cb and have been coded/decoded prior to cb.
  • a pre-defined algorithm or rule may be used by the encoder and the decoder to select adjacent blocks identically.
  • adjacent blocks may be selected as blocks A, B, C, D, and E of Figure 13 provided that they are coded/decoded prior to cb.
  • the encoder may select adjacent blocks and code information into the bitstream to identify the adjacent blocks, for example as relative coordinates of the adjacent blocks in relation to the position of cb.
  • the encoder may for example use rate-distortion optimization for the selection of adjacent blocks.
  • the encoder and the decoder may perform motion estimation or block matching for d(cb) within the depth/disparity picture containing d(cb), and each position tested in the block matching may be considered to co-locate with an adjacent block.
  • the encoder and the decoder may perform motion estimation or block matching for d(cb) within the depth/disparity picture containing d(cb), and the adjacent blocks may be selected to co-locate with the one or more positions having among those having smallest cost in block matching.
  • spatially neighboring blocks such as blocks A, B, C, D, and E of Figure 13 provided that they are coded/decoded prior to cb, may be selected for example using step 1 or 2 above.
  • the motion vectors of those blocks may be applied in the location of cb to obtain adjacent blocks, e.g. the adjacent block derived from spatially neighboring block A would be (cb,MV(A)).
  • a motion vector to be used for obtaining adjacent blocks may be derived from more than one motion vector of the selected spatially neighboring blocks.
  • a motion vector MV CV m may be obtained by taking the median of the horizontal components of the motion vectors of the selected spatially neighboring blocks and the media of the vertical components of the motion vectors of the selected spatially neighboring blocks. Then, an adjacent block may be obtained by applying MV cvm in the position of cb, i.e. the adjacent block would be (cb, MV cvm ).
  • reference pictures for motion estimation may be selected for example based on the reference indexes of the motion vectors of spatially neighboring blocks selected e.g. as described in one or more of steps 1 to 3 above.
  • location of a search window may be selected for example based on the location of adjacent block derived in one ore more of steps 1 to 3 above.
  • the encoder and the decoder may perform motion estimation or block matching for d(cb) within the selected depth/disparity reference pictures and search window.
  • the adjacent blocks may be selected to reside in the texture picture associated with the selected depth/disparity reference picture and co-locate with the one or more positions having among those having smallest cost in block matching.
  • the depth/disparity values of d(cb) may be converted to a disparity block d re f(cb) relative to a another view v re f.
  • the average disparity value of d re f(cb), avg(d re f(cb)), may be used as a motion vector pointing to view component v re f.
  • the averaging may also include quantization or rounding for example to the accuracy of motion vectors, such as quarter-pixel position.
  • An adjacent block may be obtained by applying the obtained inter-view motion vector, avg(d re f(cb)), in the position of cb, i.e. the adjacent block would be
  • d(cb) Selecting adjacent blocks by using the motion vectors in the depth/disparity picture containing d(cb) as follows.
  • spatially neighboring depth/disparity blocks relative to d(cb) such as those corresponding to A, B, C, D, and E of Figure 13, may be selected for example using step 1 or 2 above.
  • the co- located or associated block d(cb) may be selected.
  • the respective motion vectors may be denoted MV(d(A)), MV(d(B)), MV(d(C)), MV(d(D)), MV(d(E)), and MV(d(cb)).
  • an adjacent blocks may be obtained by applying the motion vectors of the selected co-located or spatially neighboring depth/disparity blocks to the position of the cb, for example the adjacent block corresponding to neighboring block A would be (cb,MV(d(A))).
  • a motion estimation process such as block matching, may be performed for the current depth/disparity block d(cb) as follows.
  • the search range may be selected to be within the current depth/disparity picture, i.e. within the picture d(cb) also resides.
  • the search range may additionally or alternatively be selected to be within a different depth/disparity picture, i.e. a picture other than the picture where d(cb) resides.
  • the search range be selected to be within a depth/disparity picture that is associated with the texture picture identified by the reference index of cb.
  • the search range may be selected to be within a depth/disparity picture that is inferred as a reference picture for direct mode or alike prediction of cb or d(cb).
  • the search range may be spatially limited to exclude fragments that are associated with not available blocks (i.e. not coded/decoded yet) in a texture picture containing the current block cb.
  • the search range may be further spatially limited not to exceed a certain an image fragment of a certain size and shape, which is surrounding or being adjacent to d(cb).
  • a block matching search may be performed within the search range using a certain algorithm.
  • the encoder indicates the used algorithm and/or parameter values for the used algorithm by encoding one or more indications into the bitstream, and the decoder decodes the one or more indications and performs the block matching search according to the decoded information on the used algorithm and/or parameter values for the used algorithm.
  • a block matching search may for example by done using a full search wherein each block position within the search range is examined.
  • a cost is calculated between the source block, d(cb), and each block within the search range that is selected by the block matching algorithm. For example, SAD, SSD, or SATD may be used as a cost in block matching.
  • the block matching algorithm may be limited to search only certain block positions, which may be co-locating with the rectangular grid used to partition the picture.
  • the block position/positions which minimizes/minimize said cost in block matching may be then processed as follows.
  • Motion information MV(xbb) may be derived for the block position (Xbb, ybb) e.g. through block d(xbb) having the size and shape of d(cb) and located at (x b b, ybb) within the depth/disparity picture containing the search range and/or through block xbb having the size and shape of d(cb) and located at (x b b, ybb) within the texture picture associated with or correspond to the depth/disparity picture containing the search range, e.g. using one of the following ways or their combination:
  • MV(xbb) may be selected to be identical to MV(d(bb)).
  • MV(xbb) may be selected to be derived from MV(d(bb n )) for example by first selecting reference index that is used for motion information for most samples within d(xbb) and then taking the median motion vector, the median motion vector components, the average motion vector, the average motion vector components, or the MV(d(bb n )) covering an area where the normalized cost is the smallest among all values of n that the selected reference index.
  • MV(xbb) may be selected to be identical to MV(bb). If block xbb covers more than one block bb n when n may be vary from one to the number of such blocks and for which motion information MV(bb n ) has been derived and coded/decoded earlier, then MV(xbb) may be selected to be derived from MV(bb n ) for example by first selecting reference index that is used for motion information for most samples within xbb and then taking the median motion vector, the median motion vector components, the average motion vector, the average motion vector components, or the MV(bbn) covering an area where the normalized cost is the smallest among all values of n that the selected reference index.
  • the motion information MV(xbb) may be used for obtaining adjacent blocks for the MVP of cb.
  • an adjacent block may be obtained by applying MV(xbb) in the location of cb, i.e. the adjacent block may be (cb,MV(xbb))
  • the motion information MV(xbb) may be used for prediction of cb.
  • a prediction block for cb may be obtained by applying MV(xbb) in the location of cb, i.e. the prediction block may be (cb,MV(xbb)).
  • motion information for the decoding of the currently coded or decoded block (cb) may be determined as follows.
  • Flow charts of the process for the Depth-based Motion Competition (DMC) in the Skip and Direct modes are shown in Fig. 17 and Fig. 18, respectively.
  • DMC Depth-based Motion Competition
  • Fig. 17 and Fig. 18 Motion vectors ⁇ MVi ⁇ of texture data blocks ⁇ A, B, C ⁇ are grouped according to their prediction direction forming Group 1 and Group 2 for temporal and inter-view, respectively.
  • the DMC process which is detailed in the grey block of Figure 17, is performed for each group independently.
  • a motion-compensated depth block d(cb,MVi) is derived, where the motion vector MVi is applied relative to the position of cb to obtain the depth block from the reference picture pointed to by MVi. Then, the similarity of d(cb) and d(cb,MVi) is estimated as shown in (10):
  • the MVi that provides a minimal SAD value within a current Group is selected as an optimal predictor for a particular direction (mvp d i r )
  • mvp opt arg jrim (sAD (mvp temp ), SAD(mvp inter )) (12)
  • the MVP for the Direct mode of B slices, illustrated in Fig. 18, is similar to the Skip mode, but DMC (marked with grey blocks) is performed over both reference pictures lists (List 0 and List 1 ) independently.
  • DMC (marked with grey blocks) is performed over both reference pictures lists (List 0 and List 1 ) independently.
  • DMC produces two predictors (mvpOdir and mvpl dir) for List 0 and List 1 , respectively.
  • the SAD values of mvpOdir and mvpl dir are computed as shown in (10) and averaged to form the SAD of bi-prediction for each direction independently:
  • the MVP for the Direct mode is selected from available mvpinter and mvptemp as it shown in (12).
  • motion vector prediction scheme can be implemented as follows. All available adjacent to cb blocks ( ⁇ , . , . ⁇ ) are classified according to the direction of their prediction (temporal or inter-view). If cb uses an inter-view reference picture, all neighboring blocks which do not utilize inter-view prediction are marked as not-available for MVP and are not considered in the median or directional MVP calculation or alike. Vice versa, if cb uses temporal prediction, neighboring blocks that used inter-view reference frames are marked as not-available for MVP. The flowchart of this process is depicted in Figure 18.
  • MVx is set to average disparity ⁇ > (cb) value which is associated with current texture cb and computed as:
  • D (cb) (l / N) ⁇ .
  • D(cb(i)) where D(cb(i)) is a disparity computed in (2) for pixel cb(i), i pixel index in block cb and N is number of pixels in block cb .
  • the inference of all or certain above-mentioned prediction parameters for cb from the neighboring block having the greatest depth/disparity similarity such as the smallest average (absolute) difference in (per pixel) depth/disparity compared to current depth block may be performed only if a depth/disparity similarity value meets a condition related to a threshold, such as exceeds a threshold.
  • a threshold such as exceeds a threshold.
  • Different thresholds for prediction parameters may also be used.
  • MVP motion vector prediction
  • a modified procedure for determination of a reference index and candidate motion vectors for motion vector prediction for a skip and/or direct mode and/or alike is disclosed herein and referred to as disparity-compensated zero-value prediction mode.
  • the average depth/disparity value for a block of depth information Av(cb_d) and the deviation of the depth/disparity values within a block of depth information Dev(cb_d) may be computed according to equations (3) and (4). Then if the deviation of the depth/disparity values within a block of depth information cb_d is less than or equal than the threshold T1 according to equation (5), the current coded block cb may be considered as homogenous.
  • a disparity value d(cb) which is common for the whole coded block cb, may be calculated according to equation (2).
  • the resulting disparity value d(cb) may be then used as motion vector predictor and the reference index may be inferred to be equal to the reference index of the inter-view reference picture or inter-view only reference picture for which the disparity applies.
  • the zero-value prediction as disclosed in the current H.264/AVC MVP may be used; e.g.
  • similarities in the depth/disparity information of the current coded block (cb) and each of its neighboring blocks (nb) are used as a basis for determining one or more parameters for the motion vector prediction of the current coded block (cb).
  • the average deviation between the current coded block cb and each of its neighboring blocks nb(A, B, C, ...) is computed according to equation (9). From the resulting deviation values, a minimum value min(diff(cb, nb)) is searched.
  • a depth/disparity based neighbor preselection process can be used for the motion vector prediction.
  • the neighboring blocks are grouped together according to their prediction direction. Accordingly, three possible groups of blocks (Gx) are available: G1 : Temporally predicted blocks (intra-view prediction), G2: Inter-view prediction block, and G3: VSP prediction blocks.
  • T3 i.e. if min(diff(cb, Gx)
  • the above embodiment of the disparity-compensated zero-value prediction mode could be used.
  • T3 i.e. if min(diff(cb, Gx)) > T3
  • the above embodiment of the disparity-compensated zero-value prediction mode could be used.
  • the last embodiment can be illustrated by the flow chart of Figure 8, wherein spatially neighboring blocks A, B and C are available for the motion vector prediction of the current block cb. If no identical prediction direction for the blocks A, B and C can be found (800), then the blocks A, B and C are grouped (802) according to their prediction direction to three possible groups, and for all groups presently available, the average deviation between the current coded block and each group of blocks is computed (804, 806, 808).
  • the prediction direction group providing the minimum value min(diff(cb, Gx)) is selected (810) as the prediction direction. Then the blocks within said group are marked (812) as available for motion vector prediction and/or mixed-direction motion-compensated prediction and the rest of blocks are marked as unavailable for motion vector prediction and/or mixed-direction motion-compensated prediction. After that motion vector prediction (MVP0), such as directional and/or median motion vector prediction of H.264/AVC, is performed (814) for the current coded block from neighboring blocks, which belong to the said group.
  • MVP0 motion vector prediction
  • a skip or direct mode or alike is used for the current block and it is concluded (816) that a conventional motion vector prediction process, such as H.264/AVC MVP, would use a "zero-value" reference index and/or a "zero-value default" motion vector predictor, then the reference index and motion vector predictor are selected in 818 as follows. If a conventional motion vector prediction process would use a "zero- value" reference index, the reference index is instead selected to refer to a reference picture having the smallest reference index and having the same prediction direction as the selected prediction direction group.
  • the disparity value d(cb) according to equation (2) is used instead of a "zero-value” motion vector predictor in coding or decoding process (820). Otherwise, the "zero-value default" motion vector predictor, for example as disclosed in the current H.264/AVC MVP is be used.
  • the flow chart of Figure 9 illustrates an embodiment for a temporal direct mode or alike.
  • the candidate reference index pairs to be used for the current cb are determined as follows.
  • the reference index variable for list 1 cri is first set to 0.
  • the co-located block A resides in a reference picture having the reference index cri in reference picture list 1 . If the prediction direction of block A is inter-view, the selection of block A may be disparity compensated as described earlier.
  • the co-located block A uses mixed- direction motion-compensated prediction from block B from a reference picture that is present in reference picture list 0.
  • block A and B are considered available and the respective reference indexes are included as a candidate reference index pair. If the prediction direction of block A is interview and the reference pictures containing block A and block B are in different views (but the same access unit) as the picture containing cb, block A and B are considered available and the respective reference indexes are included as a candidate reference index pair. Otherwise, block A and B are considered not available. Then, cri is incremented by 1 and the steps above are repeated, unless cri refers to a reference index that is not available in reference picture list 1 . If no there is no candidate reference index pair at the end of the process of 900, then for example the reference index selection for temporal direct mode of H.264/AVC may be applied.
  • the average depth/disparity deviation between the current block cb and blocks A and B of each candidate reference index pair may be computed according to equation (9). From the resulting deviation values, the candidate reference index pair with a minimum value min(diff(cb, nb)) may be selected (906). If the minimum value of the selected reference index pair min(diff(cb, nb)) is less than or equal to a threshold T (908), then temporal direct prediction or alike is applied for cb (910). In some embodiments steps 902, 906, and 908 may be omitted and temporal direct mode prediction or alike may be performed using the first candidate reference index pair found in 900.
  • camera index differences, camera translation vectors, camera separations, or alike may be used in the temporal direct mode or alike for scaling the motion vector from the co-located block when using it for deriving a motion vector predictor for the current block cb.
  • neighboring blocks to the current block being coded/decoded cb are selected.
  • selecting neighboring blocks include spatial neighbors (e.g. as indicated in Figure 7a) or temporal co-located neighbors as determined for example by the reference index selection of a temporal direct mode or alike (e.g. as indicated in Figure 7b).
  • Other examples include disparity-compensated neighbors in adjacent views whereby disparity compensation may be applied to determine correspondence of a neighboring block to cb.
  • the aspects of the invention are not limited to the mentioned methods of selecting neighboring blocks, but rather the description is given for one possible basis on top of which other embodiments of the invention may be partly or fully realized.
  • the encoder may determine the value of any of the above-mentioned thresholds for example based on encoding blocks with different values of the threshold and selecting the value of the threshold that is optimal according to the Lagrangian rate-distortion optimization equation.
  • the encoder may indicate the determined value of the threshold within the bitstream, for example by encoding it as a syntax element for example in a sequence parameter set, a picture parameter set, a slice parameter set, a picture header, a slice header, within a macroblock syntax structure, or anything alike, in some embodiments, the decoder determines the threshold based on the information encoded in the bistream, such as a codeword indicating the value of threshold.
  • the encoder performs optimization, such as rate- distortion optimization, jointly when selecting values of syntax elements for the current texture block cb and the co-located currect depth/disparity block Di(cb).
  • optimization such as rate- distortion optimization
  • the encoder may for example encoder cb and Di(cb) in multiple modes and select the pair of modes that results into the best rate-distortion performance among the tested modes.
  • a skip mode would be optimal in rate- distortion performance for Di(cb), but when the rate-distortion performance of cb and Di(cb) is optimized jointly, it may be more beneficial that for example a direct mode is selected for Di(cb) and consequently the prediction error signal for Di(cb) becomes encoded and the prediction parameter selection, such as motion vector prediction, based on decoded Di(cb) may become such that the rate-distortion performance of cb coding is improved.
  • the rate and distortion for texture and depth jointly for example one or more synthesized view may be used to derive the distortion, because texture picture distortion may not be directly comparable to depth picture distortion.
  • the encoder may also select values for syntax elements such a manner that depth/disparity based prediction parameter becomes effective. For example, the encoder may select a skip or direct mode or alike for a current texture block cb when some of the neighboring blocks have similar depth/disparity compared to the depth/disparity of cb and hence depth/disparity based selection of motion vector predictor(s) is likely to succeed well. Likewise, the encoder may avoid selecting a skip or direct mode or alike for cb when no neighboring blocks have similar depth/disparity compared to the depth/disparity of cb.
  • the encoder may encode some texture views without depth/disparity based prediction parameter derivation while other texture views may be encoded using depth/disparity based prediction parameter derivation.
  • the encoder may encoder the base view of the texture without depth/disparity based prediction parameter derivation and rather use conventional prediction mechanisms.
  • the encoder may encode a bitstream compatible with the H.264/AVC standard by encoding a base view without depth/disparity based prediction parameter derivation and hence the bitstream can be decoded by H.264/AVC decoders.
  • the encoder may encode a bitstream where a set of views is compatible with MVC by encoding a base view and the other views in the set of views without depth/disparity based prediction parameter derivation. Consequently, the set of views can be decoded by MVC decoders.
  • chroma motion information may be derived from luma motion information using pre-determined relations. For example, it may be assumed that the same reference indexes are used for the chroma components as for luma and that the motion vectors of chroma are scaled from the luma motion vectors in the same proportion as the spatial resolution of chroma pictures compared to the spatial resolution of luma pictures. In some embodiments the spatial resolution of depth/disparity pictures may differ or may be re-sampled in the encoder as a pre-processing operation to become different from that of the luma pictures of texture.
  • the depth/disparity pictures are re-sampled in the encoding loop and/or the decoding loop to become identical resolution to the respective luma pictures of texture.
  • the spatially corresponding blocks of depth/disparity pictures are found by scaling the block locations and size proportionally to the ratio of the picture extents of the depth pictures and luma pictures of texture.
  • a texture image and the depth image associated with the texture image are presented at different spatial resolution, thus coded block Cb and associated d(Cb) have different size in spatial domain (different number of pixels in horizontal and vertical directions or in either of those).
  • the encoder may use several methods to indicate the different resolutions of the texture images relative to the resolution of the depth images.
  • the resolution of the texture image and the resolution of the depth image can be separately indicated by an encoder in a sequence parameter set coded into the bitstream.
  • the encoder may encode two sequence parameter sets, one to be used for decoding texture view components and another to be used for decoding depth view components, where each sequence parameter set includes syntax elements indicating a spatial resolution.
  • the encoder encodes coded slices for texture with a reference to the sequence parameter set for texture view components and respectively coded slices for depth with a reference to the sequence parameter set for depth view components.
  • the reference for a sequence parameter set may be indirect, e.g. it may be through a reference in a picture parameter set, and the picture parameter set may be indicated in a slice header of a coded slice.
  • the decoder resolves or decodes the reference between syntax structures and uses the signaled spatial resolutions for decoding of a texture view component and depth view component.
  • the ratio between a texture resolution and a depth resolution may be pre-defined for example to 1 :1 or 2:1 (along both coordinate axes), and no information regarding one of the texture and depth resolution needs to be coded to the bitstream or decoded from the bitstream.
  • spatial resolutions of texture and depth images can be normalized (adjusted to a single resolution) by re-sampling (interpolating) of either component (texture or depth) to the resolution of another component (depth or texture, respectively).
  • Resampling to a higher resolution can be implemented by various means, for example by linear-filter interpolation (e.g. bi-linear, cubic, lanczos, or interpolation filters utilized in H.264/AVC or in HEVC), or by non-linear filter upsampling, or by simple duplication of image samples.
  • Resampling to a lower resolution can be implemented by various means, for example by decimation, or decimation with low pass filtering.
  • the encoder may determine a resampling process, parts thereof, or threshold or other parameter values to be used based on one or more cost functions and by (approximate) minimization/maximization of these one or more cost functions. For example, a peak signal-to-noise ratio (PSNR) of the resampled depth picture relative to the original depth picture may be used as a basis of a cost function.
  • PSNR peak signal-to-noise ratio
  • the encoder may perform a rate-distortion optimized selection of one or more of the resampling process, parts thereof, or threshold or other parameter values, where the distortion is based on a selected cost function.
  • the cost function used by the encoder may also be based on synthesizing a picture, where in the synthesis process the resampled depth picture is used, and applying a similarity measure, such as PSNR, to the synthesized picture.
  • the encoder may encode indications specifying at least parts of the used resampling method. Such indications may include one or more of the following: - Identification of the used resampling process in encoding and/or to-be- used resampling process in decoding
  • the encoder may encode such indications into various parts of the coded bitstream, such as sequence parameter set, picture parameter set, adaptation parameter set, picture header, or slice header.
  • a decoder may decode from the bitstream indications specifying at least parts of the resampling method used in encoding and/or to be used in decoding. The decoder may then perform the resampling process in accordance with the decoded indications.
  • the resampling may take place in the (de)coding loop, i.e. upsampled depth may be used in other (de)coding processes.
  • the upsampling may take place for a complete picture or a part of a picture, such as a set of contiguous lines of samples for example covering a line of coding blocks.
  • motion information MV(A) and MV(B) of first adjacent texture block A and the second adjacent texture block B respectively is represented with a specific accuracy (e.g. 1 ⁇ 4-pixel location accuracy of motion vectors in the H.264/AVC standard, or with 1/8-pixel location accuracy in others).
  • a specific accuracy e.g. 1 ⁇ 4-pixel location accuracy of motion vectors in the H.264/AVC standard, or with 1/8-pixel location accuracy in others.
  • in-loop upsampling of a reference image by factor of 4x along both horizontal and vertical axis may be performed at both encoder and decoder side to perform motion or disparity compensated prediction.
  • motion vector components of MV(A) and MV(B) resulting from such motion estimations are represented and processed at the 4x of the original image resolution.
  • both texture and depth data may be up-sampled to this particular resolution, which may be referred to as common upsampled in-loop resolution.
  • the up-sampling can be implemented for an entire frame (usually done in the encoder), or interpolation can be performed locally, e.g. at block level (usually done in the decoder).
  • motion vector components of MV(A) and MV(B) of adjacent texture blocks A and B, respectively may be rescaled to meet the spatial resolution of the depth image instead of resampling of the depth image. For example, if MV(A) and MV(B) of texture data are represented at the original texture resolution and the resolution of depth is smaller than resolution of texture, motion vector components (mv_x and mv_y) of MV(A) and MV(B) are downscaled by ratio at which resolution of texture and depth data are different).
  • the downscaling of motion vector components may be followed by quantizing the downscaled motion vector components to certain accuracy, where the quantization operation may include rounding for example towards the closest quantization level.
  • the quantization may be selected such a manner that no resampling of the depth picture is necessary or resampling is performed to a smaller resolution than what would be needed without quantization.
  • the resolution to which the depth pictures are resampled in the coding or decoding loop in order to perform the proposed MVP may be referred to as the in-loop depth resolution.
  • the depth picture may have a resolution half of that of the luma texture picture along both coordinates axes (e.g. luma texture picture has resolution 1024x768 and depth picture has resolution 512x384) and the motion vector components of the luma texture component have 1/4-pixel accuracy (e.g. mv_x is equal to 5.25 and mv_y is equal to 4.25).
  • the motion vector components to be utilized for depth in the proposed MVP are scaled by a ratio of 2 (e.g. mv_x equal to 2.625 and mv_y equal to 2.125, with reference to the same example as before) and would therefore have 1/8-pixel accuracy within the depth picture.
  • the scaled motion vector components may be quantized to integer-pixel accuracy within the depth picture (e.g. mv_x equal to 3 and mv_y equal to 2, with reference to the sample example as before).
  • the scaled motion vector components may be quantized to half-pixel accuracy within the depth picture (e.g.
  • mv_x equal to 2.5 and mv_y equal to 2, with reference to the sample example as before
  • the depth picture therefore needs to be upsampled by a factor of 2 along both coordinate axes (e.g. 1024x768, with reference to the same example as before), where upsampling may be performed for example using bilinear upsampling.
  • Such implementations of the proposed MVP would allow avoiding resampling of the depth data to the required resolution, and may have lower computational complexity.
  • the encoder may encode indications or syntax elements specifying motion vector scaling to be used in the proposed MVP.
  • Such indications may include one or more of the following:
  • the accuracy to which the motion vector components are scaled to, for example 1 ⁇ 4, 1 ⁇ 2, or full pixel accuracy.
  • the encoder may encode such indications into various parts of the coded bitstream, such as sequence parameter set, picture parameter set, adaptation parameter set, picture header, or slice header.
  • a decoder may decode from the bitstream indications specifying the depth motion vector scaling to be used in the proposed MVP.
  • the decoder may then perform the motion vector scaling accordingly and subsequently uses the scaled motion vectors in the proposed MVP for texture.
  • a similarity metric which is utilized in the proposed MVP can be adjusted to use a different block size and/or shape for the similarity comparison than the block size and/or shape of cb. For example, if the in-loop depth resolution differs from the in-loop luma texture resolution, the block size of the similarity comparison may be changed according to the ratio of the in-loop depth resolution to in-loop luma texture resolution.
  • a block size half of cb along both coordinate axes may be used for similarity comparison.
  • a similarity metric which is utilized in the MVP can be adjusted to reflect the difference in spatial resolution between texture or depth data. For example, if the depth picture has been resampled for the proposed MVP to the common upsampled in-loop resolution, the similarity metric which is utilized in the MVP can be adjusted according to the ratio the depth resampling was performed.
  • Said adjustment may be implemented through decimation or subsampling, if the resolution of depth data is higher than the resolution of motion vectors, or through interpolation or upsampling of depth information d(MV(A), d(cb)) if the resolution of depth is smaller than the resolution of texture data or motion vectors (MV(A), MV(B)).
  • d(MV(A), d(cb) depth information if the spatial resolution of luma texture data is double of the spatial resolution of depth data along both coordinate axes and depth is resampled to the common upsampled in-loop resolution
  • every other depth pixel along both coordinate axes in the depth picture at the common upsampled in-loop resolution may be included in the calculations of the similarity method.
  • the encoder may encode indications or syntax elements specifying the similarity metric use with the proposed MVP. Such indications may include one or more of the following:
  • the common upsampled in-loop resolution The decimation scheme and ratio to be used when selecting depth pixels for the similarity metric calculation. For example, it could be indicated that every other depth pixel (along both coordinate axes) from the depth picture at the common upsampled in-loop resolution is used in the similarity metric calculation.
  • the block size or a ratio indicating the block size relation between texture and depth images to be used in the similarity metric calculation For example, it could be indicated that a block size half of the size of the current text block cb along both coordinate axes is used for the similarity metric.
  • the encoder may encode such indications into various parts of the coded bitstream, such as sequence parameter set, picture parameter set, adaptation parameter set, picture header, or slice header.
  • a decoder may decode from the bitstream indications specifying the similarity metric calculation to be used in the proposed MVP. The decoder may then perform the indicated similarity metric calculation in the proposed MVP for texture.
  • texture and associated depth images can be represented with different sampling method.
  • each of partition of coded texture data (PU, 4x4, 8x8, 16x8, 8x16 blocks and others) may be associated with a single depth value which would represent an average depth value for a current partition of texture.
  • This format for depth image can be considered as non-uniform depth data representation, and may be different from a typical sampling scheme utilized for texture data images (uniform sampling).
  • Figure 20 shows an example of such a representation, where the gray rectangle on the right shows borders of a texture image with regular sampling, the gray rectangle on the left shows depth image borders associated with this texture data.
  • Blocks ⁇ Cb1-Cb3 ⁇ of texture image consist of a group of texture image pixels represented with uniform sampling scheme.
  • the depth information ⁇ d(Cb1 ) - d(Cb3) ⁇ however is represented a single depth value per texture blocks ⁇ Cb1 -Cb3 ⁇ , and these values are shown in red circles at the left of image below.
  • referenced depth blocks d(MV(A), d(Cb)) and d(MV(B), d(Cb)) may not be accessed directly and some operations for depth value extraction may have to be performed.
  • referenced texture block X is addressed by MV(A) from the Cb locations, and the spatial size of block X is equal to the spatial size of Cb block.
  • block X may overlap several partitions of the texture data (which are marked as
  • referenced block d(X) which is equal to d(MV(A), d(Cb)) covers depth area which is represented with several depth map values ⁇ d(Cb1 ), d(Cb2), d(Cb3) ⁇ , shown as red circles within white blocks areas.
  • a depth value d(X) for the referenced texture block X need to be extracted from available nonuniform sampled depth data ⁇ d(Cb1 ), d(Cb2) and d(Cb3) ⁇ or their neighbors by a resampling method.
  • resampling may include linear or non-linear operations of fusing (merging) or averaging of available depth samples which happen to represent referenced blocks d(X) in such nonuniform representation.
  • depth data from additional depth samples in neighborhood of ⁇ d(Cb1 ), d(Cb2), d(Cb3) ⁇ may be used to improve depth estimation process.
  • depth data at non-uniform representation can be transmitted within syntax elements of coded texture image, e.g. at the slice header or within coded texture block syntax structures.
  • operation of d(X) calculation would require extraction of depth information from syntax elements of Cb1 -Cb3 and following fusing/merging or aggregating depth information d(Cb1 )-d(Cb3) or other to produce d(X) data.
  • depth data at non-uniform representation for corresponding blocks Cb1 -Cb3 can be extracted/estimated/produced at the decoder side from available multiview representation of texture data.
  • operation of d(X) calculation would require estimation depth information for texture blocks Cb1 -Cb3 from available multiview representation of texture data for following fusing/merging or aggregating depth information d(Cb1 )-d(Cb3) or other to produce d(X) data.
  • two example methods are described by which a suitable estimate for the depth map of the current texture view component can be derived based on already transmitted information.
  • the depth data is transmitted as a part of the bitstream, and a decoder using this method decodes the depth maps of previously coded views for decoding dependent views.
  • the depth map estimate can be based on an already (de)coded depth map of another view of the same access unit or picture sampling instant. If the depth map for a reference view is coded before the current picture, the reconstructed depth map can be mapped or warped or view-synthesized into the coordinate system of the current picture for obtaining a suitable depth map estimate for the current picture.
  • a mapping is illustrated for a simple depth map, which consists of a square foreground object and background with constant depth.
  • the depth sample value is converted into a sample-accurate disparity vector. Then, each sample of the depth map is displaced by the disparity vector. If two or more samples are displaced to the same sample location, the sample value that represents the minimal distance from the camera (i.e., the sample with the larger value in some embodiments) is chosen.
  • the described mapping leads to sample locations in the target view to which no depth sample value is assigned. These sample locations are depicted as a black area in the middle of the picture of Figure 21 . These areas represent parts of the background that are uncovered due to the movement of the camera and can be filled using surrounding background sample values.
  • a simple hole filling algorithm may be used which processes the converted depth map line by line.
  • Each line segment that consists of a successive sample location to which no value has been assigned is filled with the depth value of the two neighboring samples that represent a larger distance to the camera (i.e., the smaller depth value in some embodiments).
  • the left part of Figure 21 illustrates the original depth map; the middle part illustrates the converted depth map after displacing the original samples; and the right part illustrates the final converted depth map after filling of holes.
  • the depth map estimate is based on coded disparity and motion vectors.
  • DCP disparity-compensated prediction
  • the remaining blocks are intra-coded.
  • DCP disparity-compensated prediction
  • candidate disparity vectors can only be derived using a local neighborhood, i.e., by conventional motion vector prediction. But after coding the first dependent view in a random access unit, the transmitted disparity vectors can be used for deriving a depth map estimate, as it is illustrated in Figure 22.
  • the disparity vectors used for disparity-compensated prediction are converted into depth values and all depth samples of a disparity-compensated block are set equal to the derived depth value.
  • the depth samples of intra-coded blocks are derived based on the depth samples of neighboring blocks; the used algorithm is similar to spatial intra prediction. If more than two views are coded, the obtained depth map can be mapped into other views using the method described above and used as a depth map estimate for deriving candidate disparity vectors.
  • the depth map estimate for the picture of the first dependent view in a random access unit is used for deriving a depth map for the next picture of the first dependent view.
  • the basic principle of the algorithm is illustrated in Figure 23.
  • the derived depth map is mapped into the base view and stored together with the reconstructed picture.
  • the next picture of the base view may typically be inter-coded.
  • MCP motion compensated prediction
  • the associated motion parameters are applied to the depth map estimate.
  • a corresponding block of depth map samples is obtained by motion compensated prediction with the same motion parameters as for the associated texture block; instead of a reconstructed video picture the associated depth map estimate is used as a reference picture.
  • the motion compensated prediction for depth block may not involve any interpolation.
  • the motion vectors may be rounded to sample-precision before they are used.
  • the depth map samples of intra-coded blocks are again determined on the basis of neighboring depth map samples.
  • the depth map estimate for the first dependent view which is used for the inter-view prediction of motion parameters, is derived by mapping the obtained depth map estimate for the base view into the first dependent view.
  • the estimate of the depth map is updated based on actually coded motion and disparity parameters, as it is illustrated in Figure 24.
  • the depth map samples are obtained by converting the disparity vector into a depth value.
  • the depth map samples for blocks that are coded using motion compensated prediction can be obtained by motion compensated prediction of the previously estimated depth maps, similar as for the base view.
  • a mechanism by which new depth values are determined by adding a depth correction may be used.
  • the depth correction is derived by converting the difference between the motion vectors for the current block and the corresponding reference block of the base view into a depth difference.
  • the depth values for intra-coded blocks are again determined by a spatial prediction.
  • the updated depth map is mapped into the base view and stored together with the reconstructed picture. It can also be used for deriving a depth map estimate for other views in the same access unit.
  • a depth map estimate for the base view picture is determined by motion compensated prediction using the transmitted motion parameters. This estimate is mapped into the second view and used for the inter-view prediction of motion parameters. After coding the picture of the second view, the depth map estimate is updated using the actually used coding parameters.
  • the inter-view motion parameter prediction is not used, and after decoding the first dependent view of the random access unit, the depth map may be re-initialized as described above.
  • proposed MVP scheme can be combined with other motion vector prediction schemes, such as MVP of H.264/AVC or MVP of High Efficiency Video Coding (HEVC) development.
  • MVP of H.264/AVC or MVP of High Efficiency Video Coding (HEVC) development.
  • HEVC High Efficiency Video Coding
  • proposed MVP scheme can be adaptively supplemented with alternative MVP schemes.
  • MVP selection for example can be based on frame-level rate- distortion optimization.
  • a frame is coded with different MVP schemes, and MVP which provides minimal rate-distortion cost is selected and signaled at the picture parameter set (PPS), adaptation parameter set (APS), picture header, slice header, or anything alike.
  • PPS picture parameter set
  • APS adaptation parameter set
  • MVP selection can be implemented at the slice, block, or the block partition level and an MVP index or a similar indication indicating the MVP scheme in use is signaled in the slice header or the block syntax structure prior to a decoded block partition.
  • decoder extracts the MVP index or a similar indication from the bitstream and configures decoding process accordingly.
  • Figure 10 shows a schematic block diagram of an exemplary apparatus or electronic device 50, which may incorporate a codec according to an embodiment of the invention.
  • the electronic device 50 may for example be a mobile terminal or user equipment of a wireless communication system. However, it would be appreciated that embodiments of the invention may be implemented within any electronic device or apparatus which may require encoding and decoding or encoding or decoding video images.
  • the apparatus 50 may comprise a housing 30 for incorporating and protecting the device.
  • the apparatus 50 further may comprise a display 32 in the form of a liquid crystal display.
  • the display may be any suitable display technology suitable to display an image or video.
  • the apparatus 50 may further comprise a keypad 34.
  • any suitable data or user interface mechanism may be employed.
  • the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.
  • the apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input.
  • the apparatus 50 may further comprise an audio output device which in embodiments of the invention may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection.
  • the apparatus 50 may also comprise a battery 40 (or in other embodiments of the invention the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator).
  • the apparatus may further comprise an infrared port 42 for short range line of sight communication to other devices.
  • the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB/firewire wired connection.
  • the apparatus 50 may comprise a controller 56 or processor for controlling the apparatus 50.
  • the controller 56 may be connected to memory 58 which in embodiments of the invention may store both data in the form of image and audio data and/or may also store instructions for implementation on the controller 56.
  • the controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and decoding of audio and/or video data or assisting in coding and decoding carried out by the controller 56.
  • the apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.
  • the apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network.
  • the apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and for receiving radio frequency signals from other apparatus(es).
  • the apparatus 50 comprises a camera capable of recording or detecting individual frames which are then passed to the codec 54 or controller for processing.
  • the apparatus may receive the video image data for processing from another device prior to transmission and/or storage.
  • the apparatus 50 may receive either wirelessly or by a wired connection the image for coding/decoding.
  • the system 10 comprises multiple communication devices which can communicate through one or more networks.
  • the system 10 may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, CDMA network etc), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.
  • the system 10 may include both wired and wireless communication devices or apparatus 50 suitable for implementing embodiments of the invention.
  • the system shown in Figure 12 shows a mobile telephone network 1 1 and a representation of the internet 28.
  • Connectivity to the internet 28 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.
  • the example communication devices shown in the system 10 may include, but are not limited to, an electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22.
  • PDA personal digital assistant
  • IMD integrated messaging device
  • the apparatus 50 may be stationary or mobile when carried by an individual who is moving.
  • the apparatus 50 may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle or any similar suitable mode of transport.
  • Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24.
  • the base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 1 1 and the internet 28.
  • the system may include additional communication devices and communication devices of various types.
  • the communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol- internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.1 1 and any similar wireless communication technology.
  • CDMA code division multiple access
  • GSM global systems for mobile communications
  • UMTS universal mobile telecommunications system
  • TDMA time divisional multiple access
  • FDMA frequency division multiple access
  • TCP-IP transmission control protocol- internet protocol
  • SMS short messaging service
  • MMS multimedia messaging service
  • email instant messaging service
  • Bluetooth Bluetooth
  • IEEE 802.1 1 any similar wireless communication technology.
  • a communications device involved in implementing various embodiments of the present invention may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.
  • embodiments of the invention operating within a codec within an electronic device, it would be appreciated that the invention as described below may be implemented as part of any video codec. Thus, for example, embodiments of the invention may be implemented in a video codec which may implement video coding over fixed or wired communication paths.
  • user equipment may comprise a video codec such as those described in embodiments of the invention above. It shall be appreciated that the term user equipment is intended to cover any suitable type of wireless user equipment, such as mobile telephones, portable data processing devices or portable web browsers.
  • elements of a public land mobile network may also comprise video codecs as described above.
  • PLMN public land mobile network
  • the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof.
  • some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto.
  • firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto.
  • While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
  • the embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware.
  • any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions.
  • the software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.
  • the memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.
  • the data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multi-core processor architecture, as non-limiting examples.
  • Embodiments of the inventions may be practiced in various components such as integrated circuit modules.
  • the design of integrated circuits is by and large a highly automated process.
  • Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

La présente invention concerne un appareil, un procédé et un programme informatique de codage et décodage pour vidéo. Le procédé comprend les étapes consistant à : décoder, à partir d'un train de bits, un premier bloc de texture codé d'une première image de texture codée dans un premier bloc de texture (cb), le décodage du premier bloc de texture codé consistant à : sélectionner un premier bloc de texture (A) adjacent et un second bloc de texture (B) adjacent; obtenir un premier bloc de profondeur/disparité adjacent (d (A)) et un second bloc de profondeur/disparité adjacent (d (B)); obtenir un premier bloc de profondeur/disparité (d (cb)) spatialement co-situé avec le premier bloc de texture (cb); comparer le premier bloc de profondeur/disparité (d(cb)) adjacent avec le premier bloc de profondeur/disparité (d(A)) adjacent et le second bloc de profondeur/disparité (d (B))adjacent; sélectionner une ou les deux parmi le premier bloc de texture adjacent (A) et le second bloc de texture (B) adjacent sur la base d'une valeur de similarité résultant de ladite comparaison; dériver un ou plusieurs paramètres de prédiction pour le décodage du premier bloc de texture codé (cb) à partir de valeurs associée à l'un sélectionné ou les deux parmi le premier bloc de texture (A) adjacent et le second bloc de texture (B) adjacent; et décoder le premier bloc de texture codées (cb) à l'aide d'un ou plusieurs paramètres de prédiction dérivés.
PCT/CN2012/000139 2012-02-02 2012-02-02 Appareil, procédé et programme informatique de codage et décodage pour vidéo WO2013113134A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/000139 WO2013113134A1 (fr) 2012-02-02 2012-02-02 Appareil, procédé et programme informatique de codage et décodage pour vidéo

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/000139 WO2013113134A1 (fr) 2012-02-02 2012-02-02 Appareil, procédé et programme informatique de codage et décodage pour vidéo

Publications (1)

Publication Number Publication Date
WO2013113134A1 true WO2013113134A1 (fr) 2013-08-08

Family

ID=48904349

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2012/000139 WO2013113134A1 (fr) 2012-02-02 2012-02-02 Appareil, procédé et programme informatique de codage et décodage pour vidéo

Country Status (1)

Country Link
WO (1) WO2013113134A1 (fr)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015139183A1 (fr) * 2014-03-17 2015-09-24 Mediatek Inc. Procede de signalisation en mode de partitionnement de blocs sur la base de profondeur pour le codage video trdiimensionnel et multi-vues
US9503723B2 (en) 2013-01-11 2016-11-22 Futurewei Technologies, Inc. Method and apparatus of depth prediction mode selection
US9800857B2 (en) 2013-03-08 2017-10-24 Qualcomm Incorporated Inter-view residual prediction in multi-view or 3-dimensional video coding
CN110677676A (zh) * 2019-09-27 2020-01-10 腾讯科技（深圳）有限公司 视频编码方法和装置、视频解码方法和装置及存储介质
US10708608B2 (en) 2013-07-15 2020-07-07 Sony Corporation Layer based HRD buffer management for scalable HEVC
CN111787319A (zh) * 2020-07-22 2020-10-16 腾讯科技（深圳）有限公司 一种视频信息处理方法、多媒体信息处理方法及、装置

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101248670A (zh) * 2005-09-22 2008-08-20 三星电子株式会社 估计视差矢量的方法以及使用该视差矢量估计方法对多视角运动图像编码和解码的方法和设备
CN101978697A (zh) * 2008-01-25 2011-02-16 惠普开发有限公司 针对基于块的编码的编码模式选择
CN102158710A (zh) * 2011-05-27 2011-08-17 山东大学 一种面向虚拟视图质量的深度图编码率失真判断方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101248670A (zh) * 2005-09-22 2008-08-20 三星电子株式会社 估计视差矢量的方法以及使用该视差矢量估计方法对多视角运动图像编码和解码的方法和设备
CN101978697A (zh) * 2008-01-25 2011-02-16 惠普开发有限公司 针对基于块的编码的编码模式选择
CN102158710A (zh) * 2011-05-27 2011-08-17 山东大学 一种面向虚拟视图质量的深度图编码率失真判断方法

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9503723B2 (en) 2013-01-11 2016-11-22 Futurewei Technologies, Inc. Method and apparatus of depth prediction mode selection
US10306266B2 (en) 2013-01-11 2019-05-28 Futurewei Technologies, Inc. Method and apparatus of depth prediction mode selection
US9800857B2 (en) 2013-03-08 2017-10-24 Qualcomm Incorporated Inter-view residual prediction in multi-view or 3-dimensional video coding
US10708608B2 (en) 2013-07-15 2020-07-07 Sony Corporation Layer based HRD buffer management for scalable HEVC
WO2015139183A1 (fr) * 2014-03-17 2015-09-24 Mediatek Inc. Procede de signalisation en mode de partitionnement de blocs sur la base de profondeur pour le codage video trdiimensionnel et multi-vues
CN110677676A (zh) * 2019-09-27 2020-01-10 腾讯科技（深圳）有限公司 视频编码方法和装置、视频解码方法和装置及存储介质
CN110677676B (zh) * 2019-09-27 2024-02-09 腾讯科技（深圳）有限公司 视频编码方法和装置、视频解码方法和装置及存储介质
CN111787319A (zh) * 2020-07-22 2020-10-16 腾讯科技（深圳）有限公司 一种视频信息处理方法、多媒体信息处理方法及、装置

Similar Documents

Publication Publication Date Title
US11523098B2 (en) Efficient multi-view coding using depth-map estimate and update
US20130229485A1 (en) Apparatus, a Method and a Computer Program for Video Coding and Decoding
CA2876344C (fr) Appareil, procede et produit programme d'ordinateur pour un codage video en 3d
CN108293136B (zh) 编码360度全景视频的方法、装置和计算机可读存储介质
CA2909309C (fr) Prediction harmonisee de vues intermediaires et de syntheses de vues pour un codage video 3d
US10694165B2 (en) Efficient multi-view coding using depth-map estimate for a dependent view
US9191646B2 (en) Apparatus, a method and a computer program for video coding and decoding
EP2839660B1 (fr) Appareil, procédé et programme informatique permettant le codage et de décodage de vidéos
US20140009574A1 (en) Apparatus, a method and a computer program for video coding and decoding
US20140301463A1 (en) Method and apparatus for video coding and decoding
US20140218473A1 (en) Method and apparatus for video coding and decoding
US20140098883A1 (en) Method and apparatus for video coding
US20150245063A1 (en) Method and apparatus for video coding
JP2015535405A (ja) ビデオコーディングのための方法と装置
WO2013159330A1 (fr) Appareil, et procédé et programme d'ordinateur pour codage et décodage vidéo
WO2013113134A1 (fr) Appareil, procédé et programme informatique de codage et décodage pour vidéo
WO2013159300A1 (fr) Appareil, procédé et programme informatique de codage et décodage vidéo

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12867716

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12867716

Country of ref document: EP

Kind code of ref document: A1