WO2013159300A1

WO2013159300A1 - An apparatus, a method and a computer program for video coding and decoding

Info

Publication number: WO2013159300A1
Application number: PCT/CN2012/074691
Authority: WO
Inventors: Lulu CHEN; Miska Hannuksela
Original assignee: Nokia Corporation
Priority date: 2012-04-25
Filing date: 2012-04-25
Publication date: 2013-10-31

Abstract

There is disclosed a method comprising receiving samples representing pixels of a block of a depth related information of a part of a picture. The depth map comprises samples representing a depth map in a first direction and in a second direction. A position for a sub-block in the depth map in the first direction is selected. The sub-block comprises samples in the first direction and in the second direction. A position for the sub-block in the depth map in the second is determined direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction. A prediction of a location of an edge within the sub- block is formed on the basis of samples of the block inside the sub-block. There is also disclosed apparatuses and computer program products implementing the method.

Description

AN APPARATUS, A METHOD AND A COMPUTER

PROGRAM FOR VIDEO CODING AND DECODING

TECHNICAL FIELD

The present invention relates to an apparatus, a method and a computer program for video coding and decoding.

BACKGROUND INFORMATION

This section is intended to provide a background or context to the invention that is recited in the claims. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

Various technologies for providing three-dimensional (3D) video content are currently investigated and developed. Especially, intense studies have been focused on various multiview applications wherein a viewer is able to see only one pair of stereo video from a specific viewpoint and another pair of stereo video from a different viewpoint. One approach for such multiview applications has turned out to be such wherein only a limited number of input views, e.g. a mono or a stereo video plus some supplementary data, is provided to a decoder side and all required views are then rendered (i.e. synthesized) locally by the decoder to be displayed on a display.

A variety of 3D display technologies have been developed in recent years, including autostereoscopic glasses-free displays and stereoscopic displays based on polarized or shutter glasses. In addition to the need of glasses, stereoscopic or multiview 3D displays often differ in the comfortable disparity between the views. Furthermore, the preferred view separation also depends on the viewing distance and individual preferences. Autostereoscopic displays may require even several tens of views as input. Displaying-time view synthesis may therefore be needed both for adjustment of the disparity between the rendered views for optimal viewing experience and for generation of an appropriate number of views for autostereoscopic displays.

Several technologies for view rendering are available, and for example, depth image- based rendering (DIBR) has shown to be a competitive alternative. A typical implementation of DIBR takes stereoscopic video and corresponding depth information with stereoscopic baseline as input and synthesizes a number of virtual views between the two input views. Thus, DIBR algorithms may also enable extrapolation of views that are outside the two input views and not in between them. Similarly, DIBR algorithms may enable view synthesis from a single view of texture and the respective depth view.

In the encoding of 3D video content, video compression systems, such as Advanced Video Coding standard H.264/AVC or the Multiview Video Coding MVC extension of

H.264/AVC can be used. However, the intra prediction for specified in H.264/AVC or MVC may not be optimal for video coding systems utilizing depth or disparity information.

SUMMARY

Some embodiments proceed from the consideration that a depth block can be divided into smaller sub-blocks and the location of each sub-block may be selected so that there is a depth edge on at least one side of the sub-block. In some embodiments the sub-block is located in such a way that the center of one side of the sub-block is at or near a detected edge position in the depth block. In some embodiments a two-step approach for coding/decoding a location of a depth boundary is utilized. In a first step, the depth block being encoded/decoded is split into sub-blocks (e.g. 4x4 blocks) and the location of these sub-blocks is selected based on the neighboring blocks or earlier sub-blocks to be estimated to cover a depth boundary. In a second step, the position of the depth boundary within each sub-block is encoded/decoded through coding the difference of a predicted depth boundary position and an actual depth boundary position.

According to a first aspect of the invention, there is provided a method comprising: receiving samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map in a first direction and in a second direction;

selecting a position for a sub-block in the depth map in the first direction, the sub-block comprising samples in the first direction and in the second direction;

determining a position for the sub-block in the depth map in the second direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction; and forming a prediction of a location of an edge within the sub-block on the basis of samples of the block inside the sub-block.

According to a second aspect of the invention, there is provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:

receive samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction;

select a position for a sub-block in the depth map in the first direction, the sub-block comprising samples in the first direction and in the second direction;

determine a position for the sub-block in the depth map in the second direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction; and

form a prediction of a location of an edge within the sub-block on the basis of samples of the block inside the sub-block.

According to a third aspect of the invention, there is provided a computer

program product including one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to at least perform the following: receive samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction;

form a prediction of a location of an edge within the sub-block on the basis of samples of the block inside the sub-block. According to a fourth aspect of the invention there is provided an apparatus comprising: means for receiving samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction;

means for selecting a position for a sub-block in the depth map in the first direction, the sub-block comprising samples in the first direction and in the second direction;

means for determining a position for the sub-block in the depth map in the second direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction; and

means for forming a prediction of a location of an edge within the sub-block on the basis of samples of the block inside the sub-block.

According to a fifth aspect of the invention, there is provided a method comprising: receiving encoded samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map in a first direction and in a second direction;

determining a position for the sub-block in the depth map in the second direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction; and

decoding the encoded location of the edge within the sub-block. According to a sixth aspect of the invention, there is provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:

receive encoded samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction; select a position for a sub-block in the depth map in the first direction, the sub-block comprising samples in the first direction and in the second direction;

determine a position for the sub-block in the depth map in the second direction by using set of samples neighboring a first border of the sub-block at the selected position in the first direction; and

decode the encoded location of the edge within the sub-block.

According to a seventh aspect of the invention, there is provided a computer program product including one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to at least perform the following:

receive encoded samples representing pixels of a block of a depth related information of part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction;

decode the encoded location of the edge within the sub-block.

According to an eight aspect of the invention, there is provided an apparatus comprising: means for receiving encoded samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction;

means for decoding the encoded location of the edge within the sub-block. According to a ninth aspect of the invention, there is provided a video coder configured for:

receiving samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map in a first direction and in a second direction;

forming a prediction of a location of an edge within the sub-block on the basis of samples of the block inside the sub-block.

According to a tenth aspect of the invention, there is provided a video decoder configured for:

receiving encoded samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map in a first direction and in a second direction;

decoding the encoded location of the edge within the sub-block.

DESCRIPTION OF THE DRAWINGS

For better understanding of the present invention, reference will now be made by way of example to the accompanying drawings in which:

Figure 1 shows a simplified 2D model of a stereoscopic camera setup.;

Figure 2 shows a simplified model of a multiview camera setup;

Figure 3 shows a simplified model of a multiview autostereoscopic display (ASD); Figure 4 shows a simplified model of a DIBR-based 3DV system;

Figures 5 and 6 show an example of a TOF-based depth estimation system;

Figure 7 shows spatial neighborhood of the currently coded block serving as the candidates for intra prediction in H.264/AVC;

Figure 8 shows an example of an embodiment of a depth map encoder as a simplified block diagram;

Figure 9 shows an example of an embodiment of a depth map decoder as a simplified block diagram;

Figure 10 shows schematically an electronic device suitable for employing some embodiments of the invention;

Figure 11 shows schematically a user equipment suitable for employing some embodiments of the invention;

Figure 12 further shows schematically electronic devices employing embodiments of the invention connected using wireless and wired network connections;

Figures 13a— 13c show an example of a Wedgelet partition of a block;

Figures 14a shows an example of intra prediction of a Wedgelet partition when the type of the reference block above the current block is a Wedgelet partition;

Figures 14b shows an example of intra prediction of a Wedgelet partition when the type of the reference block above the current block is a regular intra direction;

Figure 15a shows an example of constant partition value prediction of block partitions from adjacent samples of neighboring blocks;

Figure 15b shows a cross section of the block of Figure 15a showing relation between different constant partition value types;

Figure 16 depicts an example of depth map information of a block and part of depth map information of two neighboring blocks above and to the left of the block;

Figure 17 shows examples of depth edges and their binarization;

Figure 18 shows a high level flow chart of an embodiment of an encoder capable of encoding texture views and depth views; and

Figure 19 shows a high level flow chart of an embodiment of a decoder capable of decoding texture views and depth views. DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS OF THE

INVENTION

In order to understand the various aspects of the invention and the embodiments related thereto, the following describes briefly some closely related aspects of video coding.

Some key definitions, bitstream and coding structures, and concepts of H.264/AVC are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented. The aspects of the invention are not limited to H.264/AVC, but rather the description is given for one possible basis on top of which the invention may be partly or fully realized.

The H.264/AVC standard was developed by the Joint Video Team (JVT) of the Video

Coding Experts Group (VCEG) of the Telecommunications Standardisation Sector of

International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of International Standardisation Organisation (ISO) / International Electrotechnical Commission (IEC). The H.264/AVC standard is published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.264 and ISO/IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There have been multiple versions of the H.264/AVC standard, each integrating new extensions or features to the specification. These extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

Similarly to many earlier video coding standards, the bitstream syntax and semantics as well as the decoding process for error-free bitstreams are specified in H.264/AVC. The encoding process is not specified, but encoders must generate conforming bitstreams. Bitstream and decoder conformance can be verified with the Hypothetical Reference Decoder (HRD), which is specified in Annex C of H.264/ AVC. The standard contains coding tools that help in coping with transmission errors and losses, but the use of the tools in encoding is optional and no decoding process has been specified for erroneous bitstreams.

The elementary unit for the input to an H.264/AVC encoder and the output of an

H.264/ AVC decoder is a picture. A picture may either be a frame or a field. A frame typically comprises a matrix of luma samples and corresponding chroma samples. A field is a set of alternate sample rows of a frame and may be used as encoder input, when the source signal is interlaced. A macroblock (MB) is a 16x16 block of luma samples and the corresponding blocks of chroma samples. A block has boundary samples, which consist of the samples at the top-most and bottom-most rows of samples and at the left-most and right-most columns of samples.

Boundary samples adjacent to another block being coded or decoded may be used for example in intra prediction. Chroma pictures may be subsampled when compared to luma pictures. For example, in the 4:2:0 sampling pattern the spatial resolution of chroma pictures is half of that of the luma picture along both coordinate axes and consequently a macroblock contains one 8x8 block of chroma samples per each chroma component. A picture is partitioned to one or more slice groups, and a slice group contains one or more slices. A slice consists of an integer number of macroblocks ordered consecutively in the raster scan within a particular slice group.

The elementary unit for the output of an H.264/AVC encoder and the input of an

H.264/AVC decoder is a Network Abstraction Layer (NAL) unit. Decoding of partially lost or corrupted NAL units is typically difficult. For transport over packet-oriented networks or storage into structured files, NAL units are typically encapsulated into packets or similar structures. A bytestream format has been specified in H.264/AVC for transmission or storage environments that do not provide framing structures. The bytestream format separates NAL units from each other by attaching a start code in front of each NAL unit. To avoid false detection of NAL unit boundaries, encoders run a byte-oriented start code emulation prevention algorithm, which adds an emulation prevention byte to the NAL unit payload if a start code would have occurred otherwise. In order to enable straightforward gateway operation between packet- and stream- oriented systems, start code emulation prevention is performed always regardless of whether the bytestream format is in use or not.

H.264/AVC, as many other video coding standards, allows splitting of a coded picture into slices. In-picture prediction is disabled across slice boundaries. Thus, slices can be regarded as a way to split a coded picture into independently decodable pieces, and slices are therefore elementary units for transmission.

Some profiles of H.264/AVC enable the use of up to eight slice groups per coded picture. When more than one slice group is in use, the picture is partitioned into slice group map units, which are equal to two vertically consecutive macroblocks when the macroblock-adaptive frame- field (MBAFF) coding is in use and equal to a macroblock otherwise. The picture parameter set contains data based on which each slice group map unit of a picture is associated with a particular slice group. A slice group can contain any slice group map units, including non- adjacent map units. When more than one slice group is specified for a picture, the flexible macroblock ordering (FMO) feature of the standard is used.

In H.264/AVC, a slice consists of one or more consecutive macroblocks (or macroblock pairs, when MBAFF is in use) within a particular slice group in raster scan order. If only one slice group is in use, H.264/AVC slices contain consecutive macroblocks in raster scan order and are therefore similar to the slices in many previous coding standards. In some profiles of H.264/AVC slices of a coded picture may appear in any order relative to each other in the bitstream, which is referred to as the arbitrary slice ordering (ASO) feature. Otherwise, slices must be in raster scan order in the bitstream.

NAL units consist of a header and payload. The NAL unit header indicates the type of the

NAL unit and whether a coded slice contained in the NAL unit is a part of a reference picture or a non-reference picture. The header for SVC and MVC NAL units additionally contains various indications related to the scalability and multiview hierarchy.

NAL units of H.264/AVC can be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are either coded slice NAL units, coded slice data partition NAL units, or VCL prefix NAL units. Coded slice NAL units contain syntax elements representing one or more coded macroblocks, each of which corresponds to a block of samples in the uncompressed picture. There are four types of coded slice NAL units: coded slice in an Instantaneous Decoding Refresh (IDR) picture, coded slice in a non-IDR picture, coded slice of an auxiliary coded picture (such as an alpha plane) and coded slice extension (for SVC slices not in the base layer or MVC slices not in the base view). A set of three coded slice data partition NAL units contains the same syntax elements as a coded slice. Coded slice data partition A comprises macroblock headers and motion vectors of a slice, while coded slice data partition B and C include the coded residual data for intra macroblocks and inter macroblocks, respectively. It is noted that the support for slice data partitions is only included in some profiles of

H.264/AVC. A VCL prefix NAL unit precedes a coded slice of the base layer in SVC and MVC bitstreams and contains indications of the scalability hierarchy of the associated coded slice.

A non-VCL NAL unit of H.264/AVC may be of one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of stream NAL unit, or a filler data NAL unit. Parameter sets are essential for the reconstruction of decoded pictures, whereas the other non-VCL NAL units are not necessary for the reconstruction of decoded sample values and serve other purposes presented below.

Many parameters that remain unchanged through a coded video sequence are included in a sequence parameter set. In addition to the parameters that are essential to the decoding process, the sequence parameter set may optionally contain video usability information (VUI), which includes parameters that are important for buffering, picture output timing, rendering, and resource reservation. A picture parameter set contains such parameters that are likely to be unchanged in several coded pictures. No picture header is present in H.264/AVC bitstreams but the frequently changing picture-level data is repeated in each slice header and picture parameter sets carry the remaining picture-level parameters. H.264/AVC syntax allows many instances of sequence and picture parameter sets, and each instance is identified with a unique identifier. Each slice header includes the identifier of the picture parameter set that is active for the decoding of the picture that contains the slice, and each picture parameter set contains the identifier of the active sequence parameter set. Consequently, the transmission of picture and sequence parameter sets does not have to be accurately synchronized with the transmission of slices. Instead, it is sufficient that the active sequence and picture parameter sets are received at any moment before they are referenced, which allows transmission of parameter sets using a more reliable transmission mechanism compared to the protocols used for the slice data. For example, parameter sets can be included as a parameter in the session description for

H.264/AVC Real-time Transport Protocol (RTP) sessions. If parameter sets are transmitted in- band, they can be repeated to improve error robustness.

A SEI NAL unit of H.264/AVC contains one or more SEI messages, which are not required for the decoding of output pictures but assist in related processes, such as picture output timing, rendering, error detection, error concealment, and resource reservation. Several SEI messages are specified in H.264/AVC, and the user data SEI messages enable organizations and companies to specify SEI messages for their own use. H.264/AVC contains the syntax and semantics for the specified SEI messages but no process for handling the messages in the recipient is defined. Consequently, encoders are required to follow the H.264/AVC standard when they create SEI messages, and decoders conforming to the H.264/AVC standard are not required to process SEI messages for output order conformance. One of the reasons to include the syntax and semantics of SEI messages in H.264/AVC is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.

A coded picture in H.264/AVC consists of the VCL NAL units that are required for the decoding of the picture. A coded picture can be a primary coded picture or a redundant coded picture. A primary coded picture is used in the decoding process of valid bitstreams, whereas a redundant coded picture is a redundant representation that should only be decoded when the primary coded picture cannot be successfully decoded.

In H.264/AVC, an access unit consists of a primary coded picture and those NAL units that are associated with it. The appearance order of NAL units within an access unit is constrained as follows. An optional access unit delimiter NAL unit may indicate the start of an access unit. It is followed by zero or more SEI NAL units. The coded slices or slice data partitions of the primary coded picture appear next, followed by coded slices for zero or more redundant coded pictures.

An access unit in MVC is defined to be a set of NAL units that are consecutive in decoding order and contain exactly one primary coded picture consisting of one or more view components. In addition to the primary coded picture, an access unit may also contain one or more redundant coded pictures, one auxiliary coded picture, or other NAL units not containing slices or slice data partitions of a coded picture. The decoding of an access unit always results in one decoded picture consisting of one or more decoded view components. In other words, an access unit in MVC contains the view components of the views for one output time instance.

A view component in MVC is referred to as a coded representation of a view in a single access unit.

Inter-view prediction may be used in MVC and refers to prediction of a view component from decoded samples of different view components of the same access unit. In MVC, inter-view prediction is realized similarly to inter prediction. For example, inter-view reference pictures are placed in the same reference picture list(s) as reference pictures for inter prediction, and a reference index as well as a motion vector are coded or inferred similarly for inter-view and inter reference pictures. An anchor picture in MVC is a coded picture in which all slices may reference only slices within the same access unit, i.e., inter-view prediction may be used, but no inter prediction is used, and all following coded pictures in output order do not use inter prediction from any picture prior to the coded picture in decoding order. Inter-view prediction may be used for IDR view components that are part of a non-base view. A base view in MVC is a view that has the minimum value of view order index in a coded video sequence. The base view can be decoded independently of other views and does not use inter-view prediction. The base view can be decoded by H.264/AVC decoders supporting only the single-view profiles, such as the Baseline Profile or the High Profile of H.264/AVC.

In the MVC standard, many of the sub-processes of the MVC decoding process use the respective sub-processes of the H.264/AVC standard by replacing term "picture", "frame", and "field" in the sub-process specification of the H.264/AVC standard by "view component", "frame view component", and "field view component", respectively. Likewise, terms "picture", "frame", and "field" are often used in the following to mean "view component", "frame view component", and "field view component", respectively.

A coded video sequence is defined to be a sequence of consecutive access units in decoding order from an IDR access unit, inclusive, to the next IDR access unit, exclusive, or to the end of the bitstream, whichever appears earlier.

A group of pictures (GOP) and its characteristics may be defined as follows. A GOP can be decoded regardless of whether any previous pictures were decoded. An open GOP is such a group of pictures in which pictures preceding the initial intra picture in output order might not be correctly decodable when the decoding starts from the initial intra picture of the open GOP. In other words, pictures of an open GOP may refer (in inter prediction) to pictures belonging to a previous GOP. An H.264/AVC decoder can recognize an intra picture starting an open GOP from the recovery point SEI message in an H.264/AVC bitstream. A closed GOP is such a group of pictures in which all pictures can be correctly decoded when the decoding starts from the initial intra picture of the closed GOP. In other words, no picture in a closed GOP refers to any pictures in previous GOPs. In H.264/AVC, a closed GOP starts from an IDR access unit. As a result, closed GOP structure has more error resilience potential in comparison to the open GOP structure, however at the cost of possible reduction in the compression efficiency. Open GOP coding structure is potentially more efficient in the compression, due to a larger flexibility in selection of reference pictures.

The bitstream syntax of H.264/AVC indicates whether a particular picture is a reference picture for inter prediction of any other picture. Pictures of any coding type (I, P, B) can be reference pictures or non-reference pictures in H.264/AVC. The NAL unit header indicates the type of the NAL unit and whether a coded slice contained in the NAL unit is a part of a reference picture or a non- reference picture.

There is an ongoing video coding standardization project for specifying a High Efficiency Video Coding (HEVC) standard. Many of the key definitions, bitstream and coding structures, and concepts of HEVC are the same as or similar to those of H.264/AVC. Some key definitions, bitstream and coding structures, and concepts of HEVC are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein some embodiments may be implemented. The aspects of the invention are not limited to HEVC, but rather the description is given for one possible basis on top of which the invention may be partly or fully realized.

A basic coding unit in a draft HEVC standard is a treeblock. A treeblock is an NxN block of luma samples and two corresponding blocks of chroma samples of a picture that has three sample arrays, or an NxN block of samples of a monochrome picture or a picture that is coded using three separate colour planes. A treeblock may be partitioned for different coding and decoding processes. A treeblock partition is a block of luma samples and two corresponding blocks of chroma samples resulting from a partitioning of a treeblock for a picture that has three sample arrays or a block of luma samples resulting from a partitioning of a treeblock for a monochrome picture or a picture that is coded using three separate colour planes. Each treeblock is assigned a partition signalling to identify the block sizes for intra or inter prediction and for transform coding. The partitioning is a recursive quadtree partitioning. The root of the quadtree is associated with the treeblock. The quadtree is split until a leaf is reached, which is referred to as the coding node. The coding node is the root node of two trees, the prediction tree and the transform tree. The prediction tree specifies the position and size of prediction blocks. The prediction tree and associated prediction data are referred to as a prediction unit. The transform tree specifies the position and size of transform blocks. The transform tree and associated transform data are referred to as a transform unit. The splitting information for luma and chroma is identical for the prediction tree and may or may not be identical for the transform tree. The coding node and the associated prediction and transform units form together a coding unit.

Video pictures can be divided into coding units (CU) covering the area of the picture. A coding unit consists of one or more prediction units (PU) defining the prediction process for the samples within the coding unit and one or more transform units (TU) defining the prediction error coding process for the samples in the coding unit. A coding unit may consist of a square block of samples with a size selectable from a predefined set of possible coding unit sizes. A coding unit with the maximum allowed size may be named as a largest coding unit (LCU) and the video picture may be divided into non-overlapping largest coding units. A largest coding unit can further be split into a combination of smaller coding units, e.g. by recursively splitting the largest coding unit and resultant coding units. Each resulting coding unit may have at least one prediction unit and at least one transform unit associated with it. Each prediction unit and transform unit can further be split into smaller prediction units and transform units in order to increase granularity of the prediction and prediction error coding processes, respectively. Each prediction unit has prediction information associated with it defining what kind of a prediction is to be applied for the pixels within that prediction unit (e.g. motion vector information for inter predicted prediction units and intra prediction directionality information for intra predicted prediction units). Similarly, each transform unit is associated with information describing the prediction error decoding process for the samples within the transform unit (including e.g. DCT coefficient information). It may be signalled at coding unit level whether prediction error coding is applied or not for each coding unit. In the case there is no prediction error residual associated with the coding unit, it can be considered there are no transform units for the coding unit. The division of the image into coding units, and division of coding units into prediction units and transform units may be signalled in a bitstream allowing the decoder to reproduce the intended structure of these units.

In a draft HEVC standard, pictures are divided into slices and tiles. A slice may be a sequence of treeblocks but (when referring to a so-called fine granular slice) may also have its boundary within a treeblock at a location where a transform unit and prediction unit coincide. Treeblocks within a slice are coded and decoded in a raster scan order. For the primary coded picture, the division of each picture into slices is a partitioning. In a draft HEVC standard, a tile is defined as an integer number of treeblocks co- occurring in one column and one row, ordered consecutively in the raster scan within the tile. For the primary coded picture, the division of each picture into tiles is a partitioning. Tiles are ordered consecutively in the raster scan within the picture. Although a slice contains treeblocks that are consecutive in the raster scan within a tile, these treeblocks are not necessarily consecutive in the raster scan within the picture. Slices and tiles need not contain the same sequence of treeblocks. A tile may comprise treeblocks contained in more than one slice.

Similarly, a slice may comprise treeblocks contained in several tiles.

Many hybrid video codecs, including H.264/AVC and HEVC, encode video information in two phases. In the first phase, pixel or sample values in a certain picture area or "block" are predicted. These pixel or sample values can be predicted, for example, by motion compensation mechanisms, which involve finding and indicating an area in one of the previously encoded video frames that corresponds closely to the block being coded. Additionally, pixel or sample values can be predicted by spatial mechanisms which involve finding and indicating a spatial region relationship.

Prediction approaches using image information from a previously coded image can also be called as inter prediction methods which may be also referred to as temporal prediction and motion compensation. Prediction approaches using image information within the same image can also be called as intra prediction methods.

The second phase is one of coding the error between the predicted block of pixels or samples and the original block of pixels or samples. This may be accomplished by transforming the difference in pixel or sample values using a specified transform. This transform may be a Discrete Cosine Transform (DCT) or a variant thereof. After transforming the difference, the transformed difference is quantized and entropy encoded.

By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel or sample representation (i.e. the visual quality of the picture) and the size of the resulting encoded video representation (e.g. the file size or transmission bit rate).

The decoder reconstructs the output video by applying a prediction mechanism similar to that used by the encoder in order to form a predicted representation of the pixel or sample blocks (using the motion or spatial information created by the encoder and stored in the compressed representation of the image) and prediction error decoding (the inverse operation of the prediction error coding to recover the quantized prediction error signal in the spatial domain).

After applying pixel or sample prediction and error decoding processes the decoder combines the prediction and the prediction error signals (the pixel or sample values) to form the output video frame.

The decoder (and encoder) may also apply additional filtering processes in order to improve the quality of the output video before passing it for display and/or storing as a prediction reference for the forthcoming pictures in the video sequence.

A texture view refers to a view that represents ordinary video content, for example has been captured using an ordinary camera, and is usually suitable for rendering on a display. A texture view typically comprises pictures having three components, one luma component and two chroma components. In the following, a texture picture typically comprises all its component pictures or color components unless otherwise indicated for example with terms luma texture picture and chroma texture picture.

Depth-enhanced video refers to texture video having one or more views associated with depth video having one or more depth views. A number of approaches may be used for representing of depth-enhanced video, including the use of video plus depth (V+D), multiview video plus depth (MVD), and layered depth video (LDV). In the video plus depth (V+D) representation, a single view of texture and the respective view of depth are represented as sequences of texture picture and depth pictures, respectively. The MVD representation contains a number of texture views and respective depth views. In the LDV representation, the texture and depth of the central view are represented conventionally, while the texture and depth of the other views are partially represented and cover only the dis-occluded areas required for correct view synthesis of intermediate views.

Depth-enhanced video may be coded in a manner where texture and depth are coded independently of each other. For example, texture views may be coded as one MVC bitstream and depth views may be coded as another MVC bitstream. Alternatively depth-enhanced video may be coded in a manner where texture and depth are jointly coded. When joint coding texture and depth views is applied for a depth-enhanced video representation, some decoded samples of a texture picture or data elements for decoding of a texture picture are predicted or derived from some decoded samples of a depth picture or data elements obtained in the decoding process of a depth picture. Alternatively or in addition, some decoded samples of a depth picture or data elements for decoding of a depth picture are predicted or derived from some decoded samples of a texture picture or data elements obtained in the decoding process of a texture picture.

Many video encoders utilize the Lagrangian cost function to find rate-distortion optimal coding modes, for example the desired macroblock mode and associated motion vectors. This type of cost function uses a weighting factor or λ to tie together the exact or estimated image distortion due to lossy coding methods and the exact or estimated amount of information required to represent the pixel/sample values in an image area. The Lagrangian cost function may be represented by the equation:

0=ϋ+λΚ

where C is the Lagrangian cost to be minimised, D is the image distortion (for example, the mean-squared error between the pixel/sample values in original image block and in coded image block) with the mode and motion vectors currently considered, λ is a Lagrangian coefficient and R is the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

In intra prediction information of the same picture or frame than the block to be encoded/decoded is used as a reference for intra prediction. The reference may be, for example, another block or a part of another block in the same picture or frame. Intra prediction may be based on spatial sample prediction, in which another part of the same reconstructed or decoded picture or frame is used to predict sample values of the block to be encoded/decoded.

Alternatively or in addition, intra prediction may be based on transform-domain prediction, in which transform coefficients associated with another part of the same picture or frame is used to predict transform coefficients of the block to be encoded/decoded.

In H.264/AVC, intra prediction may be done for block sizes 4x4, 8x8, or 16x16 as determined by the encoder during a mode selection. H.264/AVC intra prediction generally utilizes the sample values at the block boundary of the neighboring blocks on left, top, and top- right of the block being encoded/decoded. Intra prediction modes include vertical prediction from the neighboring samples on top, horizontal prediction from the neighboring samples on left, and a few diagonal prediction directions. Furthermore, in DC prediction mode, the mean value of the neighboring boundary samples is used as a prediction value. A plane prediction mode, available for some block sizes, a linear plane is function is used with input of the upper and left- hand boundary samples from the neighboring blocks.

In a draft HEVC standard, intra prediction is similar to although more sophisticated than that in H.264/AVC. HEVC includes a greater number of prediction directions compared to that of H.264/AVC. Intra sample prediction process in HEVC is performed by extrapolating sample values from the reconstructed reference samples utilizing a given directionality. In order to simplify the process, all sample locations within one prediction block are projected to a single reference row or column depending on the directionality of the selected prediction mode. A predicted sample within the block being encoded/decoded is obtained by projecting its location to a reference row or column applying the selected prediction direction and interpolating a value for the sample at 1/32 pixel accuracy from reference samples at the reference row/column. The planar mode in HEVC uses an average of two linear predictions derived from the reference row and column. Reference samples may also undergo smoothing in some intra prediction modes and block sizes. The predicted samples in some modes, such as DC, vertical prediction and horizontal prediction, may undergo boundary smoothing to avoid false block boundaries in the prediction signal. Due to the treeblock coding/decoding order a block on the below- left may be available and used in HEVC intra prediction modes in addition to the blocks on left, above, and above right. When a coding unit is split into multiple transform units, the intra prediction may be applied for transform units one at a time rather than applying the intra prediction for prediction units. Intra prediction information may be indicated per PU and the TUs inside the same PU share the same intra prediction mode.

The Multiview Video Coding (MVC) extension of H.264 referred above enables to implement a multiview functionality at the decoder, thereby allowing the development of three- dimensional (3D) multiview applications.

Next, for better understanding the embodiments of the invention, some aspects of 3D multiview applications and the concepts of depth and disparity information closely related thereto are described briefly.

Stereoscopic video content consists of pairs of offset images that are shown separately to the left and right eye of the viewer. These offset images are captured with a specific stereoscopic camera setup and it assumes a particular stereo baseline distance between cameras. Figure 1 shows a simplified 2D model of such stereoscopic camera setup. In Fig. 1, CI and C2 refer to cameras of the stereoscopic camera setup, more particularly to the center locations of the cameras, b is the distance between the centers of the two cameras (i.e. the stereo baseline), f is the focal length of the cameras and X is an object in the real 3D scene that is being captured. The real world object X is projected to different locations in images captured by the cameras CI and C2, these locations being xl and x2 respectively. The horizontal distance between xl and x2 in absolute coordinates of the image is called disparity. The images that are captured by the camera setup are called stereoscopic images, and the disparity presented in these images creates or enhances the illusion of depth. For enabling the images to be shown separately to the left and right eye of the viewer, specific 3D glasses may be required to be used by the viewer. Adaptation of the disparity is a key feature for adjusting the stereoscopic video content to be comfortably viewable on various displays.

However, disparity adaptation is not a straightforward process. It requires either having additional camera views with different baseline distance (i.e., b is variable) or rendering of virtual camera views which were not available in real world. Figure 2 shows a simplified model of such multiview camera setup that suits to this solution. This setup is able to provide stereoscopic video content captured with several discrete values for stereoscopic baseline and thus allow stereoscopic display to select a pair of cameras that suits to the viewing conditions.

A more advanced approach for 3D vision is having a multiview autostereoscopic display (ASD) 300 that does not require glasses. The ASD emits more than one view at a time but the emitting is localized in the space in such a way that a viewer sees only a stereo pair from a specific viewpoint, as illustrated in Figure 3, wherein the house is seen in the middle of the view when looked at the right-most viewpoint. Moreover, the viewer is able see another stereo pair from a different viewpoint, e.g. in Fig. 3 the house is seen at the right border of the view when looked at the left-most viewpoint. Thus, motion parallax viewing is supported if consecutive views are stereo pairs and they are arranged properly. The ASD technologies may be capable of showing for example 52 or more different images at the same time, of which only a stereo pair is visible from a specific viewpoint. This supports multiuser 3D vision without glasses, for example in a living room environment.

The above-described stereoscopic and ASD applications require multiview video to be available at the display. The MVC extension of H.264/AVC video coding standard allows the multiview functionality at the decoder side. The base view of MVC bitstreams can be decoded by any H.264/AVC decoder, which facilitates introduction of stereoscopic and multiview content into existing services. MVC allows inter-view prediction, which can result into significant bitrate saving compared to independent coding of all views, depending on how correlated the adjacent views are. However, the bitrate of MVC coded video is typically proportional to the number of views. Considering that ASD may require 52 views, for example, as input, the total bitrate for such number of views will challenge the constraints of the available bandwidth.

Consequently, it has been found that a more feasible solution for such multiview application is to have a limited number of input views, e.g. a mono or a stereo view plus some supplementary data, and to render (i.e. synthesize) all required views locally at the decoder side. From several available technologies for view rendering, depth image-based rendering (DIBR) has shown to be a competitive alternative.

A simplified model of a DIBR-based 3DV system is shown in Figure 4. The input of a 3D video codec comprises a stereoscopic video and corresponding depth information with stereoscopic baseline bO. Then the 3D video codec synthesizes a number of virtual views between two input views with baseline (bi < bO). DIBR algorithms may also enable extrapolation of views that are outside the two input views and not in between them. Similarly, DIBR algorithms may enable view synthesis from a single view of texture and the respective depth view. However, in order to enable DIBR-based multiview rendering, texture data should be available at the decoder side along with the corresponding depth data.

In such 3DV system, depth information is produced at the encoder side in a form of depth pictures (also known as depth maps) for each video frame. A depth map is an image with per- pixel depth information. Each sample in a depth map represents the distance of the respective texture sample from the plane on which the camera lies. In other words, if the z axis is along the shooting axis of the cameras (and hence orthogonal to the plane on which the cameras lie), a sample in a depth map represents the value on the z axis.

Depth information can be obtained by various means. For example, depth of the 3D scene may be computed from the disparity registered by capturing cameras. A depth estimation algorithm takes a stereoscopic view as an input and computes local disparities between the two offset images of the view. Each image is processed pixel by pixel in overlapping blocks, and for each block of pixels a horizontally localized search for a matching block in the offset image is performed. Once a pixel-wise disparity is computed, the corresponding depth value z is calculated by equation (2): d + Ad ^<2)·

where f is the focal length of the camera and b is the baseline distance between cameras, as shown in Figure 1. Further, d refers to the disparity observed between the two cameras, and the camera offset Ad reflects a possible horizontal misplacement of the optical centers of the two cameras.

Alternatively, or in addition to the above-described stereo view depth estimation, the depth value may be obtained using the time-of-flight (TOF) principle. Figures 5 and 6 show an example of a TOF-based depth estimation system. The camera is provided with a light source, for example an infrared emitter, for illuminating the scene. Such an illuminator may be arranged to produce an intensity modulated electromagnetic emission for a frequency between e.g. 10-100 MHz, which may require LEDs or laser diodes to be used. Infrared light is typically used to make the illumination unobtrusive. The light reflected from objects in the scene is detected by an image sensor, which is modulated synchronously at the same frequency as the illuminator. The image sensor is provided with optics; a lens gathering the reflected light and an optical bandpass filter for passing only the light with the same wavelength as the illuminator, thus helping to suppress background light. The image sensor measures for each pixel the time the light has taken to travel from the illuminator to the object and back. The distance to the object is represented as a phase shift in the illumination modulation, which can be determined from the sampled data simultaneously for each pixel in the scene.

In contrast to the stereo view depth estimation, the accuracy of the TOF-based depth estimation is mostly content independent. For example, it is not suffering from the lack of textural appearance in the content. However, currently available TOF cameras have low pixel resolution sensors and the depth estimation is heavily influenced by random and systematic noise.

In the case of depth-enhanced multiview coding, the view synthesis can be utilized in an encoding loop of the encoder and in the decoding loop of the decoder, thus providing a view synthesis prediction (VSP). A view synthesis picture (a reference component) may be

synthesized from coded texture views and depth views and may contain samples that may be used for the view synthesis prediction. A view synthesis picture may also be referred to as synthetic reference component, which may be defined to contain samples that may be used for view synthesis prediction. A synthetic reference component may be used as reference picture for view synthesis prediction but is typically not output or displayed. A view synthesis picture is typically generated for the same camera location assuming the same camera parameters as for the picture being coded or decoded.

The encoder according to some example embodiments of the present invention may include one or more of the following operations for coding of intra-coded depth blocks. It should be noted here that similar principles are also applicable at a decoder side for decoding of intra- coded depth blocks. While many of example embodiments are described with reference to depth, it is to be understood that the example embodiments could use disparity or parallax in place of depth. Many of the example embodiments are described with reference to term block, which may be for example a macroblock similar to that used in H.264/AVC, a treeblock similar to that used in a draft HEVC standard, or anything alike. Likewise, many of the example embodiments may refer to a term macroblock, but a treeblock as in HEVC or anything alike could be used instead.

Depth maps are mainly characterized by sharp edges (which represent object borders) and large areas of nearly constant or slowly varying sample values (which represent object areas). While intra prediction and transform coding of modern video coding schemes, such as

H.264/AVC and HEVC, are well-suited for nearly constant regions, they can result in significant coding artifacts at sharp edges, such as ringing and blocking. Such coding artifacts become visible when the decoded depth maps are used in synthesizing intermediate views. Such coding artifacts may also reduce the compression efficiency of depth-based texture coding tools, such as view synthesis prediction (VSP) and depth-based motion vector prediction (D-MVP).

In the following, example embodiments for encoding and/or decoding a location of a depth boundary will be described in more detail. In some embodiments the encoding/decoding of a location of a depth boundary may be regarded as a two-step approach. In the first step, the depth block being encoded/decoded is split into smaller sub-blocks and the location of these sub- blocks is selected based on the neighboring blocks or earlier sub-blocks to be estimated to cover a depth boundary. In the second step, the position of the depth boundary within each sub-block is encoded/decoded through coding the difference of a predicted depth boundary position and an actual depth boundary position. In some embodiments the size of the depth block is 16x16 blocks and the size of the sub- blocks is 4x4 blocks. However, it should be noted that the size of the depth block and the size of the sub-blocks may also be different from these sizes. Some other examples of the sizes of the depth blocks are 8x8 blocks, 32x32 blocks, 64x64 blocks, etc. Some other examples of the sizes of the sub-blocks are 6x6 blocks, 8x8 blocks, 4x6 blocks, 6x4 blocks, 4x8 blocks, 8x4 blocks, etc.

An example of a depth map encoder 800 is depicted in Figure 8 as a simplified block diagram. In some embodiments the operation of the depth map encoder 800 may comprise one or more of the following operations. Similar operations may also be applicable by the depth map decoder 900 when it decodes received encoded depth map data. The depth map encoder 800 may comprise e.g. an edge determinator 802 to detect an edge position within the same depth map block and/or in a neighboring depth map block. The depth map encoder 800 may further comprise a sub-block selector 804 to select a location of sub-blocks within the current depth map block when the encoder is predicting the location of an edge in the current depth map block. The depth map encoder 800 may also comprise an edge predictor 806 to predict the location of an edge within the sub-blocks. The depth map encoder 800 may comprise memory for storing block data or may have access to a frame buffer 808, for example, in which encoded blocks of encoded frames may have been stored during the encoding process and which block data the depth map encoder 800 may use when detecting edges in depth map blocks and also in other depth map data encoding processes.

Correspondingly, in some embodiments the depth map decoder 800 may comprise, as illustrated in Figure 9, e.g. an edge determinator 902 to detect an edge position within the same depth map block and/or in a neighboring depth map block or sub-block. The edge determinator 902 may detect edges in a neighboring depth map block or sub-block identically to how the encoder detects edges in a neighboring depth map block or sub-block, respectively, in the edge determinator 802. The depth map decoder 900 may further comprise a sub-block selector 904 to select a location of sub-blocks within the current depth map block when the decoder is predicting the location of an edge in the current depth map block. The sub-block selector 904 may select a location of a sub-block identically to how the encoder selects the location of the respective sub- block in the sub-block selector 804. The depth map decoder 900 may also comprise an edge predictor 906 to predict the location of an edge within the sub-blocks. The edge predictor 906 may receive from the entropy decoder 198 one or more codewords specifying the location and shape of the depth edge within the sub-block being decoded. The edge predictor 906 may decode the mentioned codewords and use the sub-block location received from the sub-block selector 904 and/or determined depth edges of the neighboring depth map block or sub-block as determined by the edge determinator 902 to conclude the location and shape of the depth edge within the sub-block. The depth map decoder 900 may comprise memory for storing block data or may have access to a frame buffer 908, for example, in which decoded blocks of decoded frames may have been stored during the decoding process and which block data the depth map decoder 900 may use when detecting edges in depth map blocks and also in other depth map data decoding processes.

In the following the operation of the depth map encoder 184 is described in more detail.

The depth map encoder 800 receives depth map information of a current depth map block. An example of depth map information of a block and also part of depth map information of two neighboring blocks is depicted in Figure 16. In Figure 16 the circles filled with grey colour illustrate depth map samples belonging to a first region and circles having no fill illustrate depth map samples belonging to a second region. If the current depth map block 160 is a first block of a frame or a picture to be encoded or if no neighboring block is available for intra prediction for example because they reside in a different slice, or if they were coded with a mode different from intra mode and intra prediction is constrained to take place only from neighboring intra-coded blocks, a possible edge in the current depth map block may be detected by some appropriate edge detection method. Otherwise, when the current depth map block has a neighboring block which has already been encoded and is available for intra prediction, a depth edge may be detected from the neighboring block, typically from its boundary row or column of pixels that is adjacent to the block being encoded. This may be based on the row of pixels 161 or the column of pixels 162 of the neighboring depth map block adjacent to the current depth map block e.g. as follows. The position of the edge in the neighboring block above the current block may be derived, for example, examining pixel values (samples) of the lowest line 161 of the neighboring block and when a difference between two adjacent pixels exceed a first threshold thl the edge determinator 802 may determine that there is an edge at that location.

After detecting a depth edge from the block boundary pixels of a neighboring block, the encoder may detect a depth edge within the block being encoded such a way that the depth edge within the block matches to the depth edge within the block boundary pixels of the neighboring block. In some embodiments, the depth edge detection may be done within sub-blocks such a way that the depth edge within the sub-block matches to the depth edge of the previous sub- block. In the above-mentioned depth edge detection steps, one or more appropriate depth edge detection methods may be used. For example, the depth edge may be determined based on the absolute difference between two adjacent samples, and the two samples having the greatest absolute difference within a row or a column being investigated may be determined to contain the depth edge. In another example, the depth edge classification may be done as follows. The classification may use a Sobel operator using the following two 3x3 kernels to obtain a gradient magnitude image G:

— ^¾ w

where A is the source image (the reconstructed depth image).

As sequences may have different dynamic sample value ranges in G value, G may be converted to image G' using histogram equalization. In the histogram equalization, the min and max values of G' may be set to 0 and 255, respectively. Further, a first threshold Tl and a second threshold T2 may also be set to appropriate values. The encoder decoder may examine if G'(x, y) > Tl . If so, the point (x, y) is classified to the boundary points.

In various embodiments, the encoder may encode threshold values or other parameter values for depth edge classification into the bitstream so that the decoder can perform depth edge classification identically to the encoder. In many embodiments, the depth edge detection from the neighboring blocks of the block being encoded or decoded is done identically in the encoder and decoder. In many embodiments, the depth edge detection within the block being encoded is done only by the encoder, and the encoder encodes the depth edge location within the block being encoded into the bitstream as described subsequently.

The depth map encoder 800 may use sub-blocks to encode the location of the edge in the depth map block. The edge determinator 802 may use information of a part of the neighboring block to determine the position of the edge and inform the position of the edge to the sub-block selector 804 of the encoder 800. The sub-block selector 804 may first predict the location of a sub-block so that the edge lies within the sub-block. In some embodiments the prediction of the location of the first sub-block in the current depth map block utilizes position(s) of edge(s) in the neighboring blocks. As explained earlier, this may be based on the row of pixels 161 or the column of pixels 162 of the neighboring depth map block adjacent to the current depth map block. If there is an edge 163 in the neighboring block above the current block the first sub-block 164 may be horizontally positioned so that the center of the sub-block 164 is at the position of the edge in the neighboring block or as near as possible of the position of the edge in the neighboring block above the current block. In some embodiments, if there is an edge 163 in the neighboring block above the current block the first sub-block 164 may be horizontally positioned so that the location of the sub-block 164 is selected from a subset of pixel positions (e.g. every 4^th pixel location for a 4x4 sub-block) so that the sub-block is adjacent to the position of the edge in the neighboring block. If no neighboring block is available for intra prediction or no depth edge is detected from a neighboring block available for intra prediction, the encoder may encode the location of the first sub-block in the bitstream. For example, the encoder may encode an indication of the sub-block traversal order, such as top-to-bottom or left-to-right, and an indication of the horizontal (e.g. for top-to-bottom sub-block traversal order) or the vertical (for left-to-right sub-block traversal order) location of the first sub-block within the block.

The edge predictor 806 may perform the encoding of the depth edge location and shape within a sub-block into one or more codewords that can be entropy coded. The edge predictor may take as inputs a detected depth edge from the edge determinator 802 and the selected sub- block from the sub-block selector 804. The coding or binarization scheme used to produce the codeword(s) to represent the location and shape within the sub-block may pose limitations on the edge location and shape that can be represented. The encoder may determine the location and shape of the encoded depth edge for example based on rate-distortion optimization or minimization of an error between the encoded depth edge location and the detected depth edge location, which may be derived for example as an absolute error in the edge location pixel-row- wise (e.g. for top-to-bottom sub-block traversal) or pixel-column-wise (e.g. for left-to-right sub- block traversal).

After the first sub-block 164 has been located the following sub-blocks may be positioned on the basis of a position of an edge in a previous sub-block i.e. in a sub-block which has already been positioned in the current depth map block. To do that, the edge determinator 802 may examine the pixels of the last row in the previous sub-block to detect the position of a depth edge in the previous predicted or reconstructed/decoded sub-block. Alternatively, the edge determinator 802 may examine the encoded depth edge location of the previous sub-block as determined by the edge predictor 806. On the basis of the position of the edge the sub-block selector 804 may then select a position for the next sub-block within the current depth map block. In the example of Figure 16 there is a second sub-block 165 which is positioned beneath the first sub-block 164 so that the center of the second sub-block 165 is at or near the position of the edge at the bottom of the first sub-block 164.

Operations described above may be repeated e.g. until a next sub-block would lie totally outside of the current depth map block.

Generated depth map information, such as the location of the first sub-block and the codeword(s) representing the encoded depth edge within each sub-block, may be entropy encoded by an entropy encoder 188 and transmitted and/or stored in a bitstream 187.

It may occur that two sub-blocks will be positioned beside each other in the horizontal direction if the previous sub-block does not have an edge at the bottom of the sub-block. This is illustrated with sub-blocks 166 and 167 in Figure 16. The third sub-block 166 is positioned below the second sub-block 165 so that the center of the third sub-block 166 is at the position of the edge at the bottom of the second sub-block 165. In the area of the third sub-block 166 the edge is located so that it crosses the third sub-block 166 at the left side and not at the bottom of the third sub-block 166. Hence, no edge is detected at the bottom of the third sub-block 166 wherein the edge determinator 802 may examine the sides of the third sub-block 166 to determine the location of the edge. In the example of Figure 16 the edge can be detected at the left border of the third sub-block 166, wherein the fourth sub-block 167 is positioned beside the third sub-block 166 (on the left hand side) at the same vertical location with the third sub-block 166. After that, the fourth sub-block 167 is examined to determine where the edge of the current depth map block within the area of the fourth sub-block 167 exits the fourth sub-block 167 and the fifth sub-block 168 is positioned below the fourth sub-block 167 so that the center of the fifth sub-block 168 is at the edge. In the above example the sub-blocks 164— 168 were positioned in the current depth map block from top to bottom, but it may also be possible to position the sub-blocks from left to right if there is an edge at the left border of the current depth map block.

In some embodiments, the row of pixels 161 or the column of pixels 162 are not limited to have the width or the height, respectively, of the block being encoded or decoded, but for example the row of pixels 161 may include pixels of the block located above -right of the block being encoded or decoded the column of pixels 162 may include pixels of the block located below-left of the block being encoded or decoded. Pixels not adjacent to the block being encoded/decoded may be projected on the block boundary of the block being encoded/decoded according to a prediction direction determined by the encoder and concluded or decoded from the bitstream by the decoder. Such projection may result in projected prediction samples on the right-hand side or below the block being encoded/decoded. Consequently, sub-blocks may be positioned from right to left or from bottom to top. In some embodiments sub-block traversal from right to left or bottom to top may also be enabled by a block coding/decoding order other than raster scan order.

In the following a more detailed description of an embodiment will be provided referring to the example of a depth map block of Figure 16. It is assumed here, without limiting the scope of the application, that the depth map block is a macroblock of 16x16 sample values and that the sub-blocks are of size 4x4 samples. Rectangles which are depicted with broken lines in Figure 16 illustrate samples which are used for predicting the location of a sub-block.

In some embodiments the number of samples which are used for predicting the location of a sub-block is equal to the number of samples in the sub-block in one row when the location of the sub-block is derived from an edge above or below the sub-block (i.e. in the vertical direction as illustrated with sub-blocks 164, 165, 166 and 168 in Figure 16), or equal to the number of samples in the sub-block in one column when the location of the sub-block is derived from an edge to the left or to the right of the sub-block (e.g. the fourth block 167 in the example of Figure 16).

As a generalization, the depth map block can be expressed as an MxN matrix in which M is the number of rows in the depth map block and N is the number of columns in the depth map block. In some embodiments M=N=16. Correspondingly, the sub-blocks can be expressed as VxW matrices, in which V is the number of rows in the sub-block and W is the number of columns in the sub-block. In some embodiments V=W=4.

In the following examples the depth map block which is examined by using sub-blocks is a macroblock.

In some embodiments the sub-blocks may be identified by an index n e.g. in the traversal order of the sub-blocks (n = 0, 1 , ...) and the coordinate of the top-left corner sample (x,y), where the top-left corner sample of the macroblock has coordinates (0,0) and the indices increase towards right and bottom of the macroblock. Notation B(n,x,y) may be used. However, the identification of the sub-blocks may also be different from that. For example, in some embodiments the bottom-right corner sample of the macroblock has coordinates (0,0) and the indices decrease towards left and up of the macroblock.

The edge position from reconstructed/decoded samples values of neighboring

macroblocks may be derived e.g. as follows. The samples at the macroblock boundary from the neighboring macroblock above are denoted up_pix[0,...,N-1] as illustrated in Figure 16, in which M=16 and =16.

The greatest absolute deviation between any two adjacent samples among these samples at the macroblock boundary can be derived as follows:

j = arg max | up _ pix[i + 1] - up _ pix[i] | (3)

If |up_pix[)+l]-up_pix[j]|>thres, where thres is a defined threshold, the position of the topmost sub-block may be derived as Figure 16 shows. That is, the first sub-block can be identified by B(0,j-1,0).

In some embodiments, another algorithm to divide up_pix[0,..., N-l] into two sets may be used, such as the K-means clustering algorithm.

In some embodiments, the edge position is searched similarly from the samples at the macroblock boundary from the neighboring macroblock on the left, left_pix[0,..., M-l], resulting into index k to the left samples yielding the greatest absolute deviation between any two adjacent samples. In some embodiments, the search from left_pix[0,..., M-l] is done only if |up_pix[)+l]-up_pix[j]| < thres with all allowable values of j.

In some embodiments, sub-block traversal from top to bottom is selected, if |up_pix[j+l]- up_pix[j]| > |left_pix[k+l]-left_pix[k]| or, in some embodiments, if |up_pix[j+l]-up_pix[j]| > |left_pix[k+l]-left_pix[k]|. Otherwise, sub-block traversal from left to right may be selected. In some embodiments the pixels on the boundary of the current block but belonging to the neighboring block may be represented as two boundary pixel sets on the basis on the detected edge as follows. A first boundary pixel set (which may also be called as a boundary pixel set 0) contains pixels to the left of the edge, if the boundary is a horizontal boundary, or above the edge, if the boundary is a vertical boundary. A second boundary pixel set (which may also be called as a boundary pixel set 1) contains pixels to the right of the edge, if the boundary is a horizontal boundary, or below the edge, if the boundary is a vertical boundary. The first boundary pixel set can be regarded to contain pixels up_pix[0] to up_pix[j], inclusive, or pixels left_pix[0] to left_pix[k], inclusive, depending on whether traversal from top to bottom or from left to right is selected, respectively. In some embodiments, the first boundary pixel set contains both up_pix[0] to up_pix[j], inclusive, and left_pix[0] to left_pix[k], inclusive. The second boundary pixel set can be regarded to contain of up_pix[j+l] to up_pix[N-l], inclusive, or left_pix[k+l] to left_pix[M-l], inclusive, depending on whether traversal from top to bottom or from left to right is selected, respectively. In some embodiments, the second boundary pixel set can be regarded to contain both up_pix[j+l] to up_pix[N-l], inclusive, and left_pix[k+ 1 ] to left_pix[M-l], inclusive.

The term order-n depth boundary samples may be defined for up_pix[] and left_pix[] as follows. If a depth boundary was detected among up_pix[], the order-n depth boundary samples among up_pix[] consist of up_pix[j-n+l] (when j-n+1 > 0) and up_pix[j+n] (when j+n < -l). Similarly, if a depth boundary was detected among left_pix[],the order-n depth boundary samples among left_pix[] consist of left_pix[j-n+l] (when j-n+1 > 0) and left_pix[j+n] (when j+n < M-1).

Next, the other sub-block positions may be derived based on the previous sub-block B(n,x,y) in traversal order. Here, traversal from top to bottom is described.

The leftmost and rightmost columns of the previous sub-block are investigated. If the depth edge crosses the sub-block edge on the left, a new sub-block is located on the left. That is, the (n+l)th sub-block has coordinates x-W and y. If the depth edge crosses the sub-block edge on the right, a new sub-block is located on the right. That is, the (n+l)th sub-block has coordinates x+W and y.

Otherwise (if no depth edge crosses the previous sub-block from the left or right), the next sub-block is located below the n-th sub-block. That is, the (n+l)th sub-block has vertical coordinate y+V. The horizontal center of the next sub-block is chosen to match the depth edge position at the bottom-most sample line in the n-th sub-block.

A similar algorithm for traversal from left to right is as follows.

The top-most and bottom-most rows of previous sub-block are investigated. If the depth edge crosses the sub-block on the upside, a new sub-block is located on the upside. That is, the (n+l)th sub-block has coordinates x and y-V. If the depth edge crosses the sub-block on the downside, a new sub-block is located on the downside. That is, the (n+l)th sub-block has coordinates x and y+V.

Otherwise (if no depth edge crosses the previous sub-block from the top or bottom), the next sub-block is located on the right of the n-th sub-block. That is, the (n+l)th sub-block has horizontal coordinate x+W. The vertical center of the next sub-block is chosen to match the depth edge position at the rightmost sample line in the n-th sub-block.

The processing may be stopped if the (n+l)th sub-block would be completely outside the macroblock.

When the location of the depth boundary has been predicted, a prediction error of the location of the depth boundary may be encoded/decoded.

The encoder may determine the position and shape of the depth boundary at each sample row of the sub-block for example as follows:

The encoder may derive a rate and a distortion caused by the selection of different depth boundary positions. The distortion may be derived based on a sum of absolute differences between the predicted samples and the respective (i.e. co-located) input sample values. The rate may be determined based on the bitrate or bit count caused by syntax elements used to encode the depth boundary position.

In some cases, particularly when there are differences between the bitrate or bit count of different boundary positions and shapes are relatively small, the encoder may only use a distortion metric for determining a depth boundary position and shape.

The samples used in the distortion metric may include for example only the sample position on the left of the depth boundary and the sample position on the right of the depth boundary, or it may include for example all sample positions of the sub-block.

The encoder may use the following binarization scheme for different depth boundary positions and shapes within the sub-block. First, the difference of the boundary position compared to the boundary position in the adjacent sample row above may be encoded. In this example embodiment, the difference is limited to be one of -2, -1 , 0, 1 , and 2, and the boundary position is assumed to be a monotonic function. Then, the row-wise differences may be encoded using the following scheme. A similar scheme for column-wise difference may be used for horizontal traversal of sub-blocks.

Table 1. binarization scheme for different depth boundary positions(-/+: left/right shift)

Some examples of the binarization are given in Figures 17a— 17d. In Figures 17a— 17d the shaded circles illustrate depth map samples belonging to a first region and circles having no shading illustrate depth map samples belonging to a second region. In the example pattern of

Figure 17a the edge does not shift at all so the shift pattern is 0000 and the corresponding binary code is 0 according to Table 1. In the example pattern of Figure 17b the edge shifts one position to the left from row 0 to row 1 and from row 2 to row 3. Thus, the shift pattern is 0, -1 , 0, -1 and the corresponding binary code is 00101 according to Table 1. In the example pattern of Figure 17c the edge shifts two positions to the left from row 1 to row 2. Thus, the shift pattern is 0, 0, -2,

0 and the corresponding binary code is 01101 according to Table 1. In the example pattern of Figure 17d the edge shifts one position to the right from the previous block to row 0 and from row 2 to row 3. Thus, the corresponding binary code is 1 1001 according to Table 1.

The binarized depth boundary position and shape may be used as an input to context adaptive binary arithmetic coding (CABAC) or any appropriate entropy coding. Alternatively, the binarized depth boundary position and shape may be coded with any other variable-length coding scheme, such as Huffman coding.

The decoder may decode, using an appropriate entropy decoding scheme, codewords from the bitstream indicating the depth boundary position and shape. As a result the decoder may get binarized codewords indicating the row-wise difference of the boundary position compared to the predicted boundary position. The decoder may then obtain the signaled boundary position by summing up the decoded difference of the boundary position and the predicted boundary position.

The samples at a proximity of the depth boundary may also be predicted. The samples whose sample value is predicted in the proximity of a depth boundary may be selected. In some embodiments, the selection of the predicted samples is pre-determined or inferred both in the encoder and in the decoder. In some other embodiments, the encoder may indicate which samples are predicted in the bitstream for example in a sequence parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, or anything alike. The encoder may, for example, use rate-distortion optimization when determining which samples in the proximity of a depth boundary are predicted. The encoder may have for example some of the following schemes among which the selection of the predicted samples is made or the one of the following schemes may be used by the encoder and the decoder for inference of the samples being predicted:

All samples of the sub-blocks are predicted.

Order- 1 depth boundary samples are predicted.

All order-n depth boundary samples are predicted, where n includes all values from

1 to a selected threshold. The encoder may indicate the selected threshold in the bitstream for example in a sequence parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, or anything alike. The selected samples may be predicted for example using one of the following schemes: Order-n depth boundary sample is predicted from the respective order-n depth boundary sample among up_pix or left_pix (whichever is selected as the starting point for depth boundary traversal).

- Order-n depth boundary sample is predicted from the respective order-n depth boundary sample among up_pix or left_pix (whichever is selected as the starting point for depth boundary traversal), when n < threshold t. Otherwise, order-n boundary sample is predicted from the respective order-t depth boundary sample among up_pix or left_pix (whichever is selected as the starting point for depth boundary traversal).

- If the same edge is detected to cross both the top and the left side of the

macroblock, the prediction scheme of the sample values in the proximity of the depth boundary may be bi-directional and weighted according to the distance from the top or left side of the macroblock. For example, if the depth edge contains a total of N samples, and ριτ is the value of the edge sample in the neighboring macroblock at the top and piL is the value of the edge sample in the neighboring macroblock at the left, and ηχ = 1 .. N is the order number of samples along the depth edge starting from the top of the depth edge, weighting could be applied as follows: ((N + 1 - n_T) * PIT + n_T * PIL) / (N+l). In another example, a weight inversely proportional to the Euclidean distance of the current edge sample from the edge sample in the neighboring macroblock could be used.

The encoder may encode and the decoder may decode the prediction error for all or some of the samples at the proximity of the depth boundary that were previously predicted. For example, the encoder may encode a delta value between the last predicted value of order- 1 depth boundary samples and its actual value to the bitstream and the decoder may decode the delta value from the bitstream. Other order- 1 depth boundary samples may be predicted from the delta value and the order- 1 depth boundary sample among up_pix or left_pix (whichever is selected as the starting point for depth boundary traversal) for example using weighting based on Euclidean distance from the sample being predicted and the samples used for prediction. This prediction error coding and prediction scheme may be helpful if the depth is smoothly varying and a planar prediction for the non-edge samples is used.

In another example, a difference block may be formed where each column or line is formed by the delta value of the predicted order- 1 depth boundary samples and their actual value. Then the difference block can be transform-coded and quantized. In decoding, the difference block may be dequantized and transform-decoded. Decoded order- 1 depth boundary samples may then be formed by summing up the predicted values and the decoded difference values from the difference block.

After predicting and encoding/decoding the prediction error for samples in the proximity of a depth boundary, the remaining samples in the depth block may be predicted both in the encoder and in the decoder. Several methods may be used for prediction, some of which are outlined below.

In some embodiments, the remaining depth samples are predicted by the neighbor macroblocks and the reconstructed sub-blocks. The macroblock is coarsely divided into two regions by sub-blocks. The encoder may choose a prediction mode, such as vertical or horizontal prediction, for each region separately. In some embodiments, the selection of the prediction mode is performed on the basis of already encoded/decoded information, such as reconstructed or decoded neighboring macroblocks, and the same prediction mode determination is performed in the encoder and in the decoder. In other embodiments, the encoder may indicate the used prediction mode in the bitstream for example in a sequence parameter set, a picture parameter set, an adaptation parameter set, a picture header, a slice header, a macroblock syntax structure, a block syntax structure , or anything alike. In some embodiments, the prediction mode for the two regions may be indicated with separate syntax elements in the bitstream. In some embodiments, the encoder and decoder may conclude from the encoded depth edge information that a block partition, such as a prediction unit, is entirely covered by one region, and the encoder includes one prediction mode syntax element in the bitstream corresponding to that one region, omitting the prediction mode for the other region, and correspondingly the decoder only decodes one prediction mode syntax element from the bitstream corresponding to that one region.

In some embodiments a sample is predicted from a sample of a neighbor macroblock only if the sample of a neighbor macroblock is determined to be in the same depth region than the order- 1 depth boundary sample at the same side of the depth boundary as the sample being predicted.

In some embodiments, a sample to be predicted may be surrounded from up to four sides by available depth boundary samples or block boundary samples at neighboring blocks, and hence the intra prediction modes may also differ from those currently in H.264/AVC or HEVC or any similar coding or decoding method or system. For example, the H.264/AVC intra prediction modes may be changed as follows.

In the DC mode, the mean value of one of the boundary pixel set and the order- 1 depth boundary samples at the same side of the depth edge is used as the prediction value of the remaining depth sample values. For example, with reference to Figure 16, the mean value of left_pix[0] to left_pix[15], up_pix[0] to up_pix[l 1], and all the shaded order- 1 depth boundary samples may be derived and used as the prediction value of the remaining shaded samples.

Similarly, the mean value of up_pix[12] to up_pix[15] and all the white order- 1 depth boundary samples may be derived and used as the prediction value of the remaining white samples.

In the horizontal/vertical mode, if boundary samples from both sides of the current sample are available, the boundary samples may be weighted according to the Euclidean spatial distance to the sample being predicted. For example, with reference to Figure 16, if a sample at location (x,y)=(3,0) is predicted using the horizontal mode, both left_pix[0] (pi) and the order- 1 depth sample (p2) at location (11,0) are available as prediction samples. The prediction sample may be derived using m=(l l-(-l))=12 as ((m-(x-xl))*pl + (m-(x2-x))*p2) / m = ((12-(3-(- l)))*pl + (12-(1 l-3))*p2) / 12 = (8*pl + 4*p2) / 12. If only one boundary sample is available, it is used as such as a prediction. If no boundary samples are available, the value obtained by through DC prediction may be used.

In some embodiments, the depth block may be partitioned conventionally, and the prediction error of the block partitions may be encoded/decoded conventionally.

In some embodiments, the depth block is partitioned based on the samples at the proximity of the depth boundary. For example, a block partition may not cross a depth boundary, and one of the two following methods may for example be used:

In a first method, in order to use a regular NxN transform for coding, a partitioned block may be first converted to a boundary-filled block in encoding for example by repeating the order- 1 depth boundary sample for those samples that are not available (i.e., are at the other side of the depth boundary), and then the boundary-filled block may be transform-coded. The other block partition of the same NxN block may be coded similarly. In decoding the prediction error block may first be decoded into an NxN decoded boundary-filled block, and the second prediction error block of the same NxN block may be decoded into a second NxN decoded boundary-filled block. The two decoded boundary- filled blocks may be combined into one by selecting the sample from that decoded boundary-filled block which was not boundary-filled during encoding.

In a second method, a defined or signaled scanning order of the prediction error samples is used to form regular NxN blocks of samples to be transform-coded. For example, a zig-zag scanning order can be used starting from the top-left corner of the prediction error samples to be coded and traversing along diagonal pixel lines in alternating direction. The last NxN block to be transform-coded may be padded for example with the last actual sample value. In decoding, the dequantized NxN transform blocks are inverse-transformed, and an inverse scanning is used to map decoded samples from the inverse-transformed NxN blocks to the actual sample locations.

In some embodiments, a depth coding unit may be partitioned to prediction units and transform units similarly to HEVC. In some embodiments, both partitioning to prediction units and partitioning to transform units may be performed according to depth boundary. In some embodiments, partitioning to prediction units may be performed according to depth boundary, while partitioning to transform units may be done conventionally. In some embodiments, partitioning to prediction units may be done conventionally, while partitioning to transform units may be done according to depth boundary.

A high level flow chart of an embodiment of an encoder 180 capable of encoding texture views and depth views is presented in Figure 18 and a decoder 190 capable of decoding texture views and depth views is presented in Figure 19. On these figures solid lines depict general data flow and dashed lines show control information signaling. The encoder 180 may receive texture components 181 to be encoded by a texture encoder 182 and depth map components 183 to be encoded by a depth encoder 184. When the encoder 180 is encoding texture components according to AVC/MVC a first switch 185 may be switched off. When the encoder 180 is encoding enhanced texture components the first switch 185 may be switched on so that information generated by the depth encoder 184 may be provided to the texture encoder 182. The encoder of this example also comprises a second switch 186 may be operated as follows. The second switch 186 is switched on when the encoder is encoding depth information of AVC/MVC views, and the second switch 186 is switched off when the encoder is encoding depth

information of enhanced texture views. The encoder 180 may output a bitstream 187 containing encoded video information. In some embodiments the encoder 180 comprises an entropy encoder 188 to produce bitstream containing entropy encoded video information. The decoder 190 may operate in a similar manner but at least partly in a reversed order. The decoder 190 may receive the bitstream 187 containing encoded video information. In some embodiments the decoder 190 comprises an entropy decoder 198 to decode entropy encoded video information. The decoder 190 comprises a texture decoder 191 for decoding texture information and a depth decoder 192 for decoding depth information. A third switch 193 may be provided to control information delivery from the depth decoder 192 to the texture decoder 191 , and a fourth switch 194 may be provided to control information delivery from the texture decoder 191 to the depth decoder 192. When the decoder 190 is to decode AVC/MVC texture views the third switch 193 may be switched off and when the decoder 190 is to decode enhanced texture views the third switch 193 may be switched on. When the decoder 190 is to decode depth of AVC/MVC texture views the fourth switch 194 may be switched on and when the decoder 190 is to decode depth of enhanced texture views the fourth switch 194 may be switched off. The Decoder 190 may output reconstructed texture components 195 and reconstructed depth map components 196.

There is provided the following elements which can be combined into a single solution, as will be described below, or they can be utilized separately. As explained earlier, both a video encoder and a video decoder typically apply a prediction mechanism, hence the following elements may apply similarly to both a video encoder and a video decoder.

In some embodiments, the encoder may determine the value of any of the above- mentioned thresholds for example based on encoding blocks with different values of the threshold and selecting the value of the threshold that is optimal according to the Lagrangian rate-distortion optimization equation. The encoder may indicate the determined value of the threshold within the bitstream, for example by encoding it as a syntax element for example in a sequence parameter set, a picture parameter set, a slice parameter set, a picture header, a slice header, within a macroblock syntax structure, or anything alike. In some embodiments, the decoder determines the threshold based on the information encoded in the bistream, such as a codeword indicating the value of threshold.

While many of the embodiments are described for intra prediction for luma, it is to be understood that in many coding arrangements chroma intra prediction information may be derived from luma intra prediction information using pre-determined relations. For example, it may be assumed that the same reference samples are used for the chroma components as for luma. In some embodiments, depth pictures have the same spatial resolution as chroma texture pictures, and hence determining co-location as well as correspondence between depth and chroma texture blocks sizes and shapes may be done directly by using depth coordinates and block sizes as chroma texture coordinates and block sizes, respectively, or vice versa. In some embodiments, depth pictures have a different spatial resolution from chroma texture pictures. Therefore, scaling according to the proportion of the spatial resolutions may be taken into account in the determination of co-located or respective blocks in chroma texture and depth view components. Similarly, the size of block partitions may be scaled according to the proportion of the spatial resolutions in the prediction of block partitions from depth to chroma texture.

In the above, many embodiments have been described in relation to depth coding based on H.264/AVC. It needs to be understood, however, that embodiments could be realized with any coding standard or scheme, such as HEVC.

In some embodiments the spatial resolution of depth/disparity pictures may differ or may be re-sampled in the encoder as a pre-processing operation to become different from that of the luma pictures of texture. In some embodiments, the depth/disparity pictures are re-sampled in the encoding loop and/or the decoding loop to become identical resolution to the respective luma pictures of texture. In other embodiments, the spatially corresponding blocks of depth/disparity pictures are found by scaling the block locations and size proportionally to the ratio of the picture extents of the depth pictures and luma pictures of texture.

The following describes in further detail suitable apparatus and possible mechanisms for implementing the embodiments of the invention. In this regard reference is first made to Figure 10 which shows a schematic block diagram of an exemplary apparatus or electronic device 50, which may incorporate a codec according to an embodiment of the invention.

The electronic device 50 may for example be a mobile terminal or user equipment of a wireless communication system. However, it would be appreciated that embodiments of the invention may be implemented within any electronic device or apparatus which may require encoding and decoding or encoding or decoding video images.

The apparatus 50 may comprise a housing 30 for incorporating and protecting the device. The apparatus 50 further may comprise a display 32 in the form of a liquid crystal display. In other embodiments of the invention the display may be any suitable display technology suitable to display an image or video. The apparatus 50 may further comprise a keypad 34. In other embodiments of the invention any suitable data or user interface mechanism may be employed. For example the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display. The apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analogue signal input. The apparatus 50 may further comprise an audio output device which in embodiments of the invention may be any one of: an earpiece 38, speaker, or an analogue audio or digital audio output connection. The apparatus 50 may also comprise a battery 40 (or in other embodiments of the invention the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise an infrared port 42 for short range line of sight communication to other devices. In other embodiments the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB/firewire wired connection.

The apparatus 50 may comprise a controller 56 or processor for controlling the apparatus 50. The controller 56 may be connected to memory 58 which in embodiments of the invention may store both data in the form of image and audio data and/or may also store instructions for implementation on the controller 56. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and decoding of audio and/or video data or assisting in coding and decoding carried out by the controller 56.

The apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.

The apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and for receiving radio frequency signals from other apparatus(es).

In some embodiments of the invention, the apparatus 50 comprises a camera capable of recording or detecting individual frames which are then passed to the codec 54 or controller for processing. In other embodiments of the invention, the apparatus may receive the video image data for processing from another device prior to transmission and/or storage. In other embodiments of the invention, the apparatus 50 may receive either wirelessly or by a wired connection the image for coding/decoding.

With respect to Figure 12, an example of a system within which embodiments of the present invention can be utilized is shown. The system 10 comprises multiple communication devices which can communicate through one or more networks. The system 10 may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, CDMA network etc), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.

The system 10 may include both wired and wireless communication devices or apparatus

50 suitable for implementing embodiments of the invention.

For example, the system shown in Figure 12 shows a mobile telephone network 1 1 and a representation of the internet 28. Connectivity to the internet 28 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired

connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.

The example communication devices shown in the system 10 may include, but are not limited to, an electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22. The apparatus 50 may be stationary or mobile when carried by an individual who is moving. The apparatus 50 may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle or any similar suitable mode of transport.

Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 1 1 and the internet 28. The system may include additional communication devices and communication devices of various types.

The communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.1 1 and any similar wireless communication technology. A communications device involved in implementing various embodiments of the present invention may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.

Although the above examples describe embodiments of the invention operating within a codec within an electronic device, it would be appreciated that the invention as described below may be implemented as part of any video codec. Thus, for example, embodiments of the invention may be implemented in a video codec which may implement video coding over fixed or wired communication paths.

Thus, user equipment may comprise a video codec such as those described in

embodiments of the invention above. It shall be appreciated that the term user equipment is intended to cover any suitable type of wireless user equipment, such as mobile telephones, portable data processing devices or portable web browsers.

Furthermore elements of a public land mobile network (PLM ) may also comprise video codecs as described above.

In general, the various embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

The embodiments of this invention may be implemented by computer software executable by a data processor of the mobile device, such as in the processor entity, or by hardware, or by a combination of software and hardware. Further in this regard it should be noted that any blocks of the logic flow as in the Figures may represent program steps, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips, or memory blocks implemented within the processor, magnetic media such as hard disk or floppy disks, and optical media such as for example DVD and the data variants thereof, CD.

The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The data processors may be of any type suitable to the local technical environment, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multi-core processor architecture, as non-limiting examples.

Embodiments of the inventions may be practiced in various components such as integrated circuit modules. The design of integrated circuits is by and large a highly automated process. Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

Programs, such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre-stored design modules. Once the design for a semiconductor circuit has been completed, the resultant design, in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.

The foregoing description has provided by way of exemplary and non-limiting examples a full and informative description of the exemplary embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of this invention. In the following some examples will be provided. According to a first embodiment there is provided a method comprising: receiving samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map in a first direction and in a second direction;

In some embodiments of the method the determining the position of the sub-block comprises:

determining a location of an edge in the set of samples; and

selecting the position for the sub-block in the depth map in the second direction on the basis of the location of the edge in the set of samples.

In some embodiments the method comprises selecting the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

In some embodiments the method comprises at least one of the following:

examining if there is an edge at a second border of the sub-block, the second border being opposite to the first border; and if so, selecting a position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block;

examining if there is an edge at a third border of the sub-block, the third border being perpendicular to the first border; and if so, selecting a position of another sub-block in the depth map in the first direction so that the another sub-block is adjacent to the sub-block. In some embodiments the selection of the position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block further comprises selecting a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

In some embodiments the selection of the position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block further comprises:

determining the position of the sub-block based on an encoded depth edge position of a previous sub-block; and

encoding the prediction of a location of an edge within the sub-block.

In some embodiments the method comprises using as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

In some embodiments the method comprises repeating the selecting a position for a sub- block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

In some embodiments the method comprises using a vertical direction as the first direction, and a horizontal direction as the second direction.

In some embodiments the method comprises using a horizontal direction as the first direction, and a vertical direction as the second direction.

In some embodiments the method comprises using a macroblock of a video frame as the block of the depth map. In some embodiments of the method the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels. In some embodiments of the method the set of samples comprises 4 samples.

In some embodiments of the method the forming a prediction of a location of an edge comprising determining an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

In some embodiments the method comprises encoding the edge traversal pattern.

In some embodiments of the method the part of the picture comprises multiview video information.

According to a second embodiment there is provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:

In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to: determine a location of an edge in the set of samples; and select the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to perform at least one of the following:

examining if there is an edge at a second border of the sub-block, the second border being opposite to the first border; and if so, selecting a position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block and a center of the another sub-block is at the location of the edge at the second border of the sub-block;

examining if there is an edge at a third border of the sub-block, the third border being perpendicular to the first border; and if so, selecting a position of another sub-block in the depth map in the first direction so that the another sub-block is adjacent to the sub-block.

In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

In some embodiments of the apparatus said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to repeat the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a vertical direction as the first direction, and a horizontal direction as the second direction. In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a horizontal direction as the first direction, and a vertical direction as the second direction. In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a macroblock of a video frame as the block of the depth map.

In some embodiments of the apparatus the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

In some embodiments of the apparatus the set of samples comprises 4 samples.

In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to determine an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction. In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to encode the edge traversal pattern.

In some embodiments of the apparatus the part of the picture comprises multiview video information.

In some embodiments of the apparatus the apparatus is a component of a mobile station.

According to a third embodiment there is provided a computer program product including one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to at least perform the following: receive samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction;

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to:

determine a location of an edge in the set of samples; and

select the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to perform at least one of the following:

examining if there is an edge at a third border of the sub-block, the third border being perpendicular to the first border; and if so, selecting a position of another sub-block in the depth map in the first direction so that the another sub-block is adjacent to the sub-block. In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to repeat the selecting a position for a sub-block, determining a position for the sub- block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a vertical direction as the first direction, and a horizontal direction as the second direction.

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a horizontal direction as the first direction, and a vertical direction as the second direction.

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a macroblock of a video frame as the block of the depth map.

In some embodiments of the computer program product the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

In some embodiments of the computer program product the set of samples comprises 4 samples. In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to determine an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to encode the edge traversal pattern.

In some embodiments of the computer program product the part of the picture comprises multiview video information.

In some embodiments the computer program product is a software component of a mobile station.

According to a fourth embodiment there is provided an apparatus comprising:

means for receiving samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction;

In some embodiments of the apparatus the means for determining the position of the sub- block comprises:

means for determining a location of an edge in the set of samples; and means for selecting the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

In some embodiments the apparatus comprises at least one of the following:

means for examining if there is an edge at a second border of the sub-block, the second border being opposite to the first border; and if so, selecting a position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block and a center of the another sub-block is at the location of the edge at the second border of the sub-block;

means for examining if there is an edge at a third border of the sub-block, the third border being perpendicular to the first border; and if so, selecting a position of another sub-block in the depth map in the first direction so that the another sub-block is adjacent to the sub-block.

In some embodiments the apparatus comprises means for using as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

In some embodiments the apparatus comprises means for repeating the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

In some embodiments the apparatus comprises means for using a vertical direction as the first direction, and a horizontal direction as the second direction.

In some embodiments the apparatus comprises means for using a horizontal direction as the first direction, and a vertical direction as the second direction.

In some embodiments the apparatus comprises means for using a macroblock of a video frame as the block of the depth map. In some embodiments of the apparatus the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

In some embodiments of the apparatus the set of samples comprises 4 samples.

In some embodiments of the apparatus the means for forming a prediction of a location of an edge comprises means for determining an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

In some embodiments the apparatus comprises means for encoding the edge traversal pattern.

In some embodiments the apparatus is a component of a mobile station.

According to a fifth embodiment there is provided a method comprising:

receiving an encoded location of an edge within a sub-block;

selecting a position for a sub-block in a depth map in a first direction, the sub-block comprising samples in the first direction and in a second direction;

decoding the encoded location of the edge within the sub-block.

In some embodiments of the method the determining the position of the sub-block comprising:

determining a location of an edge in the set of samples; and selecting the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

In some embodiments the method comprises at least one of the following:

In some embodiments the method comprises using a vertical direction as the first direction, and a horizontal direction as the second direction. In some embodiments the method comprises using a horizontal direction as the first direction, and a vertical direction as the second direction.

In some embodiments the method comprises using a macroblock of a video frame as the block of the depth map. In some embodiments of the method the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

In some embodiments of the method the set of samples comprises 4 samples.

In some embodiments of the method the forming a prediction of a location of an edge comprises determining an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

In some embodiments the method comprises decoding the edge traversal pattern.

According to a sixth embodiment there is provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:

receive an encoded location of an edge within a sub-block;

select a position for a sub-block in a depth map in a first direction, the sub-block comprising samples in the first direction and in a second direction;

decode the encoded location of the edge within the sub-block.

In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to repeat the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

In some embodiments of the apparatus the set of samples comprises 4 samples.

In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to determine an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction. In some embodiments of the apparatus, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to decode the edge traversal pattern.

In some embodiments the apparatus is a component of a mobile station.

According to a seventh embodiment there is provided an apparatus comprising computer program product including one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to at least perform the following: receive an encoded location of an edge within a sub-block;

decode the encoded location of the edge within the sub-block.

determine a location of an edge in the set of samples; and

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block. In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to repeat the selecting a position for a sub-block, determining a position for the sub- block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

In some embodiments of the computer program product the set of samples comprises 4 samples.

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to determine an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

In some embodiments the computer program product includes one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to decode the edge traversal pattern.

An apparatus according to an eighth embodiment comprises:

means for receiving an encoded location of an edge within a sub-block;

means for selecting a position for a sub-block in a depth map in a first direction, the sub- block comprising samples in the first direction and in a second direction;

decoding the encoded location of the edge within the sub-block.

means for determining a location of an edge in the set of samples; and

means for selecting the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

In some embodiments the apparatus comprises at least one of the following: means for examining if there is an edge at a second border of the sub-block, the second border being opposite to the first border; and if so, selecting a position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block and a center of the another sub-block is at the location of the edge at the second border of the sub-block;

means for examining if there is an edge at a third border of the sub-block, the third border being perpendicular to the first border; and if so, selecting a position of another sub-block in the depth map in the first direction so that the another sub-block is adjacent to the sub-block. In some embodiments the apparatus comprises means for using as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

In some embodiments the apparatus comprises means for using a macroblock of a video frame as the block of the depth map.

In some embodiments of the apparatus the set of samples comprises 4 samples. In some embodiments of the apparatus the means for forming a prediction of a location of an edge comprises means for determining an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

In some embodiments the apparatus comprises means for decoding the edge traversal pattern.

In some embodiments of the apparatus the part of the picture comprises multiview video information. In some embodiments the apparatus is a component of a mobile station.

A video coder according to a ninth embodiment is configured for:

receiving an encoded location of an edge within a sub-block;

selecting a position for a sub-block in a depth map in the first direction, the sub-block comprising samples in a first direction and in a second direction;

determining a position for the sub-block in the depth map in the second direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction;

decoding the encoded location of the edge within the sub-block; and

forming a prediction of a location of an edge within the sub-block on the basis of the decoded location of the edge within the sub-block.

A video decoder according to a tenth embodiment is configured for:

receiving an encoded location of an edge within a sub-block;

decoding the encoded location of the edge within the sub-block.

Claims

What is claimed is:

1. A method comprising:

2. The method according to claim 1 , the determining the position of the sub-block comprising:

determining a location of an edge in the set of samples; and

3. The method according to claim 1 or 2 comprising selecting the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

4. The method according to claim 1, 2 or 3 comprising at least one of the following: examining if there is an edge at a second border of the sub-block, the second border being opposite to the first border; and if so, selecting a position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block;

5. The method according to claim 4 wherein the selection of the position of another sub- block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block further comprises selecting a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

6. The method according to claim 4 or 5, wherein the selection of the position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block further comprises:

encoding the prediction of a location of an edge within the sub-block.

7. The method according to any of the claims 1 to 6 comprising using as many samples of the depth map in the set of samples as there are samples in the first border of the sub- block.

8. The method according to any of the claims 1 to 7 comprising repeating the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

9. The method according to any of the claims 1 to 8 comprising using a vertical direction as the first direction, and a horizontal direction as the second direction.

10. The method according to any of the claims 1 to 8 comprising using a horizontal direction as the first direction, and a vertical direction as the second direction.

11. The method according to any of the claims 1 to 10 comprising using a macroblock of a video frame as the block of the depth map.

12. The method according to claim 1 1, wherein the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

13. The method according to claim 12, wherein the set of samples comprises 4 samples.

14. The method according to any of the claims 1 to 13 the forming a prediction of a location of an edge comprising determining an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

15. The method according to claim 14 comprising encoding the edge traversal pattern.

16. The method according to any of the claims 1 to 15, wherein the part of the picture comprises multiview video information.

17. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:

18. The apparatus according to claim 17, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to: determine a location of an edge in the set of samples; and

19. The apparatus according to claim 17 or 18, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to select the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

20. The apparatus according to claim 17, 18 or 19, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to perform at least one of the following:

21. The apparatus according to claim 20, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to select a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

22. The apparatus according to claim 20 or 21 , said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to determine the position of the sub-block based on an encoded depth edge position of a previous sub-block; and

encode the prediction of a location of an edge within the sub-block.

23. The apparatus according to any of the claims 17 to 22, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

24. The apparatus according to any of the claims 17 to 23, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to repeat the selecting a position for a sub-block, determining a position for the sub- block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

25. The apparatus according to any of the claims 17 to 24, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a vertical direction as the first direction, and a horizontal direction as the second direction.

26. The apparatus according to any of the claims 17 to 24, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a horizontal direction as the first direction, and a vertical direction as the second direction.

27. The apparatus according to any of the claims 17 to 26, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a macroblock of a video frame as the block of the depth map.

28. The apparatus according to claim 27, wherein the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

29. The apparatus according to claim 28, wherein the set of samples comprises 4 samples.

30. The apparatus according to any of the claims 17 to 29, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to determine an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

31. The apparatus according to claim 30, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to encode the edge traversal pattern.

32. The apparatus according to any of the claims 17 to 31 , wherein the part of the picture comprises multiview video information.

33. The apparatus according to any of the claims 17 to 32, wherein the apparatus is a component of a mobile station.

34. A computer program product including one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to at least perform the following:

determine a position for the sub-block in the depth map in the second direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction; and form a prediction of a location of an edge within the sub-block on the basis of samples of the block inside the sub-block.

35. The computer program product according to claim 34, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to:

determine a location of an edge in the set of samples; and

36. The computer program product according to claim 34, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to select the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

37. The computer program product according to claim 34, 35 or 36, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to perform at least one of the following:

38. The computer program product according to claim 37, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to select a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

39. The computer program product according to claim 37 or 38, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to:

determine the position of the sub-block based on an encoded depth edge position of a previous sub-block; and

encode the prediction of a location of an edge within the sub-block.

40. The computer program product according to any of the claims 34 to 39, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

41. The computer program product according to any of the claims 34 to 40, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to repeat the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

42. The computer program product according to any of the claims 34 to 41, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a vertical direction as the first direction, and a horizontal direction as the second direction.

43. The computer program product according to any of the claims 34 to 41, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a horizontal direction as the first direction, and a vertical direction as the second direction.

44. The computer program product according to any of the claims 34 to 43, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a macroblock of a video frame as the block of the depth map.

45. The computer program product according to claim 44, wherein the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

46. The computer program product according to claim 45, wherein the set of samples comprises 4 samples.

47. The computer program product according to any of the claims 34 to 46, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to determine an edge traversal pattern within the sub- blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

48. The computer program product according to claim 47, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to encode the edge traversal pattern.

49. The computer program product according to any of the claims 34 to 48, wherein the part of the picture comprises multiview video information.

50. The computer program product according to any of the claims 34 to 49, wherein the computer program product is a software component of a mobile station.

51. An apparatus comprising:

means for receiving samples representing pixels of a block of a depth related information of a part of a picture, the depth map comprising samples representing a depth map, the depth map comprising samples in a first direction and in a second direction; means for selecting a position for a sub-block in the depth map in the first direction, the sub-block comprising samples in the first direction and in the second direction;

52. The apparatus according to claim 51, the means for determining the position of the sub-block comprising:

means for determining a location of an edge in the set of samples; and

53. The apparatus according to claim 51 or 52, the means for determining the position of the sub-block comprising means for selecting the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

54. The apparatus according to claim 51, 52 or 53 comprising at least one of the following:

55. The apparatus according to claim 54, the means for determining the position of the sub-block comprising means for selecting a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

56. The apparatus according to claim 54 or 55, the means for determining the position of the sub-block comprising:

means for determining the position of the sub-block based on an encoded depth edge position of a previous sub-block; and

means for encoding the prediction of a location of an edge within the sub-block.

57. The apparatus according to any of the claims 51 to 56 comprising means for using as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

58. The apparatus according to any of the claims 51 to 57 comprising means for repeating the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

59. The apparatus according to any of the claims 51 to 58 comprising means for using a vertical direction as the first direction, and a horizontal direction as the second direction.

60. The apparatus according to any of the claims 51 to 58 comprising means for using a horizontal direction as the first direction, and a vertical direction as the second direction.

61. The apparatus according to any of the claims 51 to 60 comprising means for using a macroblock of a video frame as the block of the depth map.

62. The apparatus according to claim 61, wherein the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

63. The apparatus according to claim 62, wherein the set of samples comprises 4 samples.

64. The apparatus according to any of the claims 51 to 63 the means for forming a prediction of a location of an edge comprising means for determining an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

65. The apparatus according to claim 64 comprising means for encoding the edge traversal pattern.

66. The apparatus according to any of the claims 51 to 65, wherein the part of the picture comprises multiview video information.

67. The apparatus according to any of the claims 51 to 66, wherein the apparatus is a component of a mobile station.

68. A method comprising:

receiving an encoded location of an edge within a sub-block;

decoding the encoded location of the edge within the sub-block; and

69. The method according to claim 68, the determining the position of the sub-block comprising:

70. The method according to claim 68 or 69 comprising selecting the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

71. The method according to claim 68, 69 or 70 comprising at least one of the following:

72. The method according to claim 71 wherein the selection of the position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block further comprises selecting a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

73. The method according to claim 71 or 72, wherein the selection of the position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block further comprises:

determining the position of the sub-block based on a decoded depth edge position of a previous sub-block; and

decoding the prediction of a location of an edge within the sub-block.

74. The method according to any of the claims 68 to 73 comprising using as many samples of the depth map in the set of samples as there are samples in the first border of the sub- block.

75. The method according to any of the claims 68 to 74 comprising repeating the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

76. The method according to any of the claims 68 to 75 comprising using a vertical direction as the first direction, and a horizontal direction as the second direction.

77. The method according to any of the claims 68 to 75 comprising using a horizontal direction as the first direction, and a vertical direction as the second direction.

78. The method according to any of the claims 68 to 77 comprising using a macroblock of a video frame as the block of the depth map.

79. The method according to claim 78, wherein the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

80. The method according to claim 79, wherein the set of samples comprises 4 samples.

81. The method according to any of the claims 68 to 80 the forming a prediction of a location of an edge comprising determining an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

82. The method according to claim 81 comprising decoding the edge traversal pattern.

83. The method according to any of the claims 68 to 82, wherein the part of the picture comprises multiview video information.

84. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:

receive an encoded location of an edge within a sub-block;

determine a position for the sub-block in the depth map in the second direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction;

decode the encoded location of the edge within the sub-block; and

form a prediction of a location of an edge within the sub-block on the basis of the decoded location of the edge within the sub-block.

85. The apparatus according to claim 84, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to: determine a location of an edge in the set of samples; and

86. The apparatus according to claim 84 or 85, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to select the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

87. The apparatus according to claim 84, 85 or 86, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to perform at least one of the following: examining if there is an edge at a second border of the sub-block, the second border being opposite to the first border; and if so, selecting a position of another sub-block in the depth map in the second direction so that the another sub-block is at least partly adjacent to the second border of the sub-block and a center of the another sub-block is at the location of the edge at the second border of the sub-block;

88. The apparatus according to claim 87, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to select a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

89. The apparatus according to claim 87 or 88, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to determine the position of the sub-block based on a decoded depth edge position of a previous sub-block; and

decode the prediction of a location of an edge within the sub-block.

90. The apparatus according to any of the claims 84 to 89, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

91. The apparatus according to any of the claims 84 to 90, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to repeat the selecting a position for a sub-block, determining a position for the sub- block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

92. The apparatus according to any of the claims 84 to 91, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a vertical direction as the first direction, and a horizontal direction as the second direction.

93. The apparatus according to any of the claims 84 to 91, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a horizontal direction as the first direction, and a vertical direction as the second direction.

94. The apparatus according to any of the claims 84 to 93, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to use a macroblock of a video frame as the block of the depth map.

95. The apparatus according to claim 94, wherein the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

96. The apparatus according to claim 95, wherein the set of samples comprises 4 samples.

97. The apparatus according to any of the claims 84 to 96, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to determine an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

98. The apparatus according to claim 97, said at least one memory stored with code thereon, which when executed by said at least one processor, further causes the apparatus to decode the edge traversal pattern.

99. The apparatus according to any of the claims 84 to 98, wherein the part of the picture comprises multiview video information.

100. The apparatus according to any of the claims 84 to 99, wherein the apparatus is a component of a mobile station.

101. A computer program product including one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to at least perform the following:

receive an encoded location of an edge within a sub-block;

decode the encoded location of the edge within the sub-block; and

form a prediction of a location of an edge within the sub-block on the basis ofthe decoded location of the edge within the sub-block.

102. The computer program product according to claim 101, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to:

determine a location of an edge in the set of samples; and

103. The computer program product according to claim 101, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to select the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

104. The computer program product according to claim 101 , 102 or 103, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to perform at least one of the following:

105. The computer program product according to claim 104, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to select a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

106. The computer program product according to claim 104 or 105, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to:

determine the position of the sub-block based on a decoded depth edge position of a previous sub-block; and

decode the prediction of a location of an edge within the sub-block.

107. The computer program product according to any of the claims 101 to 106, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

108. The computer program product according to any of the claims 101 to 107, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to repeat the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

109. The computer program product according to any of the claims 101 to 108, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a vertical direction as the first direction, and a horizontal direction as the second direction.

110. The computer program product according to any of the claims 101 to 108, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a horizontal direction as the first direction, and a vertical direction as the second direction.

111. The computer program product according to any of the claims 101 to 110, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to use a macroblock of a video frame as the block of the depth map.

112. The computer program product according to claim 111 , wherein the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

113. The computer program product according to claim 112, wherein the set of samples comprises 4 samples.

114. The computer program product according to any of the claims 101 to 113, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to determine an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

115. The computer program product according to claim 114, including one or more sequences of one or more instructions which, when executed by one or more processors, further causes the apparatus to decode the edge traversal pattern.

116. The computer program product according to any of the claims 101 to 115, wherein the part of the picture comprises multiview video information.

117. The computer program product according to any of the claims 101 to 116, wherein the computer program product is a software component of a mobile station.

118. An apparatus comprising:

means for receiving an encoded location of an edge within a sub-blockn;

means for determining a position for the sub-block in the depth map in the second direction by using a set of samples neighboring a first border of the sub-block at the selected position in the first direction;

means for decoding the encoded location of the edge within the sub-block; and means for forming a prediction of a location of an edge within the sub-block on the basis of the decoded location of the edge within the sub-block.

119. The apparatus according to claim 1 18, the means for determining the position of the sub-block comprising:

means for determining a location of an edge in the set of samples; and

120. The apparatus according to claim 1 18 or 1 19, the means for determining the position of the sub-block comprising means for selecting the position for the sub-block in the depth map in the second direction so that a center of the sub-block in the second direction is at the location of the edge in the set of samples.

121. The apparatus according to claim 1 18 or 119 comprising at least one of the following:

122. The apparatus according to claim 121 , the means for determining the position of the sub-block comprising means for selecting a center of the another sub-block to be at the location of the edge at the second border of the sub-block.

123. The apparatus according to claim 121 or 122, the means for determining the position of the sub-block comprising:

124. The apparatus according to any of the claims 118 to 123 comprising means for using as many samples of the depth map in the set of samples as there are samples in the first border of the sub-block.

125. The apparatus according to any of the claims 118 to 124 comprising means for repeating the selecting a position for a sub-block, determining a position for the sub-block, and forming a prediction of a location of an edge within the sub-block until a prediction of the edge within the block of the depth map has been formed.

126. The apparatus according to any of the claims 118 to 125 comprising means for using a vertical direction as the first direction, and a horizontal direction as the second direction.

127. The apparatus according to any of the claims 118 to 125 comprising means for using a horizontal direction as the first direction, and a vertical direction as the second direction.

128. The apparatus according to any of the claims 118 to 127 comprising means for using a macroblock of a video frame as the block of the depth map.

129. The apparatus according to claim 128, wherein the size of the block of the depth map is 16 x 16 pixels and the size of the sub-blocks is 4 x 4 pixels.

130. The apparatus according to claim 129, wherein the set of samples comprises 4 samples.

131. The apparatus according to any of the claims 118 to 130 the means for forming a prediction of a location of an edge comprising means for determining an edge traversal pattern within the sub-blocks, the edge traversal pattern being indicative of a shift of the edge from one sample to another sample in the first direction or in the second direction.

132. The apparatus according to claim 131 comprising means for decoding the edge traversal pattern.

133. The apparatus according to any of the claims 1 18 to 132, wherein the part of the picture comprises multiview video information.

134. The apparatus according to any of the claims 1 18 to 133, wherein the apparatus is a component of a mobile station.

135. A video coder configured for:

136. A video decoder configured for:

receiving an encoded location of an edge within a sub-block;

selecting a position for a sub-block in a depth map in a first direction, the sub-block comprising samples in a first direction and in the second direction;

decoding the encoded location of the edge within the sub-block; and