JP2015509345A - Video encoding method and apparatus - Google Patents

Abstract

A slice header prediction method and apparatus for 3D video encoding / decoding are provided. In some exemplary embodiments, a header prediction method can produce the following features: Any decoding order for texture and depth components is supported. It is also possible to flexibly predict syntax elements from any slice header that appears first in decoding order within the same access unit. Prediction is switched on and off based on the view component. The syntax elements of the slice header are classified into several sets. For each set, not only the prediction source but also the use of prediction can be controlled individually. Using an exemplary embodiment of such a method, all syntax elements of the slice header may be predicted. [Selection] Figure 8

Description

  The present application relates generally to an apparatus, method and computer program for encoding and decoding video.

background

  This section describes the background and context of the invention described in the claims. The discussion in this section may include concepts that may be pursued, and not necessarily only those that have already been conceived or pursued.

  Therefore, unless otherwise specified in the present application, the contents described in this section are not prior art to the specification and claims of this application, and are regarded as prior art only by what is described in this section. Must not.

  Currently, various technologies for providing three-dimensional (3D) video content are being researched and developed. In particular, intensive research has been conducted on various multi-view applications where only one set of stereo video can be viewed from a particular viewpoint, or another set of stereo video can be viewed from another viewpoint. One of the most feasible approaches to such a multi-view application is that only a limited number of input views, eg mono or stereo video and additional data are provided to the decoder side so that all necessary views are displayed on the display. It is understood that the data is rendered (ie, synthesized) locally by the decoder.

  Several view rendering techniques are available, for example, depth image-based rendering (DIBR) is seen as a competitive alternative. A typical implementation of DIBR takes as input stereoscopic video and corresponding depth information with a stereoscopic baseline, and synthesizes a number of virtual views between the two input views. For this reason, the DIBR algorithm can extrapolate not only between two input views, but also outside views. Similarly, the DIBR algorithm can synthesize a view from a single texture view and a corresponding depth view.

  Depending on the video coding standard, a header is introduced into the slice layer and its lower layer, or a parameter set concept is introduced into an upper layer of the slice layer. Examples of parameter sets may include sequence level data such as all pictures, groups of pictures (GOP), picture sizes and display windows, adopted option coding modes, macroblock allocation maps, and the like. Each example of the parameter set may include a unique identifier. Each slice header may include a reference to a parameter set identifier, and the parameter value of the referenced parameter set may be used when decoding that slice. The parameter set divides the order of transmission and decoding of low-frequency pictures and GOPs, and sequence-level data from sequences, GOPs, and picture boundaries. The parameter set may be transmitted out of band using a reliable transmission protocol as long as it is decoded before reference. When transmitted in-band, the parameter set may be repeated multiple times in order to increase error resilience over conventional video coding schemes. The parameter set may be transmitted at session setup time. However, in some systems such as a broadcast system, parameter set out-of-band transmission may not be feasible, and the parameter set is carried within the band by the NAL unit.

Abstract

  According to an exemplary embodiment of the present invention, a slice header prediction method and apparatus for 3D video encoding / decoding are provided. In some exemplary embodiments, a header prediction method can produce the following features: Any decoding order for texture and depth components is supported. In addition, syntax elements can be flexibly predicted from an arbitrary slice header that appears first in decoding order within the same access unit. Prediction is switched on and off based on the view component. The syntax elements of the slice header are classified into several sets. For each set, not only the prediction source but also the use of prediction can be controlled individually. Using an exemplary embodiment of such a method, all syntax elements of the slice header may be predicted.

  In some exemplary embodiments, the slice header prediction tool is summarized as follows. The syntax elements of the slice header are grouped into slice group (GOS, Group of Slice) parameter sets. The GOS parameter set may be maximally valid for the access unit. A GOS parameter set specified for the access unit may be created. The slice header in the texture view of the base view implicitly creates a GOS parameter set. The GOS parameter set may be included inline in the bitstream.

  In some exemplary embodiments, the GOS parameter set includes three types of syntax elements or structures. The GOS parameter set may include syntax elements that can be copied from the identified GOS parameter set. Such syntax structure includes reference picture list change, prediction weight table, and decoded reference picture marking. The GOS parameter set may also include a syntax structure that is invariant across view components. The GOS parameter set may include syntax elements that are invariant across access units.

  A GOS parameter set may inherit the syntax structure from multiple other GOS parameter sets. For example, the reference picture list change is inherited from a specific GOS parameter set, but the decoded reference picture marking may be inherited from another GOS parameter set.

  The GOS parameter set may be repeated. If the GOS parameter set is repeated at each slice, the same degree of error tolerance as having a complete slice header may be obtained.

  Various exemplary embodiments of the invention are presented in the claims.

According to the first aspect of the present invention, the following method is provided. The method includes encoding an uncompressed picture into an encoded picture that includes a slice, the encoding comprising:
Categorizing the syntax elements for the slice into a first set and a second set;
Determining syntax element values for the first set and the second set;
• selectively encoding the first set in a first slice group parameter set or slice header, wherein the encoding is:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
* Encoding the first set of syntax element values;
* The encoding comprising any one of omitting both;
• selectively encoding the second set in a second slice group parameter set or slice header, wherein the encoding is:
* Providing indication information of inclusion of said related second set from another slice group parameter set;
* Encoding the second set of syntax element values;
* Includes the encoding, including any one of omitting both.

According to a second aspect of the present invention, there is provided an apparatus comprising at least one processor and at least one memory containing computer program code. The at least one memory and the computer program code are stored in the device using the at least one processor:
Classifying the syntax elements for the slices that the encoded picture contains into a first set and a second set;
Determining syntax element values for the first set and the second set;
• selectively encoding the first set in a first slice group parameter set or slice header, the following:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
* Encoding the first set of syntax element values;
* The encoding comprising any one of omitting both;
Selectively encoding the second set in a second slice group parameter set or slice header, the following:
* Providing indication information of inclusion of said related second set from another slice group parameter set;
* Encoding the second set of syntax element values;
* Configured to perform the encoding, including any one of omitting both.

According to a third aspect of the invention, a computer program product is provided that includes one or more sequences of one or more instructions. When the one or more sequences of the one or more instructions are executed by one or more processors, the device at least:
Classifying the syntax elements for the slices that the encoded picture contains into a first set and a second set;
Determining syntax element values for the first set and the second set;
• selectively encoding the first set in a first slice group parameter set or slice header, the following:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
* Encoding the first set of syntax element values;
* The encoding comprising any one of omitting both;
Selectively encoding the second set in a second slice group parameter set or slice header, the following:
* Providing indication information of inclusion of said related second set from another slice group parameter set;
* Encoding the second set of syntax element values;
* Performing the encoding, including any one of omitting both.

According to the fourth aspect of the present invention, the following apparatus is provided. This device
Means for classifying syntax elements for slices included in the coded picture into a first set and a second set;
Means for determining syntax element values for the first set and the second set;
Means for selectively encoding said first set in a first slice group parameter set or slice header, comprising:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
* Encoding the first set of syntax element values;
* The means for encoding comprising any one of omitting both;
Means for selectively encoding said second set in a second slice group parameter set or slice header, comprising:
* Providing indication information of inclusion of said related second set from another slice group parameter set;
* Encoding the second set of syntax element values;
* The means for encoding comprises any one of omitting both.

According to the fifth aspect of the present invention, the following method is provided. The method includes decoding a coded slice of a coded picture, the decoding comprising:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first syntax element set and the second syntax element set to be used for decoding the encoded slice, and decoding the first and second sets The thing is:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

According to a sixth aspect of the invention, there is provided an apparatus comprising at least one processor and at least one memory containing computer program code. The at least one memory and the computer program code use the at least one processor to cause the apparatus to decode an encoded slice of an encoded picture, where the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first syntax element set and the second syntax element set to be used for decoding the encoded slice, and decoding the first and second sets The thing is:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

According to a seventh aspect of the present invention there is provided a computer program product comprising one or more sequences of one or more instructions. When the one or more sequences of the one or more instructions are executed by one or more processors, the device at least:
Decoding an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first syntax element set and the second syntax element set to be used for decoding the encoded slice, and decoding the first and second sets The thing is:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

According to the eighth aspect of the present invention, the following method is provided. The method includes decoding a coded slice of a coded picture, the decoding comprising:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first syntax element set and the second syntax element set to be used for decoding the encoded slice, and decoding the first and second sets The thing is:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first slice group parameter set Decoding the syntax elements of the set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second slice group parameter set Decoding the syntax elements of the set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

According to a ninth aspect of the present invention there is provided an apparatus comprising at least one processor and at least one memory containing computer program code. The at least one memory and the computer program code use the at least one processor to cause the apparatus to decode an encoded slice of an encoded picture, where the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first syntax element set and the second syntax element set to be used for decoding the encoded slice, and decoding the first and second sets The thing is:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first slice group parameter set Decoding the syntax elements of the set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second slice group parameter set Decoding the syntax elements of the set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

According to a tenth aspect of the present invention, there is provided a computer program product comprising one or more sequences of one or more instructions. When the one or more sequences of the one or more instructions are executed by one or more processors, the device at least:
Decoding an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first syntax element set and the second syntax element set to be used for decoding the encoded slice, and decoding the first and second sets The thing is:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first slice group parameter set Decoding the syntax elements of the set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second slice group parameter set Decoding the syntax elements of the set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

According to an eleventh aspect of the present invention, the following apparatus is provided. The apparatus is a method comprising means for decoding a coded slice of a coded picture, the means for decoding comprising:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Means for identifying;
Means for decoding the first syntax element set and the second syntax element set to be used for decoding the coded slice, and decoding the first and second sets Means:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first slice group parameter set Means for decoding the syntax elements of the set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second slice group parameter set Means for decoding the syntax elements of the set;
* Means for decoding the coded slice using the decoded first and second syntax element sets.

For a clearer understanding of exemplary embodiments of the invention, reference should be made to the following description, taken in conjunction with the accompanying drawings, in which:
1 shows a block diagram of a video encoding system according to an exemplary embodiment. FIG. 1 illustrates a video encoding apparatus according to an exemplary embodiment. 2 illustrates a video encoding configuration including a plurality of devices, networks and network elements according to an exemplary embodiment. FIG. 4 shows a block diagram of video encoding and decoding according to an exemplary embodiment. FIG. 4 shows a block diagram of video encoding and decoding according to an exemplary embodiment. A simplified model of a DIBR-based 3DV system is shown. A simple 2D model of a stereoscopic camera setup is shown. An example of access unit definition and coding order is shown. FIG. 5 illustrates a high level slow chart of an embodiment for an encoder that can encode texture views and depth views. FIG. FIG. 6 illustrates a high level slow chart of an embodiment for a decoder capable of decoding texture and depth views. Fig. 6 schematically shows the structure of an access unit according to an exemplary embodiment. An example of a component picture including one component picture delimited NAL unit and two coded slice NAL units is shown. An example of inter-CPD prediction of a slice parameter structure is shown.

Detailed Description of Embodiments

  Next, a plurality of embodiments of the present invention will be described in the context of a video coding configuration. However, it should be noted that the present invention is not limited to such a specific configuration.

  Actually, various embodiments can be widely applied in an environment where an improvement in handling of a reference picture is required. For example, the present invention may be applicable to a video encoding system such as a streaming system, a DVD player, a digital television receiver, a personal video recorder or system, a personal computer, a portable computer, or a computer program executed on a communication device. Furthermore, the present invention may be applicable to network elements such as a transcoder that handles video data and a cloud computing configuration.

  The H.264 / AVC standard is a video coding expert group (VCEG) from the ITU-T (International Telecommunication Union Telecommunication Standardization Sector) and video specialists from the ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) Developed by the Integrated Video Team (JVT) by the House Group (MPEG). The H.264 / AVC standard is published by both parental standards organizations and is called ITU-T recommendation H.264 and ISO / IEC international standard 14496-10. ISO / IEC14496-10 is known as MPEG-4 Part 10 Advanced Video Coding (AVC). There are several virgins in the H.264 / AVC standard, each of which integrates new extensions and specifications. Such extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

  Currently, a joint project between VCEG and MPEG (JCT-VC) is working on a standardization project for High Efficiency Video Coding (HEVC).

  In this section, important definitions of H.264 / AVC and HEVC, bitstreams, coding structures, and some of the concepts are described as examples of video encoders and decoders, coding methods, decoding methods, and bitstream structures. Embodiments of the invention may be implemented in such examples. Some important definitions, bitstreams, coding structures, and concepts in H.264 / AVC are identical to those in the current draft of the HEVC. Therefore, these are also described below. The aspect of the present invention is not limited to H.264 / AVC or HEVC. This specification is intended to illustrate possible principles for implementing some or all of the invention.

  As with many conventional video coding standards, H.264 / AVC and HEVC also define not only error-free bitstream decoding processing but also bitstream syntax and meaning. Although the encoding process is not defined, the encoder must always check the bit stream. The compatibility of the bit stream and the decoder can be verified using a hypothetical reference decoder (HRD). The standard includes coding tools that help counter transmission errors and transmission losses. However, the use of such a tool for encoding is optional, and no decoding process is specified for the wrong bitstream.

  The basic unit for input to an H.264 / AVC or HEVC encoder and output from an H.264 / AVC or HEVC decoder is a picture, respectively. In H.264 / AVC, a picture may be either a frame or a field. In the HEVC draft currently being created, the picture is a frame. The frame includes a matrix of luminance (luma) samples and corresponding color difference (chroma) samples. A field is a set of alternate sample rows of a frame and may be used as an encoder input if the source signal is interlaced. The color difference picture may be subsampled when compared to the luminance picture. For example, in the 4: 2: 0 sampling pattern, the spatial resolution of the color difference picture is half that of the luminance picture on both coordinate axes.

  In H.264 / AVC, 16 × 16 block luminance samples and corresponding color difference sample blocks are macroblocks. For example, in the 4: 2: 0 sampling pattern, the macroblock includes 8 × 8 block color difference samples for each color difference component. In H.264 / AVC, a picture is divided into one or more slice groups, and the slice group includes one or more slices. In H.264 / AVC, a slice is composed of an integer number of macro blocks, and is consecutive in the order of raster scan within a specific slice group.

  In the HEVC draft standard, a video picture is divided into a plurality of coding units (CU) covering a picture area. A CU consists of one or more prediction units (PU) and one or more conversion units (TU). PU specifies the prediction process for the samples in the CU, and TU specifies the encoding process of the prediction error for the samples in the CU. A CU usually consists of square sample blocks and has a size that can be selected from a set of predefined possible CU sizes. The CU with the maximum allowable size is usually called LCU (maximum coding unit), and the video picture is divided into non-overlapping LCUs. The LCU may be divided into smaller CU combinations, for example, by recursively dividing the LCU and the CU obtained as a result of the division. Each CU resulting from the split typically has at least one PU and at least one TU associated with it. Each PU and TU may be further divided into a plurality of smaller PUs and TUs in order to increase the granularity of the prediction process and the prediction error encoding process. The PU division may be performed by dividing the CU into four square PUs having the same size. Alternatively, it may be performed by dividing the CU vertically or horizontally into two rectangular PUs in a symmetric or asymmetric manner. Dividing a picture into CUs and CUs into PUs and TUs is usually conveyed by a bitstream signal that causes the decoder to reconstruct the desired structure from these units.

  In the HEVC draft standard, a picture is divided into tiles. The tile is rectangular and contains an integer number of LCUs. In the currently working HEVC draft, tile partitioning forms a regular grid, and the height and width of the tiles differ from one another by the largest LCU. In the HEVC draft, a slice consists of an integer number of CUs. The CUs are scanned within the tile or in the LCU raster scan order within the picture if no tiles are used. Within the LCU, the CU has a specific scan order.

  HEVC Working Draft (WD) 5 defines the main rules and concepts for picture partitioning as follows: Partitioning is defined as dividing a set into multiple subsets so that each element of a set is exactly one of the subsets.

  The basic coding unit of HEVC WD5 is a tree block. A tree block of a picture has three sample arrays, that is, luminance samples of N × N blocks and corresponding two blocks of color difference samples. Alternatively, it is a sample of N × N blocks for a monochrome picture or a picture encoded using three separate color planes. Tree blocks may be partitioned for separate encoding and decoding processes. Tree block partitioning (partitioning) has three sample arrays: one block of luminance samples obtained by tree block partition of a picture and two blocks of color difference samples corresponding thereto. Alternatively, a block of luminance samples that are obtained by tree block partitioning of a monochrome picture or a picture encoded using three separate color planes. Each tree block is assigned a partition signal that identifies a block size for intra or inter prediction encoding and a block size for transform encoding. Partitioning is recursive quadtree partitioning. The root of the quadtree is associated with the tree block. The quadtree is split until it reaches a leaf node, also called a coding node. The encoding node is a root node of two trees, a prediction tree and a transformation tree. The prediction tree specifies the position and size of the prediction block. Prediction data associated with a prediction tree is called a prediction unit. The transformation tree specifies the location and size of the transformation block. The conversion data associated with the conversion tree is called a conversion unit. The luminance and color difference division information is the same in the prediction tree, but may be the same or different in the conversion tree. The prediction unit and the conversion unit associated with the encoding node together form an encoding unit.

  In HEVC WD5, a picture is divided into slices and tiles. A slice may be a sequence of tree blocks, but (when called a so-called high-definition slice) there may be a boundary where the transform unit and the prediction unit in the tree block match. Tree blocks within a slice are encoded and decoded in raster scan order. Partitioning each picture into slices for the first coded picture is partitioning.

  In HEVC WD5, a tile is defined as an integer tree block that exists in one column or row, and this tree block is contiguous in the raster scan order within the tile. Partitioning each picture into tiles for the first coded picture is also partitioning. The tiles are consecutive in the raster scan order in the picture. A slice then contains tree blocks that are contiguous in raster scan order, but such tree blocks need not be contiguous in raster scan order within a picture. Also, slices and tiles need not contain the same tree block sequence. A tile may include tree blocks included in multiple slices. Similarly, one slice may include a tree block included in a plurality of slices.

  In H.264 / AVC and HEVC, prediction across slice boundaries in a picture may be invalid. Thus, a slice may be considered as a way to divide an encoded picture into parts that are independently decoded and is therefore often considered the basic unit of transmission. In many cases, the encoder may indicate in the bitstream which types of intra-picture prediction are stopped when crossing a slice boundary. This information is taken into account when determining which prediction sources are available by the operation of the decoder. For example, when an adjacent macroblock or CU exists in another slice, it may be considered that samples from the adjacent macroblock or CU cannot be used for intra prediction.

  A syntax element is defined as an element of data represented by a bit stream. A syntax structure is defined as zero or more elements of data represented by a particular order of bitstreams.

  The basic units for output from an H.264 / AVC or HEVC encoder and input to an H.264 / AVC or HEVC decoder are network abstraction layer (NAL) units, respectively. For transmission over packet-oriented networks and storage in structured files, NAL units may be encapsulated in packets or similar structures. In H.264 / AVC and HEVC, a byte stream format is specified for a transmission or storage environment that does not provide a frame structure. The byte stream format separates NAL units from each other by adding a start code to the head of each NAL unit. To prevent false detection of NAL unit boundaries, the encoder performs a byte oriented start code emulation prevention algorithm. This adds an emulation prevention byte to the NAL unit payload if the start code occurs in another way. In order to allow direct gateway operation between packet-oriented and stream-oriented systems, start code emulation prevention may always be performed regardless of whether the byte stream format is used.

  A NAL unit consists of a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of NAL unit and whether the encoded slice included in the NAL unit is a reference picture or a non-reference picture. H.264 / AVC includes a 2-bit syntax element nal_ref_idc. When this is 0, it indicates that the encoded slice included in the NAL unit is part of a non-reference picture, and when it exceeds 0, it is included in the NAL unit. This indicates that the included encoded slice is a part of the reference picture. The HEVC draft includes a 1-bit syntax element nal_ref_idc and is also called nal_ref_flag. When this is 0, it indicates that the encoded slice included in the NAL unit is a part of the non-reference picture, and when it is 1, it indicates that the encoded slice included in the NAL unit is a part of the reference picture. SVC and MVC NAL unit headers may additionally contain various indications related to extensibility and multi-view hierarchy. In HEVC, the NAL unit header includes a syntax element temporal_id and specifies a time identifier for the NAL unit. The bitstream generated by excluding all VCL-NAL units with a temporal_id greater than the selected value and including all other VCL-NAL units is compatible. As a result, a picture having temporal_id equal to TID does not use any picture having temporal_id exceeding TID as an inter prediction reference. In the HEVC draft, initialization of the reference picture list is limited to only reference pictures that are marked “used for reference” and whose temporal_id is less than or equal to temporal_id of the current picture (current picture).

  NAL units are classified into Video Coding Layer (VCL) NAL units and non-VCL-NAL units. A VCL-NAL unit is usually a coded slice NAL unit. In H.264 / AVC, a coded slice NAL unit includes syntax elements representing one or more coded macroblocks, each corresponding to a sample block of an uncompressed picture. In HEVC, a coded slice NAL unit includes a syntax element that represents one or more CUs. In H.264 / AVC and HEVC, a coded slice NAL unit may be indicated to be a coded slice of an Instantaneous Decoding Refresh (IDR) picture or a coded slice of a non-IDR picture. In HEVC, a coded slice NAL unit may be indicated as a coded slice of a Clean Decoding Refresh (CDR) picture (also called a Clean Random Access picture).

  The non-VCL-NAL unit may be, for example, one of the following types: sequence parameter set; picture parameter set; supplemental enhancement information (SEI) NAL unit; access unit delimiter; sequence NAL unit termination; stream NAL unit Termination; or supplementary data NAL unit. The parameter set may be necessary for the reconstruction of the decoded picture, but many of the other non-VCL-NAL units are not necessary for the reconstruction of the decoded sample values.

  Parameters that are unchanged in the encoded video sequence may be included in the sequence parameter set. In addition to the parameters that are important for the decoding process, the sequence parameter set may include video usability information (VUI). This includes important parameters for buffering, picture output timing, rendering, and resource reservation. In H.264 / AVC, three NAL units including a sequence parameter set are defined. The sequence parameter set NAL unit includes all the data for the H.264 / AVC VCL-NAL unit in the sequence. The sequence parameter set extended NAL unit includes data for auxiliary coded pictures. The subset sequence parameter set NAL unit is for the MCL and SVC VCL-NAL units. The picture parameter set includes parameters that are unchanged in a plurality of encoded pictures.

  In the HEVC draft, there is a third type of parameter set called an adaptation parameter set (APS), which includes parameters that are invariant in a plurality of coded pictures. In the HEVC draft, the APS syntax structure is a parameter or syntax element related to context-based adaptive binary arithmetic coding (CABAC), adaptive sample offset, adaptive loop filtering, or deblocking filtering. including. In the HEVC draft, the APS is a NAL unit that is encoded without reference or prediction from other NAL units. An identifier called the syntax element aps_id is included in the APS-NAL unit. This is also included in the slice header and is used to represent a specific APS.

  The syntax of H.264 / AVC and HEVC allows various parameter instances, and each instance is identified by a unique identifier. In H.264 / AVC, each slice header includes an identifier of a picture parameter set active for decoding a picture including the slice. Each picture parameter set includes an identifier of the active sequence parameter set. As a result, the transmission of pictures and sequence parameter sets need not be precisely synchronized with the transmission of slices. In fact, it is sufficient that the active sequence and picture parameter sets are received before they are referenced, and parameters are set “out of band” using a more reliable transmission mechanism than the protocol for slice data. The set can be transmitted. For example, the parameter set may be included as a parameter in a session description for a Real-time Transport Protocol (RTP) session. The parameter set may be repeated to increase error tolerance when transmitted in-band.

  A SEI-NAL unit may contain one or more SEI messages. These are not necessary for decoding the output picture, but assist related processing such as picture output timing, error detection, error concealment, and resource reservation. A plurality of SEI messages are defined by H.264 / AVC and HEVC, and SEI messages uniquely used by organizations and companies are also defined by SEI messages of user data. H.264 / AVC and HEVC include the prescribed SEI message syntax and meaning, but nothing is defined about the message handling on the receiving side. As a result, the encoder needs to comply with the H.264 / AVC standard or HEVC standard, and the decoder must comply with the H.264 / AVC standard or HEVC standard, respectively, when creating the SEI message. However, it is not necessary to process SEI messages according to the output regulations. One reason for including the syntax and meaning of SEI messages in H.264 / AVC and HEVC is to allow the same information to be interpreted and interoperated in different system specifications in the same way. The system specification requires that a specific SEI message be enabled on both the encoding side and the decoding side, and a process for handling a specific SEI message may be defined on the receiving side.

  An encoded picture is an encoded representation of a picture. An encoded picture in H.264 / AVC includes a VCL-NAL unit necessary for decoding the picture. In H.264 / AVC, a coded picture is a primary coded picture or a redundant coded picture. The primary encoded picture is used in the effective bitstream decoding process. On the other hand, the redundant coded picture is a redundant representation that is decoded only when the primary coded picture is not correctly decoded. In the HEVC draft, redundant coded pictures are not defined.

  In H.264 / AVC and HEVC, an access unit includes a primary encoded picture and a NAL unit associated therewith. In H.264 / AVC, the order of appearance of NAL units within an access unit is limited as follows. The NAL unit separated by the additional access unit can indicate the starting point of the access unit. This is followed by zero or more SEI-NAL units. The encoded slice of the primary encoded picture appears next. In H.264 / AVC, an encoded slice of zero or more redundant encoded pictures may follow an encoded slice of a primary encoded picture. A redundant coded picture is a coded representation of a picture or part of a picture. The redundant coded picture may be decoded when the decoder cannot receive the primary coded picture due to transmission loss, data corruption in the physical storage medium, or the like.

  In H.264 / AVC, the access unit may include auxiliary coded pictures. This is a picture that supplements / complements the primary encoded picture and can be used in display processing or the like. The auxiliary encoded picture may be used as, for example, an alpha channel or an alpha plane that specifies a transmission level of a decoded picture sample. The alpha channel or alpha plane may be used in layer components and rendering systems. The output picture is created by superimposing pictures that are at least partially transparent on the surface. The auxiliary coded picture has the same syntax and meaning limitation as the monochrome redundant coded picture. In H.264 / AVC, the auxiliary encoded picture includes the same number of macroblocks as the primary encoded picture.

  An encoded video sequence is defined as a sequence of consecutive access units. This sequence is the order of decoding processing, including the IDR access unit, from there to the next IDR access unit, immediately before it or until the last occurrence of the bitstream.

  A picture group (GOP) and its characteristics may be defined as follows. The GOP is decoded regardless of whether the previous picture was decoded. An open GOP is a picture group in which when a decoding process starts from the first intra picture, pictures ahead of the first intra picture in the output order are not correctly decoded. In other words, an open GOP picture may refer to a picture belonging to the previous GOP (by inter prediction). The H.264 / AVC decoder can recognize an intra picture at the beginning of an open GOP by a recovery point SEI message in the H.264 / AVC bitstream. The HEVC decoder can recognize the first intra picture of the open GOP. This is because the CDR-NAL unit type, which is a special NAL unit type for the coded slice, is used. A closed GOP is a picture group in which all pictures are correctly decoded when the decoding process starts from the first intra picture. In other words, in a closed GOP, there is no picture that refers to a picture belonging to the previous GOP. In H.264 / AVC and HEVC, a closed GOP starts with an IDR access unit. As a result, the closed GOP structure has a higher error recovery capability than the open GOP structure. However, it comes at the price of potentially reducing compression efficiency. The open GOP coding structure allows for more efficient compression with high flexibility in reference picture selection.

  The H.264 / AVC and HEVC bitstream syntax indicates whether a particular picture is a reference picture for intra prediction of another picture. Pictures of any coding type (I, P, B) can be H.264 / AVC and HEVC reference pictures or non-reference pictures. The NAL unit header indicates the type of NAL unit and whether the encoded slice included in the NAL unit is a reference picture or a non-reference picture.

  Many hybrid video codecs, including H.264 / AVC and HEVC, encode video information in two stages. In the first stage, pixel or sample values for a particular picture region or “block” are predicted. Such pixel or sample values can be predicted, for example, by a motion compensation mechanism. This mechanism includes the search and labeling of regions in one of the previously encoded video frames that correspond closely to the block to be encoded. In addition, pixel or sample values may be predicted by a spatial mechanism that includes spatial domain relationship retrieval and marking.

  A prediction approach using image information from a previously encoded image is also referred to as an inter prediction method, and is also referred to as temporal prediction and motion compensation. A prediction approach using image information in the same image is also called an intra prediction method.

  The second stage is any encoding of the error between the predicted block of pixels or samples and the original block of pixels or samples. This may be achieved by converting the difference between pixel values or sample values using a specific transformation. This transformation may be a discrete cosine transform (DCT) or a modification thereof. After transforming the difference, the transformed difference is quantized and entropy coded.

  By changing the fidelity of the quantization process, the encoder can correct the accuracy of the pixel or sample representation (ie the visual quality of the picture) and the size of the resulting encoded video representation (ie the file size and transmission bit rate). The balance between can be controlled.

  The decoder reconstructs the output video by applying a prediction mechanism similar to that used by the encoder to form a block representation of the predicted pixels or samples and decode the prediction error (where prediction Representation formation uses motion information and spatial information created by the encoder and stored in the compressed representation of the image, and prediction error decoding recovers the prediction error signal quantized in the spatial domain. Is done using the reverse operation).

  After pixel or sample prediction and error decoding processing, the decoder combines the prediction signal and the prediction error signal (pixel value or sample value) to form an output video frame.

  The decoder (and encoder) may also apply additional filtering processing to improve the quality of the output video before sending it to the display and / or storing it as a predictive reference for subsequent pictures in the video sequence. Good.

  In many video codecs, including H.264 / AVC and HEVC, motion information is indicated by a motion vector associated with each of the motion compensated image blocks. Each of these motion vectors is an image block of a picture to be encoded (by an encoder) or a picture to be decoded (by a decoder) and a predictor block in one of the previously encoded or decoded pictures (or pictures). Represents the amount of movement between. H.264 / AVC and HEVC, like many other video compression standards, divide a picture into rectangular meshes. For each of these rectangles, the same block in one of the reference pictures is shown for inter prediction. The position of the prediction block is encoded as a motion vector indicating the relative position of the prediction block with respect to the block to be encoded.

  The inter prediction process may be characterized by one or more of the following factors.

Accuracy of motion vector representation

  For example, the motion vector may be 1/4 pixel accurate, and the sample value at the fractional pixel location may be obtained using a finite impulse response (FIR) filter.

Block partitioning (partitioning) for inter prediction

  Many coding standards, including H.264 / AVC and HEVC, can select the block size and shape for the motion vector applied for motion compensated prediction at the encoder, and the motion performed at the encoder The selected size and shape can be shown in the bitstream so that the compensated prediction can be reconstructed by the decoder.

Number of reference pictures for inter prediction

  The original data of inter prediction is a previously decoded picture. In many coding standards including H.264 / AVC and HEVC, a plurality of reference pictures can be stored for inter prediction, and a reference picture to be used can be selected according to a block bias. For example, the reference picture may be selected with respect to the bias of the macroblock or macroblock pattern in H.264 / AVC, or the bias of the PU or CU of HEVC. Many encoding standards such as H.264 / AVC and HEVC include in the bitstream a syntax structure that allows the decoder to create one or more reference picture lists. A reference picture index indicating a reference picture list may be used to indicate which of a plurality of reference pictures is used for inter prediction for a specific block. The reference picture index may be encoded into the bitstream by some inter-coding method by the encoder, or may be extracted (by the encoder and decoder) using adjacent blocks, etc., by another inter-coding method. Good.

In order to efficiently represent the motion vector prediction motion vector in the bitstream, the motion vector may be differentially encoded with respect to the prediction motion vector for each block. In many video codecs, the motion vector predictor is generated in a predetermined manner, for example by calculating the median of the coding / decoding motion vectors of neighboring blocks. Another method for performing motion vector prediction is to create a list of prediction candidates from adjacent blocks and / or coexistence blocks in a reference picture on the time axis, and signal the selected candidates as motion vector predictions. In addition to prediction of motion vector values, a reference index of a previously encoded / decoded picture may be predicted. The reference index is usually predicted from adjacent blocks and / or coexistence blocks in a reference picture on the time axis. Differential encoding of motion vectors is usually disabled when crossing slice boundaries.

Multi-hypothesis motion compensated prediction
In H.264 / AVC and HEVC, a single prediction block can be used in a P slice (for this reason, the P slice is called a single prediction slice). Also, for bi-predictive slices, also called B slices, a linear combination of two motion compensated prediction blocks can be used. The individual block of the B slice may be bi-predicted, uni-predicted or intra-predicted, and the individual block of the P slice may be uni-predicted or intra-predicted. The reference picture for the bi-predictive picture is not limited to the subsequent picture and the preceding picture in the output order, and an arbitrary reference picture may be used. In many coding standards such as H.264 / AVC and HEVC, a specific reference picture list, called reference picture list 0, is configured for P slices, and two reference picture lists, list 0 and list 1 Is configured for B slices. For B slices, forward prediction refers to prediction from reference pictures in reference picture list 0, and backward prediction refers to prediction from reference pictures in reference picture list 1. Here, the prediction reference pictures may have a decoding process and an output order related to each other or to the current picture.

Weighted prediction Many coding standards use a prediction weight of 1 for prediction blocks of inter (P) pictures and a prediction weight of 0.5 for each prediction block of B pictures (as a result of averaging). In H.264 / AVC, weighted prediction can be performed in both P and B slices. In implicit weighted prediction, the weight is proportional to the picture order count, and in positive weighted prediction, the prediction weight is explicitly indicated.

  In many video codecs, motion compensated prediction residuals are first transformed with a transformation kernel (such as DCT) and then encoded. This is because there is usually also a correlation between the residuals, and such transformations often help to reduce such correlations and allow for more efficient coding.

  In the HEVC draft, each PU defines prediction types that are applied to pixels in each PU, and prediction information associated with each PU (eg, motion vector information for inter-predicted PUs, Intra-predicted PUs have intra-prediction direction information). Similarly, each TU is associated with information (including DCT coefficient information and the like) describing prediction error decoding processing for samples in each TU. Whether or not prediction error coding is applied to each CU may be transmitted at the CU level. If there is no residual prediction error associated with a CU, it is assumed that there is no TU for that CU.

  Depending on the encoding format and codec, a so-called short-term reference picture is distinguished from a long-term reference picture. Such distinction may affect some decoding processes as scaling of motion vectors in temporal direct mode or implicit weighted prediction. If both reference pictures used for temporal direct mode are short-term reference pictures, the motion vectors used in the prediction are scaled according to the difference in picture order count (POC) between the current picture and each reference picture. Also good. However, if at least one reference picture for temporal direct mode is a long-term reference picture, default motion vector scaling may be used, for example, scaling the motion in half. Similarly, when a short-term reference picture is used in shadow weighted prediction, the prediction weight may be scaled according to the POC difference between the POC of the current picture and the POC of the reference picture. However, when a long-term reference picture is used in implicit weighted prediction, a default prediction weight may be used, and 0.5 or the like may be used in implicit weighted prediction for a bi-prediction block.

  A video encoding format such as H.264 / AVC includes a syntax element frame_num and is used for various decoding processes related to a plurality of reference pictures. In H.264 / AVC, the frame_num value of the IDR picture is 0. The frame_num value of the non-IDR picture is equal to the value obtained by adding 1 to the frame_num of the previous reference picture in the order of 0 decoding (in the case of a modulo operation, the frame_num value returns to 0 after the maximum value).

  H.264 / AVC and HEVC include the concept of picture order count (POC). The POC value is given to each picture and does not decrease as the order of pictures in the output increases. Therefore, POC indicates the output order of pictures. The POC may be used in decoding processing, for example, implicit scaling of motion vectors in the temporal direct mode of bi-predictive slices, weights generated implicitly in weighted prediction, initialization of a reference picture list, and the like. POC may also be used to verify output order conformance. In H.264 / AVC, a POC is specified in relation to a previous IDR picture or a picture that includes a memory management control operation that marks all pictures as “unused for reference”.

  H.264 / AVC specifies decoding reference picture marking processing in order to control memory consumption at the decoder. The maximum number of reference pictures used for inter prediction is represented by M and is determined by a sequence parameter set. A reference picture is marked “used for reference” when it is decoded. If the number of pictures that are marked “used for reference” in decoding a reference picture exceeds M, at least one picture is marked “unused for reference”. There are two types of marking operations for decoding reference pictures: adaptive memory control and sliding window. The marking operation mode of the decoded reference picture is selected based on the picture. Adaptive memory control may be explicitly signaled which pictures are marked “unused for reference” and may assign a long-term index to the short-term reference picture. Adaptive memory control may require the presence of a memory management control operation (MMCO) parameter in the bitstream. The MMCO parameter may be included in the syntax element of the decoded reference picture marking. If sliding window mode of operation is used and M pictures are marked as "used for reference", the first decoded short-term reference in the short-term reference picture marked as "used for reference" The picture is marked “unused for reference”. In other words, the sliding window mode of operation is a first-in-first-out buffer operation for short-term reference pictures.

  Depending on the memory management control operation of H.264 / AVC, all reference pictures other than the current picture are marked as “unused for reference”. Instantaneous decoding refresh (IDR) pictures contain only intra-coded slices and perform the same “reset” on the reference picture.

  In the HEVC draft, the decoding process related to the syntax structure of the reference picture marking is replaced with the syntax structure of the reference picture set (RPS). Instead, the decryption process is used for the same purpose. A reference picture set that is valid or active for a particular picture is all the reference pictures that are used as references for that picture, and all that remain marked "used for reference" for any subsequent picture in decoding order Of reference pictures. The reference picture set has six subsets, which are called RefPicSetStCurr0, RefPicSetStCurr1, RefPicSetStFoll0, RefPicSetStFoll1, RefPicSetLtCurr, and RefPicSetLtFoll, respectively. The notation for these six subsets is as follows: “Curr” represents a reference picture included in the reference picture list of the current picture. For this reason, it may be used as an inter prediction reference for the current picture. “Foll” represents a reference picture not included in the reference picture list of the current picture. However, it may be used as a reference picture in subsequent pictures in decoding order. “St” represents a short-term reference picture and is usually identified by a specific number of least significant bits of the POC value. “Lt” represents a long-term reference picture and is identified in a specific way. Normally, the difference in POC value for the current picture is greater than that represented by the specific number of least significant bits described above. “0” represents a reference picture having a POC value smaller than the POC value of the current picture. “1” represents a reference picture having a POC value larger than the POC value of the current picture. RefPicSetStCurr0, RefPicSetStCurr1, RefPicSetStFoll0, and RefPicSetStFoll1 are collectively referred to as the short-term subset of the reference picture set, and RefPicSetLtCurr and RefPicSetLtFoll are collectively referred to as the long-term subset of the reference picture set. The reference picture set may be specified in the picture parameter set and captured for the slice header via an index to the reference picture set. The reference picture set may be specified by a slice header. The long-term subset of the reference picture set is usually specified only by the slice header, but the short-term subset of the same reference picture set may be specified by the picture parameter set or may be specified by the slice header. Pictures included in the reference picture set used by the current slice (current slice) are marked as “used for reference”, and pictures not included in the reference picture set used by the current slice are marked as “unused for reference” . If the current picture is an IDR picture, RefPicSetStCurr0, RefPicSetStCurr1, RefPicSetStFoll0, RefPicSetStFoll1, RefPicSetLtCurr, and RefPicSetLtFoll are all set to empty.

  A decoded picture buffer (DPB) may be used in an encoder and / or a decoder. There are two reasons for buffering decoded pictures. One is for reference in inter prediction, and the other is for rearranging decoded pictures in the order of output. Because H.264 / AVC and HEVC provide considerable flexibility in both reference picture marking and output reordering, using separate buffers for reference picture buffering and output picture buffering wastes memory resources. there's a possibility that. For this reason, the DPB may include a reference picture and an integrated decoded picture buffering process for output rearrangement. The decoded picture may be deleted from the DPB when it is no longer used as a reference and need not be output.

  In many coding modes such as H.264 / AVC and HEVC, the inter prediction reference picture is indicated by an index to the reference picture list. The index is encoded by CABAC or variable length encoding. In general, the smaller the index, the shorter the corresponding syntax element. Two reference picture lists (reference picture list 0 and reference picture list 1) are created for each bi-prediction (B) slice, and one reference picture list (reference picture) for each inter-prediction (P) slice. • List 0) is formed.

  Typical high efficiency video codecs, such as HEVC draft codecs, use additional motion information encoding / decoding mechanisms and are commonly referred to as merging processes / mechanisms or merge mode processes / mechanisms. As a result, all motion information of the block / PU is predicted and used without being changed / modified. The aforementioned motion information for PU includes: 1) PU is uni-predicted using only reference picture list 0, PU is uni-predicted using only reference picture list 1, or Information on whether the PU is uni-predicted using both reference picture list 0 and reference picture list 1; 2) motion vector value corresponding to reference picture list 0; 3) reference picture in reference picture list 0 Index; 4) Motion vector value corresponding to reference picture list 1; 5) Reference picture index in reference picture list 1. Similarly, prediction of motion information is performed using motion information of adjacent blocks and / or coexistence blocks in a reference picture on the time axis. Usually, a list called a merge list is formed by including motion prediction candidates related to available adjacent / coexistence blocks, and an index of motion prediction candidates selected in the list is signaled. Thus, the selected candidate motion information is copied to the current PU motion information. This type of encoding / decoding for CUs is usually done in skip mode or merge base when the merge mechanism is used throughout the CU and the prediction signal for the CU is used as a reconstructed signal, ie when the prediction residual is not processed.・ This is called skip mode. For each PU, a merge mechanism is also used in addition to the skip mode, in which case the prediction residual may be utilized to improve the prediction quality. This type of prediction mode is usually referred to as an intermerged mode.

  Reference picture lists such as reference picture list 0 and reference picture list 1 may be created in two steps. In the first step, an initial reference picture list is created. The initial reference picture list may be created based on prediction layer information such as frame_num, POC, temporal_id, GOP structure, or a combination thereof. In the second step, an initial reference picture list may be stored by a reference picture list reordering (RPLR) instruction. The RPLR instruction is also called a reference picture list change syntax structure and is included in the slice header. The RPLR instruction indicates a picture arranged at the head of each reference picture list. The second step is also called a reference picture list change process, and an RPLR instruction may be included in the reference picture list change syntax structure. If a reference picture set is used, reference picture list 0 may be initialized to include RefPicSetStCurr0, RefPicSetStCurr1, RefPicSetLtCurr in this order. Reference picture list 1 may be initialized to include RefPicSetStCurr1, RefPicSetStCurr0 in this order. The initial reference picture list may be changed through a reference picture list change syntax structure. Pictures in the initial reference picture list may be identified through an entry index for the list.

  The merge list may be created based on reference picture list 0 and / or reference picture list 1, and may be created using, for example, a reference picture list combination syntax structure included in the slice header syntax. . The reference picture list joint syntax structure may be created in the bitstream at the encoder and decoded from the bitstream at the decoder. This syntax structure indicates the contents of the merge list. This syntax structure may indicate that reference picture list 0 and reference picture list 1 are combined and is another reference picture list combination for a prediction unit that is unidirectionally predicted. This syntax structure includes a flag. When the flag is equal to a specific value, reference picture list 0 and reference picture list 1 are the same, and reference picture list 0 is used as a combination of reference picture lists. May be shown. This syntax structure may include a list of entries, each entry specifying a reference picture list (list 0 or list 1), and a reference index for the specified list. The entry specifies a reference picture to be included in the merge list.

  A syntax structure for decoding reference picture marking may be present in the video coding system. For example, when the decoding of a picture is complete, if there is a marking syntax structure for the decoded reference picture, it will adaptively mark the picture as "unused for reference" or "used for long-term reference" May be used. If there is no decoded reference picture marking syntax structure and the number of pictures marked “used for reference” will not increase any more, the reference picture marking in the sliding window may be used. This basically marks the reference as the unused reference picture that was decoded first (in decoding order).

  The syntax structure of the reference picture list may include three parts: a reference picture list 0 description, a reference picture list 1 description, and an idle reference picture list description. The reference picture list 0 description is for P and B slices, and the reference picture list 1 description is for B slices. The idle reference picture list description is for slices that contain a reference picture that is not included in either reference picture list 0 or 1, but remains marked “used for reference”. In other words, it may be one syntax structure or the like that provides information for both reference picture marking and reference picture list creation.

  When decoding of the slice is started, the syntax structure of the reference picture list may be analyzed. For P and B slices, the syntax structure includes a reference picture list description for list 0 and is decoded. The syntax structure of the reference picture list description may list pictures identified in the order in which picture order count (POC) values appear in the reference picture list. For B slices, the reference picture list syntax structure includes the reference picture list description for list 1 and is decoded.

  The initialization process of the reference picture list and / or the change process of the reference picture list may be omitted, and the reference picture list may be described directly in the syntax structure.

  Alternatively or additionally, the reference picture list syntax structure may include a reference picture list description for the idle reference picture list, which is decoded if present.

  Pictures in any reference picture list may be marked as “used for reference”. Pictures that are not in any reference picture list may be marked “unused for reference”.

  In other words, reference picture list creation and reference picture marking processing and syntax structure may be handled by a single integration processing and syntax structure.

  Even if the reference pictures in the idle reference picture list have a specific order determined by the syntax structure of the reference picture list description, usually the specific requirements regarding the order in which the encoder should list the idle reference pictures Note that does not exist. In a sense, an idle reference picture list can often be considered an unordered list or set.

  Reference picture list 0 and list 1 may include reference pictures that are indicated as unused in the current slice reference. For example, a reference index exceeding num_ref_idx_l0_active_minus1 may not be used for the current slice reference. It may be specified that unreferenced reference pictures in reference picture list 0 and list 1 are marked as “used for reference”. Or, if a specific reference picture is only included in list 0 and list 1 as an unreferenced reference picture and not included in the idle reference picture list, or list 0 and list 1 as referenced reference pictures The reference picture may be specified to be marked “unused for reference”. Alternatively, the desired marking rules between the above two cases and other deterministic marking rules may be controlled by the encoder and shown in the bitstream, such as with SPS syntax in a continuous parameter set. The inclusion of a reference picture as a non-reference picture in list 0 or list 1 may be prioritized over inclusion in an idle reference picture list, for example if fewer bits are consumed during encoding.

  In scalable video coding, a video signal is encoded into a base layer and one or more enhancement layers. An enhancement layer may increase temporal resolution (ie, frame rate), spatial resolution, or simply increase the quality of video content represented by another layer or part thereof. Each layer, together with all its respective subordinate layers, represents a representation of the video signal at a particular spatial resolution, temporal resolution and quality level. In this application, a scalable layer with all dependent layers is referred to as a “scalable layer representation”. In order to generate the original signal representation with specific fidelity, a portion of the scalable bitstream corresponding to the scalable layer representation is extracted and decoded.

  In some cases, enhancement layer data may be truncated after a specific or arbitrary position. Here, each truncation position may include additional data that expresses with higher visual quality. Such scalability is called fine-grained / granularity scalability (FGS). FGS was included as part of the draft version of the SVC standard, but was excluded from the final version of the SVC standard. Therefore, in the following, FGS will be explained using a part of the draft version of the SVC standard. The scalability provided by the untruncated enhancement layer is called coarse-grained / granularity scalability (CGS). This includes traditional quality (SNR) scalability and spatial scalability together. The SVC standard supports so-called medium-grained / granularity scalability (MGS). In MGS, a high-quality picture is encoded in the same way as an SNR scalable layer picture, but is indicated by the syntax element quality_id exceeding 0 using the same high-level syntax element as the FGS layer picture.

  SVC uses an inter-layer prediction mechanism and can predict specific information from layers other than the currently reconfigured layer or from the next lower layer. The information that can be predicted between layers includes intra texture, motion, and residual data. Inter-layer motion prediction includes prediction of block coding mode, header information, and the like, and motion from a lower layer may be used for prediction of an upper layer. In the case of intra coding, prediction from surrounding macroblocks and coexisting macroblocks in the lower layer is possible. Such a prediction technique is called an intra prediction technique because it does not use information from previously encoded access units. Further, residual data from the lower layer is also used for prediction of the current layer.

  SVC specifies a concept called single loop decoding. This can be achieved by using the intra-constrained texture prediction mode. Inter-layer intra-texture prediction is a macroblock (MB), and can be applied to an MB in which the corresponding block of the base layer is located. At the same time, such intra MBs in the base layer use constrained intra prediction (eg, the syntax element “constrained_intra_pred_flag” is equal to 1). In single loop decoding, the decoder performs motion compensation and full picture reconstruction only for the scalable layer desired for playback (referred to as the “desired layer” or “target layer”). Thus, the decoding complexity can be greatly reduced. All layers other than the desired layer need not be completely decoded. This is because all or part of MB data that is not used for inter-layer prediction (inter-layer texture intra prediction, inter-layer motion prediction, or inter-layer residual prediction) is not necessary for the reconstruction of the desired layer.

  While a single decoding loop is necessary for decoding most pictures, the second decoding loop is selectively applied to reconstruct the base representation. This base representation is necessary as a prediction reference, but need not be output or displayed, so it is reconstructed only for so-called key pictures ("store_ref_base_pic_flag" equals 1).

  The scalability structure in the SVC draft is characterized by three syntax elements: "temporal_id", "dependency_id", and "quality_id". The syntax element “temporal_id” is used to indicate a temporal scalability hierarchy or indirectly a frame rate. The frame rate of the scalable layer representation including a picture having a small maximum value of “temporal_id” is lower than the frame rate of the scalable layer representation including a picture having a large maximum value of “temporal_id”. A given time layer typically depends on a lower time layer (ie, a time layer with a smaller value of “temporal_id”), but not on any upper time layer. The syntax element “dependency_id” is used to indicate a CGS inter-layer coding dependency hierarchy (including both SNR and spatial scalability as described above). A picture with a small “dependency_id” value at any temporal level position may be used for inter-layer prediction in coding of a picture with a large “dependency_id” value. The syntax element “quality_id” is used to indicate the quality level hierarchy of the FGS or MGS layer. If the same “dependency_id” value is used at any time level position, a picture whose “quality_id” value is equal to QL uses a picture whose “quality_id” value is equal to QL-1 for inter-layer prediction. An encoded slice with a “quality_id” greater than 0 may be encoded as either a truncable FGS slice or a non-truncable MGS slice.

  For simplicity, all data units (such as network abstraction layer units / NAL units in the case of SVC) in access units with the same “dependency_id” value are called dependency units or dependency expressions. Within one dependent unit, all data units with the same “quality_id” value are called quality units or layer representations.

  A base representation, also called a decoded base picture, is a decoded picture obtained from the decoding result of a video coding layer (VCL) NAL unit in a dependent unit whose “quality_id” value is equal to 0, and “store_ref_base_pic_flag” is set to 1. An extended representation, also called a decoded picture, is obtained from a normal decoding process result, and all layer representations existing for the maximum dependent representation are decoded.

  As mentioned above, CGS includes both spatial scalability and SNR scalability. Spatial scalability is initially designed to support video representations with different resolutions. For each time instance, VCL-NAL units are encoded with the same access unit, and these VCL-NAL units correspond to different resolutions. During decoding, the low resolution VCL-NAL unit provides motion fields and residuals. These may be inherited by final decoding and reconstruction of high resolution pictures. Compared to conventional video compression standards, the spatial scalability of SVC is generalized so that the base layer can be a cropped and zoomed version of the enhancement layer.

  The MGS quality layer is indicated by “quality_id” similarly to the FGS quality layer. For each dependency unit (having the same “dependency_id”), there is a layer whose “quality_id” is equal to 0, and there may be other layers whose “quality_id” is greater than 0. Such layers with a "quality_id" greater than 0 are either MGS layers or FGS layers depending on whether the slice was encoded as a truncable slice.

  In the basic form of the FGS enhancement layer, only inter-layer prediction is used. Therefore, the FGS enhancement layer is freely truncated without propagating errors in the decoding sequence. However, the basic form of FGS has low compression efficiency. This problem arises because only low quality pictures are used for inter prediction references. Therefore, the use of FGS extended pictures as inter prediction references has been proposed. However, even with these proposals, when a part of FGS data is discarded, there is a possibility that a mismatch between encoding and decoding called drift occurs.

  The feature of the SVC draft standard is that the FGS-NAL unit is freely discarded or truncated, but the feature of the SVCV standard is that the MGS-NAL unit is freely discarded without compromising the bitstream compatibility (but Cannot be truncated). As described above, when such FGS or MGS data is used for inter prediction reference at the time of encoding, the discarding or truncation of the data causes a decoded picture mismatch between the decoder side and the encoder side. This mismatch is called drift.

  To control drift due to discarding or truncating FGS or MGS data, SVC has applied the following solutions: In a particular dependent unit, the base representation (by decoding only CGS pictures with “quality_id” equal to 0 and all lower layer data dependent on them) is stored in the decoded picture buffer. When encoding the next dependency unit with the same “dependency_id” value, all NAL units, including FGS-NAL or MGS-NAL units, use the base representation for the inter prediction reference. As a result, all drift due to discard or truncation of the FGS / MGS-NAL unit in the previous access unit is stopped at this access unit. For other dependent units with the same “dependency_id” value, all NAL units use this decoded picture for inter prediction reference for high coding efficiency.

  Each NAL unit includes a syntax element “use_ref_base_pic_flag” in each NAL unit header. When the value of this element is equal to 1, NAL unit decoding uses the base representation of the reference picture during inter prediction processing. The syntax element “store_ref_base_pic_flag” specifies whether to store the base representation of the current picture for inter prediction with respect to the subsequent picture (when the value is 1) or not (when the value is 0).

NAL units with "quality_id" greater than 0 do not include syntax elements for reference picture list creation and weighted prediction. That is, the syntax element “num_ref_active_lx_minus1” (x is 0 or 1), the reference picture list rearrangement syntax table, and the weighted prediction syntax table do not exist. As a result, the MGS or FGS layer must inherit these syntax elements from NAL units whose “quality_id” in the same dependent unit is equal to 0 as needed.

  In SVC, the reference picture list consists of either only the base representation (when "use_ref_base_pic_flag" is 1) or only the decoded picture not marked as "base representation" (when "use_ref_base_pic_flag" is 0), and at the same time It is not composed of both.

  As indicated earlier, MVC is an extension of H.264 / AVC. Many of the definitions and concepts of H.264 / AVC, syntax structure, meaning, and decoding process are applied to MVC as they are or with specific generalizations and restrictions. The definition and concept of MVC, syntax structure, meaning, and part of the decoding process are explained below.

  An MVC access unit is defined as a set of NAL units that are contiguous in decoding order and includes a single primary encoded picture consisting of one or more view components. The access unit may include other NAL units including one or more redundant encoded pictures, auxiliary encoded pictures, slices of encoded pictures or slice data division in addition to primary encoded pictures. If the decoding of the access unit does not result in decoding errors, bitstream errors, or other errors that may affect decoding, a decoded picture consisting of one or more decoded view components is obtained. In other words, the MVC access unit includes view components of multiple views for one output time instance.

  The view component of MVC is also called a coded representation of a view in a single access unit.

  In MVC, inter-view prediction may be used, which refers to view component prediction from decoded samples of different view components in the same access unit. In MVC, inter-view prediction is realized in the same manner as inter prediction. For example, inter-view reference pictures are placed in the same (one or more) reference picture list as inter prediction reference pictures, and not only motion vectors but also reference indices are encoded in the same way between views and reference pictures. Or estimated.

  An anchor picture is an encoded picture, and all slices in the anchor picture can refer only to slices in the same access unit. That is, inter-view prediction can be used, but inter prediction is not used, and all coded pictures that follow in output order do not use inter prediction from any picture before the coded picture in decoding order. Inter-view prediction may be used for IDR view components that are part of a non-base view. The base view of MVC is a view having the maximum value of the view order index in the encoded video sequence. Base views are decoded independently of other views and do not use inter-view prediction. The base view can be decoded by an H.264 / AVC decoder that supports only a single view profile such as an H.264 / AVC baseline profile or a high profile.

  In the MVC standard, many of the sub-processes of the MVC decoding process include the terms “picture”, “frame”, and “field” in the specifications of the sub-process of the H.264 / AVC standard as “view component”, “frame By substituting “view component” and “field view component”, each sub-process of the H.264 / AVC standard can be used. Similarly, in the following, the terms “picture”, “frame”, and “field” are frequently used to mean “view component”, “frame / view component”, and “field / view component”, respectively.

  In scalable multi-view coding, the same bitstream may include multiple views of encoded view components, and at least some of the encoded view components may be encoded using quality and / or spatial scalability.

  Texture view refers to a view showing normal video content. This is taken with an ordinary camera, for example, and is suitable for rendering on a normal display. A texture view typically includes a picture with three components, one luma component and two chromas. In the following, a texture picture usually includes all of its component pictures or color components, unless indicated by the phrase luminance texture picture and chrominance texture picture.

  Depth-enhanced video refers to a texture video with one or more views associated with a depth video with one or more depth views. Various approaches to depth-enhanced video may be used, including video plus depth (V + D), multiview video plus depth (MVD), layered depth video (LDV) ) Use. In video + depth (V + D) representation, a single texture view and associated depth view are each represented as a sequence of texture pictures and depth pictures. MVD includes multiple texture views and respective depth views. In the LDV representation, the texture and depth of the central view are represented as usual, but the texture and depth of the other views are partially represented, covering only unobstructed areas for accurate view synthesis of the intermediate view. To do.

  The depth extension video may be encoded in a manner in which texture and depth are encoded independently of each other. For example, the texture view may be encoded as an MVC bitstream and the depth view may be encoded as another MVC bitstream. Alternatively, the depth extension video may be encoded using a method in which texture and depth are encoded in an integrated manner. When joint coding of texture and depth view is applied to the depth-enhanced video representation, some of the decoded samples of the texture picture or some of the data elements for decoding the texture picture are either part of the decoded samples of the depth picture or depth Predicted or derived from some of the data elements obtained in the picture decoding process. Alternatively, or in addition, a part of the decoded sample of the depth picture or a part of the data element for decoding the depth picture may be a part of the decoded sample of the texture picture or a part of the data element obtained by the decoding process of the texture picture. Predicted or derived from

  Solutions for multi-view 3D video (3DV) applications have only a limited number of input views, eg mono or stereo views and additional data, and render (ie, synthesize) all required views locally at the decoder It is understood that. From several available view rendering technologies, depth image-based rendering (DIBR) is seen as a competitive alternative.

  Fig. 5 shows a simplified model of a DIBR-based 3DV system. The input of the 3D video codec includes the corresponding depth information along with the stereoscopic video and the stereoscopic baseline b0. The 3D video codec synthesizes multiple virtual views between two input views, along with a baseline (bi <b0). The DIBR algorithm can extrapolate not only between two input views, but also the outer views. Similarly, the DIBR algorithm can synthesize a view from a single texture view and a corresponding depth view. However, texture data should be available at the decoder side with corresponding depth data to enable DIBR based multi-view rendering.

  In such a 3DV system, depth information for each video frame is created on the encoder side in the form of a depth picture (called a depth map). A depth map is an image with depth information for each pixel. Each sample in the depth map represents the distance from the surface where the camera is placed to the respective texture sample. In other words, if the z-axis is along the shooting direction of the camera (and thus orthogonal to the plane on which the camera is located), the depth map sample represents the value of the z-axis.

  Depth information can be acquired by various means. For example, the depth of the 3D scene may be calculated from the parallax recorded by the shooting camera. The depth estimation algorithm takes a stereoscopic view as input and calculates the local parallax between the two offset images for that view. Each image is processed pixel by pixel in overlapping blocks, and a matching block in the offset image for each pixel block is searched locally in the horizontal direction. When the parallax in the pixel direction is calculated, the corresponding depth value z is calculated by equation (1).

z = (f · b) / (d + Δd) ... Formula (1)

  Here, f is the focal length of the camera, b is the baseline distance between the cameras, and is shown in FIG. Furthermore, d represents the parallax observed between the two cameras, and the camera offset Δd represents the horizontal position shift that can occur with respect to the optical center of the two cameras. However, since the algorithm is based on block matching, the quality of disparity estimation over depth depends on the content and is not accurate in most cases. For example, it is impossible to perform depth estimation directly on an image portion having no texture and including a very smooth region or a high noise level.

  A disparity / disparity map such as a parallax map defined in ISO / IEC international standard 23002-3 may be processed in the same manner as a depth map. There is a direct correspondence between depth and parallax, and the other can be calculated from one through a mathematical equation.

The encoding and decoding order for texture and depth view components within an access unit is typically such that the data for the encoded view component is not interleaved by other encoded view components, and the data for the access unit is also bitstream or decoding order. Are not interleaved by other access units. For example, as shown in FIG. 7, two sets of texture / depth views (T0 _t , T1 _t , T0 _{t + 1} , T1 _{t + 1} , T0) are assigned to different access units (t, t + 1, t + 2). _{t + 2} , T1 _{t + 2} , D0 _t , D1 _t , D0 _{t + 1} , D1 _{t + 1} , D0 _{t + 2} , D1 _{t + 2} ). Here, the access unit t consisting of the texture / depth view components (T0 _t , T1 _t , D0 _t , D1 _t ) has the texture / depth view components (T0 _{t + 1} , T1 _{t + 1} , D0 _{t + 1} , D1 _{t + 1} ) is ahead of the access unit t + 1.

  The encoding and decoding order of the view components in the access unit may follow the encoding format and may be determined by the encoder. The texture view component may be encoded before the associated depth view component of the same view. Therefore, such depth view components may be predicted from related texture view components of the same view. Such a texture view component may be encoded by, for example, an MVC encoder and decoded by an MVC decoder. An extended texture view component represents herein a texture view component that is encoded after an associated depth view component of the same view. Thus, it may be predicted from the associated depth view component. Texture and depth view components of the same access unit are typically encoded in a view dependent order. Texture and depth view components can be reordered in any order with respect to one another as long as the above constraints are followed.

  The texture view and depth view may be encoded into a single bitstream in which part of the texture view conforms to one or more video standards such as H.264 / AVC and / or MVC. In other words, the decoder may be able to decode some of the texture views of such bitstreams and exclude the remaining texture and depth views.

  In this context, an encoder that encodes one or more texture and depth views into a single H.264 / AVC and / or MVC compliant bitstream is also referred to as a 3DV-ATM encoder. The bitstream generated by such an encoder can be referred to as a 3DV-ATM bitstream. The 3DV-ATM bitstream may include some texture and depth views that cannot be decoded by the H.264 / AVC and / or MVC decoder. A decoder that can decode all views from a 3DV-ATM bitstream can also be called a 3DV-ATM decoder.

  A 3DV-ATM bitstream can contain a selected number of AVC / MVC compliant texture views. A depth view for an AVC / MVC compliant texture view may be predicted from the texture view. The remaining texture views may use enhanced texture coding and the depth view may use depth coding.

  A high level flowchart of an embodiment of an encoder 200 capable of encoding texture and depth views is shown in FIG. 8, and a decoder 210 capable of decoding texture and depth views is shown in FIG. In these figures, a solid line represents a general data flow, and a broken line represents a control information signal. The encoder 200 may receive a texture component 201 encoded with the texture encoder 202 and a depth map component 203 encoded with the depth encoder 204. While the encoder 200 is encoding the texture component according to AVC / MVC, the first switch 205 may be turned off. While the encoder 200 is encoding the enhanced texture component, the first switch 205 may be turned on so that the information generated by the depth encoder 204 is provided to the texture encoder 202. The encoder of this embodiment also includes a second switch 206 that is controlled as follows. The second switch 206 is turned on while the encoder is encoding the depth information of the AVC / MVC view, and is turned off while the encoder is encoding the depth information of the extended texture view. The encoder 200 may output a bitstream 207 that includes encoded video information.

  The decoder 210 may operate similarly except that at least a portion is in reverse order. The decoder 210 may receive a bitstream 207 that includes encoded video information. The decoder 210 includes a texture decoder 211 that decodes texture information and a depth decoder 212 that decodes depth information. A third switch 213 may be provided to control information distribution from the depth decoder 212 to the texture decoder 211, and a fourth switch 214 may control information distribution from the texture decoder 211 to the depth decoder 212. May be provided. When the decoder 210 decodes the AVC / MVC texture view, the third switch 213 may be switched off, and when the decoder 210 decodes the extended texture view, the third switch 213 is switched on. May be. When the decoder 210 decodes the depth of the AVC / MVC texture view, the fourth switch 214 may be turned on, and when the decoder 210 decodes the depth of the extended texture view, the fourth switch 214 It may be switched off. The decoder 210 may output a reconstructed texture component 215 and a reconstructed depth map component 216.

Many video encoders utilize a Lagrangian cost function to search for motion vectors associated with a rate-distortion optimal coding mode, eg, the desired macroblock mode. This type of cost function fixes together the exact or estimated image distortion due to lossy coding and the exact or estimated amount of information needed to represent the pixel / sample values of the image area. To do this, a weighting factor or λ is used. The Lagrangian cost function can be expressed as:

C = D + λR

  Where C is the Lagrangian cost to be minimized and D is the image distortion due to this mode and the currently considered motion vector (eg the mean square error of the pixel / sample values between the original image block and the encoded image block) , Λ is a Lagrangian coefficient, and R is the number of bits required to represent the data (including the amount of data for representing candidate motion vectors) required to reconstruct the image block at the decoder.

  FIG. 1 shows a block diagram of a video encoding system according to an exemplary embodiment. This block diagram is shown as a block diagram outlining an exemplary apparatus or exemplary electronic device 50 that incorporates a codec in accordance with an embodiment of the present invention. FIG. 2 shows a layout of the device according to an exemplary embodiment. Each element of FIGS. 1 and 2 is described below.

  The electronic device 50 may be a user equipment or the like in a mobile terminal or a wireless communication system. However, it should be understood that embodiments of the present invention can be implemented in any electronic device or apparatus that requires encoding and decoding, or encoding and decoding of a video image.

  The apparatus 50 may include a housing 30 that incorporates and protects the device. The device 50 may also comprise a display 32 in the form of a liquid crystal display. In other embodiments of the invention, the display may be by any suitable display technology suitable for displaying images and videos. The device 50 may also include a keypad 34. In other embodiments of the present invention, any suitable data interface or user interface mechanism may be used. For example, the user interface may be implemented as a virtual keyboard or a data input system on a part of the touch-sensitive display. The device may include a microphone 36 and any suitable audio input of a digital or analog signal. The device 50 may also comprise an audio output device, and in the embodiments of the present invention may be any one of the following: earphone 38, speaker, analog audio or digital audio output connection. The device 50 may also include a battery 40 (or in other embodiments of the invention may be powered by any suitable portable energy device such as a solar cell, fuel cell, clockwork generator, etc. ). The apparatus may also include an infrared port 42 for short visible range communication with other devices. In other embodiments, the device 50 may further comprise any suitable short-range communication solution such as Bluetooth wireless communication or USB / FireWire wired connection.

  The device 50 may include a controller 56 or processor that controls the device 50. Controller 56 may be connected to memory 58. In embodiments of the present invention, the memory may store both data in the form of images and audio data and / or may store instructions implemented in the controller 56. The controller 56 may be connected to the codec circuit 54. The codec circuit is suitable for performing the encoding / decoding of audio and / or video data and assisting the encoding / decoding performed by the controller 56.

  The device 50 may also comprise a card reader 48 and a smart card 46. For example, a UICC and UICC reader suitable for providing user information and providing authentication information for user authentication and authorization in the network may be provided.

  The device 50 may comprise a radio interface circuit 52 connected to the controller and suitable for generating a radio communication signal. Wireless communication is, for example, communication in a mobile communication network, a wireless communication system, or a wireless local area network. The device 50 may also include an antenna 44 connected to the wireless interface circuit 52. The antenna transmits the radio signal generated by the radio interface circuit 52 to the other device (s) and receives the radio signal from the other device (s).

  In some embodiments of the present invention, the device 50 comprises a camera capable of recording or detecting individual frames that are passed to the processing codec 54 or controller. In some embodiments of the present invention, an apparatus may receive processing video image data from another device prior to transmission and / or storage. In some embodiments of the present invention, device 50 may receive the encoding / decoding image either wirelessly or wired.

  FIG. 3 shows a video encoding configuration including multiple devices, networks and network elements according to an exemplary embodiment. FIG. 3 shows an example of a system that can be used in an embodiment of the present invention. The system 10 includes a plurality of communication devices that can communicate over one or more networks. The system 10 may include any combination of wireless or wired networks, such as a wireless cellular network (GSM (registered trademark), UMTS, CDMA network, etc.) or a wireless local area defined by any of the IEEE 802.x standards. A network (WLAN), a Bluetooth personal area network, an Ethernet (registered trademark) local area network, a token ring local area network, a wide area network, and the Internet may be included. However, it is not limited to these.

  The system 10 may include both wireless and wired communication devices and may include an apparatus 50 suitable for implementing embodiments of the present invention. For example, the system shown in FIG. 3 shows expressions representing the cellular phone network 11 and the Internet 28. Connections to the Internet 28 may include, but are not limited to, long-range wireless connections, short-range wireless connections, and various wired connections. Wired connections include, but are not limited to, telephone lines, cable lines, power lines, and other similar communication lines.

  Exemplary communication devices shown in system 10 are electronic devices and devices 50, personal digital assistants (PDAs) 16, a combination of PDA and mobile phone 14, integrated messaging device (IMD) 18, desktop computer 20, notebook type A computer 22 may be included. However, it is not limited to these. The device 50 may be a fixed type or a portable type that can be carried by a moving person. Further, the device 50 may be disposed on the moving means. Such transportation means include, but are not limited to, cars, trucks, taxis, buses, trains, ships / boats, airplanes, bicycles, motorcycles, and other similar transportation means.

  Further, some devices may send and receive telephone calls and messages, or communicate with a service provider through a wireless connection 25 with the base station 24. The base station 24 may be connected to a network server 26 that enables communication between the mobile phone network 11 and the Internet 28. The system may include additional communication devices and various types of communication devices.

  Communication devices may communicate using various transmission technologies, including code division multiple access (CDMA), global systems for mobile communications (GSM (registered trademark)), universal mobile phone systems (UMTS), Division multiple access (TDMA), frequency division multiple access (FDMA), TCP-IP (transmission control protocol-internet protocol), short message service (SMS), multimedia message service (MMS), e-mail, IMS (instant messaging service) ), Bluetooth, IEEE 802.11, and other similar wireless communication technologies. However, it is not limited to these. Communication devices included in implementations of the various embodiments of the invention can communicate via various media. Such media include, but are not limited to, wireless, infrared, laser, cable connections, and other suitable connections.

  Figures 4a and 4b show block diagrams of video encoding and decoding according to an exemplary embodiment.

  FIG. 4 a shows an encoder comprising a pixel predictor 302, a prediction error encoder 303 and a prediction error decoder 304. FIG. 4 a also illustrates an embodiment of a pixel predictor 302 that includes an inter predictor 306 and an intra predictor 308, a mode selector 310, a filter 316, and a reference frame memory 318. In this embodiment, the mode selector 310 includes a block processor 381 and a cost evaluator 382. The encoder may also include an entropy encoder 330 that performs entropy encoding of the bitstream.

  FIG. 4 b shows an embodiment of the inter predictor 306. The inter predictor 306 includes a reference frame selector 360 that selects one or a plurality of reference frames, a motion vector definer 361, a prediction list generator 363, and a motion vector selector 364. These components or parts thereof may be part of the prediction processor 362 and may be implemented by other means.

  Pixel predictor 302 receives an image 300 that is encoded by both inter predictor 306 and intra predictor 308 (inter predictor 306 determines the difference between this image and motion compensated reference frame 318; The intra predictor 308 determines the prediction of the image block based only on the processed portion of the current frame or picture). The output from both the inter predictor and the intra predictor is sent to the mode selector 310. Both inter predictor 306 and intra predictor 308 may have multiple intra prediction modes. Therefore, inter prediction and intra prediction are performed in each mode,

  A prediction signal may be provided to the mode selector 310. Mode selector 310 also receives a copy of image 300.

  The mode selector 310 determines the type of encoding mode used for encoding the current block. When the mode selector 310 decides to use the inter prediction mode, the mode selector 310 sends the output of the inter predictor 306 to the output of the mode selector 310. When the mode selector 310 decides to use the intra prediction mode, it sends an output for one of the intra prediction modes to the output of the mode selector 310.

  The mode selector 310 may use, for example, a Lagrangian cost function in the cost evaluator block 382 to select an encoding mode and its parameter values. Here, the parameter value is a motion vector based on a normal block, a reference index, an intra prediction direction, or the like. This type of cost function is the amount of information (exact or estimated) required to represent the image distortion (accurate or estimated) image loss and the pixel / sample values of the image area by the lossy coding method. Is fixed together, using a weighting factor λ, expressed as: C = D + λ × R. Where C is the Lagrangian cost to be minimized, D is the image distortion due to this mode and its parameters (mean square error, etc.), R is the data required to reconstruct the image block at the decoder (candidate motion vectors) The number of bits required to represent (which may include the amount of data to represent).

  The output of the mode selector is sent to the first adder 321. The first adder may subtract the output of the pixel predictor 302 from the image 300 to generate a first prediction error signal 320 that is an input to the prediction error encoder 303.

  The pixel predictor 302 further receives from the temporary reconstructor 339 a combination of the predicted representation of the image block 312 and the output 338 of the prediction error decoder 304. The temporarily reconstructed image 314 may be sent to an intra predictor 308 and a filter 316. A filter 316 that receives the temporary expression filters the temporary expression and outputs a final reconstructed image 340 stored in the reference frame memory 318. The reference frame memory 318 may be connected to the inter predictor 306 so that the subsequent image 300 is used as a reference image to be compared in the inter prediction operation. In many embodiments, the reference frame memory 318 can store multiple decoded pictures. One or more of such decoded pictures may be used in the inter predictor 306 as a reference picture for subsequent images 300 to be compared in an inter prediction operation. In some cases, the reference frame memory 318 is also referred to as a decoded picture buffer.

  The operation of the pixel predictor 302 may be configured to perform any pixel prediction algorithm known in the art.

  Pixel predictor 302 may also include a filter 385 that filters the predicted values before outputting them from pixel predictor 302.

  The operations of the prediction error encoder 303 and the prediction error decoder 304 will be described in detail later. In the next embodiment, the encoder generates an image in units of 16 × 16 pixel macroblocks. Such an image will form a full image or picture. However, it should be noted that FIG. 4a is not limited to a 16 × 16 block size, and blocks of any size and shape can generally be used. Similarly, it should be noted that FIG. 4a is not limited to macroblock division of a picture, and may be divided into blocks that can be used as coding units by any other picture division. Thus, for the following example, the pixel predictor 302 outputs a 16 × 16 pixel sized prediction macroblock sequence, and the first adder 321 includes the first macroblock and the prediction macroblock (pixel prediction) of the image 300. A 16 × 16 pixel residual data macroblock sequence representing the difference between the output and the output of the output of the unit 302.

  The prediction error encoder 303 includes a transform block 342 and a quantizer 344. Transform block 342 transforms first prediction error signal 320 into a transform domain. This conversion is, for example, a DCT conversion or its variant. The quantizer 344 quantizes a transform domain signal such as a DCT coefficient to obtain a quantization coefficient.

  A prediction error decoder 304 receives the output from the prediction error encoder 303 and generates a decoded prediction error signal 338. The decoded prediction error signal is combined with the prediction representation of the image block 312 by the second adder 339 to generate a temporary reconstructed image 314. The prediction error decoder includes an inverse quantizer 346 that dequantizes a quantized coefficient value such as a DCT coefficient and a reconstructed converted signal in order to approximately reconstruct the converted signal. An inverse transform block 348 that performs an inverse transform on the image may be provided. The output of inverse transform block 348 includes the reconstruction block (s). The prediction error decoder may also comprise a macroblock filter (not shown) that can further filter the reconstructed macroblock according to the decoded information and filter parameters.

  Next, the operation of an exemplary embodiment of the inter predictor 306 will be described in detail. Inter predictor 306 receives the current block for inter prediction. Here, it is assumed that one or more encoded neighboring blocks already exist for the current block, and the motion vector related to it is also defined. For example, the left block and / or the upper block of the current block may be such a block. The prediction of the spatial motion vector for the current block can be performed using, for example, motion vectors of encoded adjacent blocks and / or non-adjacent blocks of the same slice or frame. Alternatively, the prediction can be performed using a linear or nonlinear function of spatial motion vector prediction, combining various spatial motion vector predictors in linear or nonlinear motion, or any suitable means that does not use temporal reference information. It may be broken. It is also possible to configure a motion vector predictor by combining both spatial prediction and temporal prediction information of one or more encoded blocks. This type of motion vector predictor is also called a spatio-temporal motion vector predictor.

  A reference frame used for encoding may be stored in a reference frame memory. Each reference frame may be included in one or more reference picture lists. In the reference picture list, each entry has a reference index for identifying a reference frame. If the reference frame is no longer used as a reference, it may be removed from the reference frame memory and marked as “not used for reference”, or the storage location of the reference frame is occupied by a new reference frame and is not a reference frame. It may be.

  Real-time transfer protocol (RTP) is widely used for real-time transfer of timed media such as voice and video. In RTP transfer, media data is encapsulated in a plurality of RTP packets. The RTP payload format may be defined for transmission of a coded bitstream of a specific format by RTP. For example, a draft of the SVC RTP payload format is specified in RFC 6190 by the Internet Engineering Task Force (IETF). In the SVC RTP payload format, a type of NAL unit called payload content scalability information (PACSI) NAL unit is defined. The PACSI-NAL unit, if present, is the first NAL unit in an aggregate packet that includes multiple NAL units and is not present in other types of packets. The PACSI-NAL unit exhibits scalability characteristics that are common to all remaining NAL units in the payload. This makes it easier for media aware network elements (MANEs) to decide whether to forward / process / discard aggregated packets. The sender may create a PACSI-NAL unit. The receiving side may ignore the PACSI-NAL unit and use it as a clue to efficiently process the aggregated packet. If the first aggregation unit of an aggregation packet includes a PACSI-NAL unit, there is at least one additional aggregation unit in the same packet. The RTP header field is set according to the remaining NAL units of the aggregate packet.

  As described above, an access unit may include other component types (eg, main texture component, redundant texture component, auxiliary component, depth / disparity component), another view, and another scalable layer slice. A syntax element common to slices, for example, a conventional syntax element included in a slice header, may have the same value for different slices in the same access unit. Nevertheless, such syntax elements are encoded at each slice according to the prior art. Next, various embodiments for reducing the bit rate and bit count used for coding syntax elements having the same value in multiple slices within an access unit are shown.

  In many such embodiments, at least one subset of the syntax elements conventionally included in the slice header is included in the GOS (slice group) parameter set by the encoder. The encoder may encode the GOS parameter set as a NAL unit. The NAL unit of the GOS parameter set may be included in the bitstream together with the coded slice NAL unit or the like, but may be transmitted out of band as in the case of the other parameter sets described above.

  In many embodiments, the GOS parameter set syntax structure includes an identifier and may be used, for example, in referencing a particular GOS parameter set instance from a slice header or another GOS parameter set. In some embodiments, the GOS parameter set syntax structure does not include an identifier, and both the encoder and decoder estimate the identifier using, for example, a bitstream order and a default numbering scheme for the GOS parameter set syntax structure. May be.

  Also, in some embodiments, the encoder and decoder infer the contents and instances of the GOS parameter set from other syntax structures that have been encoded or decoded or already in the bitstream. For example, the GOS parameter set may be implicitly created from the slice header in the texture view of the base view. The encoder and decoder may estimate an identification value for such an estimated GOS parameter set. For example, it may be estimated that the GOS parameter set created from the slice header in the texture view of the base view has an identification value equal to 0.

  In some embodiments, the GOS parameter set is valid within the specific access unit associated with it. For example, the syntax structure of a GOS parameter set is included in the NAL unit sequence for a particular access unit, which sequence is in decoding order or bitstream order, and the GOS parameter set is valid from its appearance position to the end of the access unit. May be. In some embodiments, the GOS parameter set is valid for various access units.

  The encoder may encode different GOS parameter sets for one access unit. If at least one subset of the values of syntax elements encoded in the slice header is known to be identical to or predicted / estimated in subsequent slice headers, the encoder shall encode the GOS parameter set You may decide.

  A limited numbering space is used for GOS parameter set identifiers. For example, a fixed length code may be used, or may be determined as an unsigned integer value within a specific range. The encoder may use a specific GOS parameter set identification value for the initial GOS parameter set. Next, if the first GOS parameter set is not referred to by any slice header or GOS parameter set, for example, the same GOS parameter set identification value may be used for the second GOS parameter set. The encoder may repeat the syntax structure of the GOS parameter set in the bitstream, for example to obtain high robustness against transmission errors.

In many embodiments, the syntax structures that can be included in a GOS parameter set are conceptually organized into a set of syntax elements. A GOS parameter set syntax element set may be formed, for example, based on one or more of the following principles:
A syntax element indicating a scalable layer and / or other scalable characteristics;
A syntax element indicating a view and / or other multi-view characteristics;
Syntax elements associated with specific component types such as depth / disparity;
Syntax elements associated with access unit identification and decoding order and / or output order and / or other syntax elements that are invariant to all slices of the access unit;
A syntax element that is invariant across all slices of the view component;
Syntax elements related to reference picture list changes;
Syntax elements associated with the set of reference pictures used;
Syntax elements associated with decoding reference picture markings;
Syntax elements associated with the prediction weight table for weighted prediction;
A syntax element that controls deblocking filtering;
A syntax element that controls adaptive loop filtering;
A syntax element that controls the sample adaptive offset;
Any combination of the above set.

For each syntax element set, the encoder may have one or more of the following options when encoding the GOS parameter set:
The syntax element set may be encoded into a GOS parameter set syntax structure. That is, the value of the encoded syntax element of the syntax element set may be included in the syntax structure of the GOS parameter set;
The syntax element set may be included in the GOS parameter set by reference. This reference may be given as an identifier to another GOS parameter set. The encoder may use a separate reference GOS parameter set for each syntax element set;

  It may be indicated that the syntax element set does not exist in the GOS parameter set, and may be estimated.

  Options that can be selected for a particular syntax element set when the encoder encodes the GOS parameter set may depend on the type of syntax element set. For example, the syntax element set associated with the scalable layer may always be present in the GOS parameter set. On the other hand, a set of syntax elements that is invariant across all slices of the view component is not available to be included by reference and may optionally be present in the GOS parameter set. In addition, syntax elements associated with reference picture list changes may be included by reference, included directly as is, or may not be present in the GOS parameter set syntax structure. The encoder may encode a sign that is in a bitstream, such as a GOS parameter set syntax structure, indicating the type of option used for encoding. The encoding table and / or entropy encoding may depend on the type of syntax element. The decoder may use a coding table and / or entropy decoding located in the coding table and / or entropy coding used in the encoder based on the type of syntax element being decoded.

  The encoder may comprise a plurality of means for indicating an association between the syntax element set and the GOS parameter set originally used for the value of the syntax element set. For example, the encoder may encode a loop of syntax elements. Each entry in such a loop indicates the identification value of the GOS parameter set used as a reference and is encoded as a syntax element that identifies the syntax element set that is copied from the reference GOP parameter set. In another embodiment, the encoder may encode a plurality of syntax elements, each of which indicates a GOS parameter set. The last GOS parameter set in the loop that contains a particular syntax element set is a reference to the syntax element set in the GOS parameter set when the encoder is currently encoding into the bitstream. The decoder analyzes the encoded GOS parameter set from the bitstream and reproduces the same GOS parameter set as the encoder.

  In the exemplary embodiment, the syntax structure, the meaning of syntax elements, and the decoding process may be defined as follows. Syntax elements in the bitstream are shown in bold font. Each syntax element is described by its name (all underscores with an underscore character), 1 or 2 syntax categories may be used, and 1 or 2 descriptors may be used as encoding representations. is there. The decoding process is performed according to the value of the syntax element and the value of the syntax element that has been decoded previously. The value of a syntax element is represented in the usual (non-bold) format when used in a syntax table or text. In some cases, the syntax table may use values of other variables derived from the syntax element values. These variables are represented in the syntax table or text using lowercase and uppercase letters without underscore characters. Variables starting with a capital letter are generated for decoding the current syntax structure and all subordinate syntax structures. A variable that begins with an uppercase letter may be used in the decoding process for a later syntax structure without showing the original syntax structure of the variable. Variables that start with a lowercase letter are also used in the context in which the variable was created. In some cases, a “mnemonic” name that can be converted to a numeric value of a syntax element value or variable value is also used. The “mnemonic” name may be used independently of the numerical value. The relationship between numbers and names is specified in the text. The name consists of one or more strings separated by underscore characters. Each string starts with a capital letter and may contain a capital letter in the middle.

  In the exemplary embodiment, common notations such as arithmetic operators, logical operators, relational operators, binary operators, assignment operators, and range notations as defined in H.264 / AVC, HEVC draft, etc. are used. May be. Further, a common mathematical function as defined in H.264 / AVC, HEVC draft, etc. may be used. Common rules regarding the priority and execution order of operations may be used as defined in H.264 / AVC, HEVC draft, and the like.

In the exemplary embodiment, the following descriptors are used to define the parsing process for each syntax element.
B (8): Byte (8 bits) with a bit string of arbitrary pattern.
Se (v): Signed integer type Exp-Golomb coding syntax element starting from the left bit.
U (n): n-bit unsigned integer. When n is “v” in the syntax table, the number of bits changes depending on the values of other syntax elements. The parsing process for this descriptor is defined by the next n bits from the bitstream interpreted as a binary representation of the unsigned integer with the most significant bit described first.
Ue (v): Unsigned integer type Exp-Golomb coding syntax element starting from the left bit.

The Exp-Golomb bit string may be converted into a code number (codeNum) using, for example, the following table.

The code number corresponding to the Exp-Golomb bit string may be converted into se (v) using the following table, for example.

  In an exemplary embodiment, the syntax structure may be defined as follows: A series of sentences in parentheses is a compound sentence and is functionally treated as a single sentence. The “while” syntax specifies whether or not a condition is true. If the condition is true, a single sentence (or compound sentence) is repeatedly specified until the condition is not true. The "do ... while" syntax once specifies the evaluation of a sentence, then continues to determine whether the condition is true, and if the condition is true, repeatedly specifies the evaluation of the sentence until the condition is not true To do. The "if ... else" syntax specifies whether the condition is true, specifies the evaluation of the first sentence if the condition is true, and specifies the evaluation of an alternative sentence otherwise. The alternative text associated with the "else" clause of this syntax can be omitted if alternative text evaluation is not required. The “for” syntax specifies the evaluation of the initial value sentence, the condition judgment continues, and if the condition is true, the evaluation of the first sentence and the subsequent sentence is repeatedly specified until the condition is not true.

  Next, an exemplary embodiment of 3DV-ATM will be described.

  A 3DV-NAL unit is defined for the GOS parameter set and the coding slice and coding depth slice of the extended texture view. The NAL unit header length of a 3DV-NAL unit (for example, using NAL unit type 21) is 2 bytes. The second byte includes temporal_id, 3dv_nal_unit_type, and gos_param_id. 3dv_nal_unit_type specifies whether a NAL unit contains a GOS parameter set or a 3DV slice. When the NAL unit includes a GOS parameter set, gos_param_id gives an identification value of the GOS parameter set defined in the NAL unit. When the NAL unit includes a slice, gos_param_id refers to the GOS parameter set including the value of the syntax element of the slice header of the slice. The syntax elements previously included in the MVC-NAL unit header extension are present in the GOS parameter set.

Below, the nal_unit syntax is italicized and appended to this section. Assume that NAL unit type 21 is reserved for 3DV-NAL units.

The syntax of nal_unit_header_3dv_extension () may be specified as follows.

  The meaning of nal_unit_header_3dv_extension () may be specified as follows.

  When 3dv_nal_unit_type (bold in the original text and a bitstream syntax element) is 0, it specifies that the NAL unit includes the GOS parameter set. When 3dv_nal_unit_type is 1, it specifies that the NAL unit includes a coded slice 3DV extension.

  temporal_id (bold in the text and is a bitstream syntax element) identifies the temporal identifier of the NAL unit. If nal_unit_type is 1 or 5, and there is no NAL unit immediately before the NAL unit with nal_unit_type of 14, temporal_id is estimated to be equal to the temporal_id value of the non-base view of the same access unit. The value of temporal_id may be the same for all of the prefix encoded slice MVC extensions and 3DV-NAL units in the access unit. When the access unit includes a NAL unit whose nal_unit_type is 5 or non_idr_flag is 0, temporal_id may be 0. The assignment of a value to temporal_id may be restricted by substream extraction processing.

gos_param_id (bold in the original text and a bitstream syntax element) identifies the identifier of the GOS parameter set. When 3dv_nal_unit_type is 0, gos_param_id specifies the identifier of the GOS parameter set used for encoding slice 3DV extension included in the NAL unit. When 3dv_nal_unit_type is 1, gos_param_id specifies the identifier of the GOS parameter set specified by the NAL unit. When 3dv_nal_unit_type is 1, gos_param_id may be in a range including 0-15. When 3dv_nal_unit_type is 0, gos_param_id may be in a range including 1 to 15.

The syntax of 3dv_nal_unit (), that is, the NAL unit type 21 is defined as follows.

The syntax of gos_param_set (), that is, the syntax structure of the GOS parameter set is specified as follows.

  The meaning of gos_param_set () is specified as follows.

  When slice_param_for_au_flag (bold in the original text is a bitstream syntax element) is 0, the value of the syntax element included in the slice_param_for_au () syntax structure is the GOS parameter set or slice header syntax in the same access unit. Identifies that it is the same as one in the tax structure. When slice_param_for_au_flag is 1, it specifies that the slice_param_for_au () syntax structure exists in this GOS parameter set.

When gos_parameter_inheritance_flag is 0, ref_pic_list_modification_flag [i], pred_weight_flag [i], and dec_ref_pic_marking_flag [i] are specified as follows:
-When ref_pic_list_modification_flag [i] ("ref_pic_list_modification_flag" is bold in the original text and is a bitstream syntax element) is 1, the ref_pic_list_3dv_modification () syntax structure is effectively present in the GOS parameter set identified by gos_param_id Identify what to do. The meaning when ref_pic_list_modification_flag [i] is 0 is not specified.
-When pred_weight_flag [i] ("pred_weight_flag" is bold in the original text and is a bitstream syntax element) is 1, the pred_weight_table () syntax structure is effectively present in the GOS parameter set identified by gos_param_id Identify what to do. The meaning when pred_weight_flag [i] is 0 is not specified.
-When dec_ref_pic_marking_flag [i] ("dec_ref_pic_marking_flag" is bold in the original text and is a bitstream syntax element) is 1, the dec_ref_pic_marking () syntax structure is effectively present in the GOS parameter set identified by gos_param_id Identify what to do. The meaning when dec_ref_pic_marking_flag [i] is 0 is not specified.

When gos_parameter_inheritance_flag is 1, ref_gos_param_id [i], ref_pic_list_modification_flag [i], pred_weight_flag [i], dec_ref_pic_marking_flag [i] are specified as follows:
Ref_gos_param_id [i] ("ref_gos_param_id" is bold in the original text and is a bitstream syntax element) is a GOS parameter set and is valid for the GOS parameter set identified by gos_param_id [ref_pic_list_modification_flag [i ], Pred_weight_flag [i], and dec_ref_pic_marking_flag [i] identify the GOS parameter set used as a source for obtaining the syntax structure identified. When ref_gos_param_id [i] is 0, the slice header of the texture view component of the base view of the same access unit is valid for the GOS parameter set identified by gos_param_id, ref_pic_list_modification_flag [i], pred_weight_flag [i], and dec_ref_pic_marking_flag Identifies that it is used as a source for obtaining the syntax structure identified by [i].
-When ref_pic_list_modification_flag [i] ("ref_pic_list_modification_flag" is bold in the original text and is a bitstream syntax element) is 1, the ref_pic_list_3dv_modification () syntax structure of the GOS parameter set identified by ref_gos_param_id [i] is , Specifies that the GOS parameter set identified by gos_param_id is valid. The meaning when ref_pic_list_modification_flag [i] is 0 is not specified.
・ When pred_weight_flag [i] ("pred_weight_flag" is bold in the original text and is a bitstream syntax element) is 1, the pred_weight_table () syntax structure of the GOS parameter set identified by ref_gos_param_id [i] , Specifies that the GOS parameter set identified by gos_param_id is valid. The meaning when pred_weight_flag [i] is 0 is not specified.
-When dec_ref_pic_marking_flag [i] ("dec_ref_pic_marking_flag" is bold in the original text and is a bitstream syntax element) is 1, the dec_ref_pic_marking () syntax structure of the GOS parameter set identified by ref_gos_param_id [i] , Specifies that the GOS parameter set identified by gos_param_id is valid. The meaning when dec_ref_pic_marking_flag [i] is 0 is not specified.

The syntax of slice_param_for_3dv_view_component () is specified as follows. The syntax structure includes syntax elements whose values are invariant for all slices of the view component.

  The meaning of slice_param_for_3dv_view_component () is specified as follows.

  When the related NAL unit is a 3DV-NAL unit that refers to the GOS parameter set identified by gos_param_id, the meanings of non_idr_flag, priority_id, view_id, anchor_pic_flag, and inter_view_flag are the same as in MVC.

  When depth_flag (originally bold and bitstream syntax element) is 0, specifies that the NAL unit that references the GOS parameter set identified by gos_param_id contains a slice of the extended texture view component . When depth_flag is 1, it specifies that the NAL unit that refers to the GOS parameter set identified by gos_param_id includes a slice of the depth view component.

  When single_slice_flag (bold in the original text and a bitstream syntax element) is 0, it is specified that the view component that refers to the GOS parameter set identified by gos_param_id has a plurality of slices. When single_slice_flag is 1, it is specified that the view component that refers to the GOS parameter set identified by gos_param_id has exactly one slice.

  When initialisation_param_equal_flag (bold in the original text, which is a bitstream syntax element) is 0, it specifies that the slice_param_for_3dv syntax structure does not exist. When initialisation_param_equal_flag is 1, it specifies that a slice_param_for_3dv syntax structure exists.

The syntax of slice_param_for_3dv () is specified as follows. This syntax structure may be included in the slice_param_for_3dv_view_component () syntax structure or the slice_header_in_3dv_extension () syntax structure.

  The meaning of slice_param_for_3dv () is specified as follows.

  When slice_param_for_3dv () is included in the slice_param_for_3dv_view_component () syntax structure, the value of the syntax element is applied to all slices of the view component. When slice_param_for_3dv () is included in the slice_header_in_3dv_extension () syntax structure, the value of the syntax element is applied to slices included in the same NAL unit.

  The meaning specified in H.264 / AVC is applied to the syntax element of slice_param_for_3dv () with the following changes. slice_type (bold in the text and is a bitstream syntax element) has the additional constraint that its value is not equal to 3, 4, 8, or 9. When colour_plane_id exists, the meaning specified in H.264 / AVC is applied. When depth_flag is 0, separate_colour_plane_flag is estimated to be 1, and colour_plane_id is estimated to be 0. direct_spatial_mv_pred_flag (bold in the original text and a bitstream syntax element) has the same meaning as specified in H.264 / AVC with the following changes: When RefPicList1 [0] indicates an inter-view reference component or an inter-view only reference component, this reference component belongs to the same access unit as the current view component, and direct_spatial_mv_pred_flag is 1. num_ref_idx_l0_active_minus1 (bold in the original text and a bitstream syntax element) and num_ref_idx_l1_active_minus1 (bold in the original text and a bitstream syntax element) have the same meaning as defined in MVC. When dmvp_flag (bold in the original text and is a bitstream syntax element) is 0, it specifies that the inter-view and intra-view prediction processing defined by MVC is applied. When dmvp_flag is 1, it specifies that depth-based inter-view and intra-view prediction processing is applied. When depth_weighted_pred_flag (bold in the original text and a bitstream syntax element) is 0, it specifies that depth range based weighted prediction is not used for P or B slices of the depth view component. When depth_weighted_pred_flag is 1, it specifies that depth range based weighted prediction is used for P or B slices of the depth view component.

The syntax of slice_param_for_au () is specified as follows. The syntax structure includes syntax elements whose values are unchanged for all slices of the access unit including the 3DV-NAL unit.

  The meaning of slice_param_for_au () is defined as follows. The meanings defined in H.264 / AVC apply with the following additional restrictions. The value of each syntax element of slice_param_for_au may be unchanged in all slice headers included in the same access unit and the slice_header_for_au syntax structure.

The syntax of slice_header_in_3dv_extension () is specified as follows. The values of single_slice_flag and initialization_param_equal_flag are obtained from the GOS parameter set identified by gos_param_id.

  The meaning of slice_header_in_3dv_extension () is specified as follows. The meaning specified in H.264 / AVC applies. If there is no syntax element or structure in slice_header_in_3dv_extension (), its value is inherited from the GOS parameter set identified by gos_param_id.

  Next, an exemplary embodiment of HEVC and possible scalable extensions are described. The scalable extensions described above include, for example, medium and / or coarse grain quality scalability, spatial scalability, extended spatial scalability, multi-view coding, depth extension coding, auxiliary picture coding, bit depth scalable coding, or combinations thereof. May be included.

  By enabling scalable extension, an access unit can be composed of a relatively large number of component pictures. This is not only a dependency expression and a layer expression, but also an encoded texture / depth view component. The coding size of some component pictures may be relatively small. This is because, for example, it is considered that the difference with respect to the base view or the base layer is expressed, or the depth component is relatively easy to compress. As a result, the overhead of the NAL unit header and slice header increases in proportion to the sharing of the number of bytes used for such component pictures.

  The HEVC codec is vulnerable to transmission errors, and any type of error concealment can result in an increase in both error magnitude and spatial domain over time. Many transmission systems such as MPEG-DASH are error-free systems.

  Some of the scalability characteristics of SVC and MVC are provided in the NAL unit header. This is due to the relatively large size of the NAL unit header, which is 4 bytes for SVC and MVC coded slices. It is possible to reduce the size of the NAL unit header if the NAL unit header or slice header is given a parameter set reference and provided with scalability characteristics. However, in such a design, an entity that performs sub-bitstream extraction or scalable adaptation from a bitstream, such as a multimedia gateway or a multipoint conference control unit (MCU), accesses the parameter set and It is required to maintain the activated state.

  In the following exemplary embodiment, the following several technical areas are provided together to provide a corresponding solution. First, this exemplary embodiment can provide a link for scalable extension of HEVC. Second, it enables sub-bitstream extraction that does not require access to and analysis of the parameter set and maintenance of the active track of the parameter set. Third, the exemplary embodiment can make the NAL unit header size smaller than the SVC and MVC 4-byte header. Fourth, when the picture includes a plurality of slices, the number of overhead bytes in the slice header can be reduced. Fifth, the exemplary embodiment can be further added to reduce the number of bytes in the slice header in the scalable extension.

  A component picture may be defined as a component picture delimiter NAL unit followed by a plurality of encoded slice NAL units. However, the subsequent encoded slice NAL unit is an encoded slice NAL unit that continues to the last unit in the decoding order among the last or next component picture delimiter NAL units of the access unit, and is the last or next Excluding separator NAL units. In HEVC without actually scalable extension, the component picture may be considered to include the coded picture of the access unit. In future scalable extensions, component pictures can also include view components, depth maps, dependency representations, layer representations, and the like.

  A plurality of component pictures may be separated from each other by component picture delimiter NAL units, and may carry values of common syntax elements used for decoding an encoded slice of the component picture.

  In this exemplary embodiment, each component picture is given a component picture dependent identifier (cpd_id). This identifier is signaled for both the component picture break NAL unit and the coded slice, forming an association between both.

  FIG. 10 schematically shows the structure of an access unit according to an exemplary embodiment.

  An access unit may begin with an access unit delimited NAL unit. Access unit delimiters may or may not be present. Zero or more SEI-NAL units may follow the access unit delimiter NAL unit (if present). A component picture delimiter NAL unit precedes each component picture. A component picture includes one or more coded slice NAL units. One access unit may include one or more component pictures.

The syntax element of the slice header is classified into a syntax structure. Each structure has the same characteristics within one component picture. That is, it remains unchanged for all the encoded slices of the component picture or changes for each encoded slice of the component picture. For example, the following syntax element structure or slice parameter structure may be defined:
1. Picture identification information (idr_pic_id and associated POC)
2. Reference picture set
3. Adaptive parameter set ID
4. Deblocking filter control
5. Adaptive loop filter control
6. Create reference picture list
7. Prediction weight table for weighted prediction.

  When coding a component picture break NAL unit, it is indicated which of the component picture break NAL units is present, and this may be shared by all coded slices of the component picture. The structure of syntax elements that do not exist in the component picture delimiter NAL unit may exist in the slice header.

  FIG. 11 shows an example of a component picture including one component picture delimiter NAL unit and two encoded slice NAL units. The component picture NAL unit includes three of the slice parameter syntax structures listed above: picture identification information, reference picture set, and adaptive parameter set ID. The coded slice inherits these three slice parameter structures from the component picture NAL unit. In this embodiment, the slice header of the encoded slice also includes a reference picture list creation structure. This structure is adapted for each coding slice in this embodiment and is not included in the component picture NAL unit. Although the remaining three structures are not shown in this embodiment, the operations of the deblocking filter and the adaptive loop filter are effectively controlled by APS. In addition, since weighted prediction is not used in this embodiment, there is no prediction weight table.

In an environment where it is desired to decode a slice independently even if one or more component picture breaks are lost, any of the following strategies can be used.
The encoder may choose not to encode the syntax element of the slice header in the component picture delimiter, but to encode the syntax element in the slice header as usual. For this reason, it is possible to achieve the same error resistance as the current HEVC WD.
・ It is also possible to adopt a mechanism that repeats component picture delimiter NAL units. This mechanism can also be adopted in HEVC. For example, an SEI mechanism that can arrange an SEI message at an arbitrary position of a bit stream is adopted, and component picture delimiters can be repeated together with this SEI message. Alternatively, or in addition, a mechanism at the transport level can be used. For example, if the sender knows that a component picture delimited NAL unit is appropriate in the transport packet, for example, when transmitting by a mechanism such as a PACSI-NAL unit in the SVC RTP payload format, repeat the component picture delimited NAL unit. Can do.

  In order to reduce the number of bits used for slice header parameters even when there are multiple component pictures in the access unit, prediction of the selected parameters may be performed beyond the component picture delimiter NAL unit. For example, encoding of depth-enhanced multi-view video has an advantage that a part of a slice parameter structure is predicted between a texture view component and a depth view component of the same view_id. For other syntax elements, it is better to inherit syntax elements from different view components in the same component type (texture or depth).

  In fact, a component picture break NAL unit for a non-base component picture can include the structure of the syntax element indicated by the content of another component picture break NAL unit or a reference to it. This reference is given in units of cpd_id values. The syntax elements of the component picture delimiter NAL unit whose cpd_id is CPDID1 are clustered into a syntax element set. Each syntax element set may be selectively copied from a component picture delimiter NAL unit that is in the same access unit and has cpd_id of CPDID2 first. Since CPDID1 is larger than CPDID2, the slice header parameters are shared effectively and flexibly within the component picture.

  FIG. 12 shows an example of a multi-view plus depth access unit with two sets of texture and depth component pictures. The bitstream order of these component pictures is assumed as follows: base view texture picture, base view depth picture, non-base view texture picture, and non-base view depth picture. In this embodiment, the picture identification information and the reference picture set structure are the same for all component pictures, and are included in the component picture delimiter NAL unit whose cpd_id exceeds 0 by reference. The reference picture list for the texture component picture is the same, and the reference picture list for the depth component picture is the same. However, the reference picture list of texture component pictures is different from the list of depth component pictures. Therefore, the prediction source changes with respect to the reference picture list creation structure. In this embodiment, it is assumed that the remaining four slice parameter structures are included or not present in the slice header. Since cpd_id of the last two picture delimiter NAL units is not used for prediction of subsequent picture delimiter NAL units, it may be the same value (2).

  Each component picture is given a component picture dependent identifier (cpd_id) and is signaled with a NAL unit header. There is a restriction on the value of cpd_id so that sub-bitstream extraction can be performed based on cpd_id. In other words, the bit stream formed by excluding the component picture delimiter NAL unit and the coded slice NAL unit having a specific cpd_id exceeding 0 is a conforming bit stream.

  In this exemplary embodiment, cpd_id is included in the NAL unit header. Therefore, the range is limited (for example, 5 bits). In general, the number of component pictures in an access unit can be greater than the maximum value (eg 32) of this range. As a result, the cpd_id value may have to be reused within the access unit. In some exemplary embodiments, the component picture break NAL unit has a nested prediction structure. That is, the component picture delimited NAL unit whose cpd_id is CPDID1 is predicted from the next component, and is not predicted from other component picture delimited NAL units. In the following, the component picture delimited NAL units in the access unit are indexed as 0, 1, 2,... In decoding order (that is, bitstream order). The index of the current component picture delimited NAL unit whose cpd_id is CPDID1 is currIndex, and currIndex is greater than zero. This is illustrated by the following pseudo code.

currSmallestCpdId = CPDID1
for (i = currIndex-1, j = 0; i>0; i--) {
if (cpdId [i] <currSmallestCpdId) {
refCpdIdx [j] = i
j ++
currSmallestCpdId = cpd_id [i]
}
}
numRefCpdIdx = j

The pseudo code is as follows:
The input parameter cpdId [i] gives the cpd_id value of the component picture delimited NAL unit with index i in the access unit;
The output parameter numRefCpdIdx gives the number of component picture delimited NAL units used to predict the current component picture delimited NAL unit;
• If numRefCpdIdx is greater than 0, refCpdIdx [j] gives the index of the component picture delimiter NAL unit used to predict the current component picture delimiter NAL unit; where j is between 0 and numRefCpdIdx-1 is there.

  As a result, sub-bitstream extraction within the access unit can be performed with finer granularity as follows. When a component picture consisting of a component picture delimiter NAL unit whose cpd_id is all CPDID1 followed by a coded slice NAL unit is deleted from the bitstream, the component picture to be deleted from the bitstream shall be determined by the following algorithm: Can do. As described above, the component pictures in the access unit are indexed as 0, 1, 2,... In decoding order (that is, bit stream order). The current (to be deleted) component picture index is currIndex and the total number of component pictures in the access unit is numIndex.

for (i = currIndex + 1, j = 0; i <numIndex; i ++) {
if (cpdId [i]> CPDID1) {
toBeRemovedIdx [j] = i
j ++
}
else
break
}
numToBeRemovedIdx = j

In this pseudocode, "break" exits the loop (as in the C programming language). The inputs and outputs are as follows:
The input parameter cpdId [i] gives the cpd_id value of the component picture delimited NAL unit with index i in the access unit;
The output parameter numToBeRemovedIdx gives the number of component pictures to be removed from the access unit along with the current component picture;
• If numToBeRemovedIdx0 is exceeded, toBeRemovedIdx [j] gives the index of the component picture to be removed from the access unit along with the current component picture; where j is in the range of 0 to numToBeRemoved−1.

  This kind of deletion processing or sub-bitstream extraction processing does not check which component picture delimiter NAL unit is actually used for prediction. On the other hand, only information about which component picture delimiter NAL unit is used for prediction is used, as controlled by the constraint due to the content of cpd_id. However, this sub-bitstream extraction process may operate only with the cpd_id value accessible by the NAL unit header, and is used by a direct method such as a media gateway or MCU.

  In the following, exemplary embodiments according to part of the syntax structure are described.

The NAL unit syntax may include:

  The meaning of cpd_id is added as follows. cpd_id (bold in the original text and a bitstream syntax element) is an identifier of a component picture. The value of cpd_id is restricted as described above.

  The NAL unit table may include:

The syntax structure of the component picture delimiter NAL unit according to the exemplary embodiment is as follows.

  The structure_idc (bold in the original text and a syntax element of the bit stream) is an identifier of the structure used in this syntax structure. The structure_idc is used to indicate the presence and combination of various syntax elements present in the slice header and component picture delimiter NAL unit defined in the HEVC scalable extension. A component picture delimited NAL unit having an unrecognizable structure_idc value may be ignored in the decoding process.

  When single_slice_type_flag (bold in the original text and a bitstream syntax element) is 0, it is specified that the component picture may include a slice of another slice type. When single_slice_type_flag is 1, it is specified that there is a possibility that component pictures include slices of the same slice type.

  When pred_flag (bold in the original text and a bitstream syntax element) is 0, it specifies that the following slice parameter structure is included in the current NAL unit. When pred_flag is 1, it specifies that the following slice parameter structure is indicated by a reference from a component picture delimiter NAL unit whose cpd_id is ref_cpd_id [idx].

  ref_cpd_id [idx] (bold in the text and is a bitstream syntax element) specifies that the component picture delimiter NAL unit is used as a reference to the indicated slice parameter structure.

  slice_param_flag [i] [idx] ("slice_param_flag" is bold in the text and is a bitstream syntax element) specifies that the i-th slice parameter structure is included in the current component picture delimiter NAL unit . This value may be assigned by reference to another component picture break NAL unit.

According to an exemplary embodiment, the slice header syntax may include:

  During decoding or analysis of a slice header, syntax elements included in the preceding component picture delimiter NAL unit with the same cpd_id are valid.

  The following slice parameter syntax structure may be defined. This syntax structure includes parameters present in the slice header of the HEVC draft standard.

  The meaning of the syntax elements in these structures is unchanged compared to the meaning given in the HEVC draft standard.

For HEVC scalable extensions, one or more new structure_idc values may be used. A scalable extension may also allow the use of one or more new slice parameter structures. Implementation of scalable extension for component picture delimited NAL units with dependency_id and quality_id similar to SVC is shown below.

  Predicting or including another reference from a specific component picture delimited NAL unit by reference is subject to the structure_idc value of the reference component picture delimited NAL unit and / or the predicted component picture delimited NAL unit for prediction. May be. For example, a particular slice parameter structure may be valid for depth component pictures and may not be present for texture component pictures. Therefore, no prediction is made for such a slice parameter structure.

  The exemplary embodiments described above have been described using bitstream syntax. However, it should also be understood that corresponding configurations and / or computer programs may be present in an encoder that generates a bitstream and / or a decoder that decodes a bitstream. Similarly, it should also be understood that while the exemplary embodiment has been described with reference to an encoder, the resulting bitstream and decoder are provided with corresponding elements. Similarly, it should also be understood that while the exemplary embodiments have been described with reference to a decoder, the encoder comprises a configuration and / or computer program for generating a bitstream that is decoded by the decoder. .

  Although the foregoing examples describe embodiments of the present invention that operate in an electronic device codec, it should be understood that the present invention may be implemented as part of any video codec as described below. . Thus, for example, embodiments of the invention may be implemented in a video codec that may implement video coding over a fixed or wired communication path.

  The user equipment may then comprise such a video codec as described in the embodiments of the present invention described above. The phrase “user equipment” may represent any type of wireless user equipment, such as a mobile phone, a portable data processing device, or a portable web browser.

  Furthermore, a public land mobile network (PLMN) may include the video codec described above.

  In general, the various embodiments may be implemented in hardware or application specific circuits, software, logic, or combinations thereof. For example, in some cases it may be implemented in hardware, while in other cases it may be implemented in firmware or software executed by a computer device such as a controller or microprocessor. Various aspects of the invention are described or illustrated using block diagrams, flowcharts, or other graphical descriptions. These blocks, devices, systems, technologies, or methods described herein are, by way of non-limiting example, hardware, software, firmware, application specific circuits and logic, general purpose hardware, controllers, and other computing devices. , Or a combination thereof, should be understood.

  Embodiments of the invention may then be implemented by computer software, hardware, or a combination of software and hardware that can be executed by the data processor of the mobile device. Also, in this regard, any block of logic flow shown in the accompanying drawings represents a program step or an interconnected logic circuit / block / function, or a combination of a program step, logic circuit / block / function. Note that it may be. The software may be stored in a physical medium such as a memory chip, a memory block mounted in a processor, a magnetic medium such as a hard disk or a flexible disk, or an optical medium such as a DVD or a CD that is a data variant thereof.

  Various embodiments of the present invention can be implemented using computer program code residing in memory, causing an associated apparatus to perform the invention. For example, the terminal device may include a circuit and an electronic device that process / transmit / receive data, a computer program code in a memory, and a processor. When the processor executes the computer program code, the processor causes the terminal device to perform the configuration of the present embodiment. Furthermore, the network device may include a circuit and an electronic device for processing / transmitting / receiving data, computer program code in a memory, and a processor. When the processor executes the computer program code, the processor causes the network device to perform the configuration of the present embodiment.

  The memory may be of any type suitable for the local technical environment. For example, it may be implemented using a variety of compatible data storage technologies such as semiconductor-based memory devices, magnetic memory device systems, optical memory device systems, fixed and mobile memories, and the like. The data processor may be of any type suitable for a local technical environment, including, but not limited to, one or more general purpose computers, application specific computers, microprocessors, digital signal processors (DSPs), multicores. A processor based on the processor architecture may be included.

  Embodiments of the present invention can be implemented with a variety of elements, such as integrated circuit modules, and the design of integrated circuits is often an automated process. Complex and powerful software tools are available that translate logic level designs into semiconductor circuit designs for etching and forming on semiconductor substrates.

  Programs offered by vendors such as Synopsys, Inc. in Mountain View, California and Cadence Design in San Jose, Calif., Are based on proven design rules and a library of proven design modules. Arrange the elements. Once the design of the semiconductor circuit is complete, it is sent to a semiconductor manufacturing facility or so-called “fab” in the form of a standard electronic format such as Opus or GDSII.

  The foregoing description describes in full and detailed non-limiting embodiments of the present invention. However, it will be apparent to one skilled in the art to which this application pertains that various modifications and adaptations are possible in view of the foregoing description in conjunction with the accompanying drawings and claims. . Further, all of these matters and similar variations taught by the present invention are all within the scope of the present invention.

  In addition, some examples are given below.

According to a first embodiment, there is provided a method for encoding an uncompressed picture into an encoded picture including a slice, the method comprising encoding the uncompressed picture into an encoded picture including a slice. Is done. This encoding is
Categorizing the syntax elements for the slice into a first set and a second set;
Determining syntax element values for the first set and the second set;
• selectively encoding the first set in a first slice group parameter set or slice header, wherein the encoding is:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
* Encoding the first set of syntax element values;
* The encoding comprising any one of omitting both;
• selectively encoding the second set in a second slice group parameter set or slice header, wherein the encoding is:
* Providing indication information of inclusion of said related second set from another slice group parameter set;
* Encoding the second set of syntax element values;
* Includes the encoding, including any one of omitting both.

  In some embodiments, the method includes including at least one subset of syntax elements in the slice group parameter set.

  In some embodiments, the method includes estimating the content or instance of the slice group parameter set from other syntax structures that are encoded or decoded or already in the bitstream.

  In some embodiments, the method includes forming the slice group parameter set from a slice header of a texture view component of a base view.

  In some embodiments, the method includes generating an identification value for the estimated slice group parameter set.

  In some embodiments of the method, the slice group parameter set is valid within a particular access unit associated with it.

  In some embodiments of the method, the syntax structure of the slice group parameter set is included in a network abstraction layer unit sequence for a specific access unit, the sequence being in decoding order or bitstream order, the slice group parameter set. The set is valid from its appearance position to the end of the access unit.

  In some embodiments of the method, the slice group parameter set is valid for multiple access units.

  In some embodiments, the method includes encoding a plurality of slice group parameter sets for one access unit.

  In some embodiments, the method determines whether the encoded subset of at least one subset of syntax element values in a slice header is the same as in a subsequent slice header; The encoding of the slice group parameter set in the bitstream.

  In some embodiments of the method, the syntax structure of the slice group parameter set includes an identifier.

  In some embodiments, the method includes using the identifier to refer to a particular instance of a slice group parameter set.

  In some embodiments, the method includes using the identifier to reference the slice group parameter set from a slice header or another slice group parameter set.

  In some method embodiments, a predetermined numbering space is used for the identifier.

  In some embodiments, the method uses a specific slice group parameter set identification value for the first slice group parameter set, and the first slice group parameter set is a slice header or a slice group parameter set. If not referenced by any, it includes using the slice group parameter set identification value for a second slice group parameter set.

  In some embodiments, the method includes repeating the syntax structure of the slice group parameter set in a bitstream.

  In some embodiments, the method includes identifying the syntax structure of the slice group parameter set using a bitstream order of the syntax structure of the slice group parameter set and a predetermined numbering scheme.

In some embodiments, the method includes forming a syntax element set for the slice group parameter set from at least one of the following:
A syntax element indicating a scalable layer and / or other scalable characteristics;
A syntax element indicating a view and / or other multi-view characteristics;
Syntax elements associated with specific component types of multiview video;
Syntax elements associated with access unit identification information;
Syntax elements associated with the decoding order;
Syntax elements related to output order;
Syntax elements associated with other syntax elements that are invariant to all slices of the access unit;
A syntax element that is invariant across all slices of the view component;
Syntax elements related to reference picture list changes;
Syntax elements associated with the set of reference pictures used;
Syntax elements associated with decoding reference picture markings;
Syntax elements associated with the prediction weight table for weighted prediction;
A syntax element that controls deblocking filtering;
A syntax element that controls adaptive loop filtering;
A syntax element that controls the sample adaptive offset.

In some embodiments, the elements include one or more of the following when encoding the slice group parameter set:
Encoding the syntax element into a syntax structure of a slice group parameter set;
Including the syntax element set in the slice group parameter set by reference;
Indicating that the syntax element set is not in the slice group parameter set.

  In some embodiments, the method includes a syntax element set associated with a scalable layer in the slice group parameter set; and a syntax element that is invariant in all slices of a view component is included in the slice group parameter set. including.

  In some embodiments, the method includes, by reference or inclusion, a syntax element associated with a change of a reference picture list in the syntax structure of the slice group parameter set, or from the syntax structure of the slice group parameter set. Including absence.

  In some embodiments, the method includes encoding the slice group parameter set as a network abstraction layer unit.

  In some embodiments, the method includes encoding a network abstraction layer (NAL) unit of a slice group parameter set in a bitstream with an encoded slice NAL unit.

According to a second embodiment, an apparatus is provided comprising at least one processor and at least one memory containing computer program code. The at least one memory and the computer program code are stored in the device using the at least one processor:
Classifying the syntax elements for the slices that the encoded picture contains into a first set and a second set;
Determining syntax element values for the first set and the second set;
• selectively encoding the first set in a first slice group parameter set or slice header, the following:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
* Encoding the first set of syntax element values;
* The encoding comprising any one of omitting both;
Selectively encoding the second set in a second slice group parameter set or slice header, the following:
* Providing indication information of inclusion of said related second set from another slice group parameter set;
* Encoding the second set of syntax element values;
* Configured to perform the encoding, including any one of omitting both.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, the apparatus further includes a subset of syntax elements in the slice group parameter set. To include at least one.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further encoded or decoded to the apparatus when executed by the at least one processor, or the bit Let the stream infer the contents or instances of the slice group parameter set from other existing syntax structures.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further executed by the at least one processor from the slice view of a texture view component of a base view to the apparatus. A slice group parameter set is formed.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, the apparatus further comprising an identification value for the estimated slice group parameter set. To create.

  In some embodiments of the apparatus, the slice group parameter set is valid within the particular access unit associated with it.

  In some embodiments of the apparatus, when the at least one memory and code stored in the memory are executed by the at least one processor, the apparatus further includes a syntax structure of the slice group parameter set, Included in the network abstraction layer unit sequence for a particular access unit. Here, the sequence is in the order of decoding or bitstream, and the slice group parameter set is valid from its appearance position to the end of the access unit.

  In some embodiments of the apparatus, the slice group parameter set is valid for multiple access units.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further executed by the at least one processor, the apparatus further comprising a plurality of slice groups for one access unit. Encode the parameter set.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, the apparatus further comprising at least one of syntax element values in a slice header. Determining whether the subset to be encoded is the same as in the subsequent slice header; and if so, encoding the slice group parameter set in the bitstream .

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further identifierd in the syntax structure of the slice group parameter set when executed by the at least one processor. Is included.

  In some embodiments of the device, the at least one memory and code stored in the memory further refer to the device for a particular instance of a slice group parameter set when executed by the at least one processor. The identifier is used for this purpose.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further executed by the at least one processor from the apparatus further from a slice header or another slice group parameter set. The identifier is used to refer to the slice group parameter set.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory, when executed by the at least one processor, further provide the apparatus with a predetermined numbering space for the identifier. Let it be used.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further specified to the apparatus for a first slice group parameter set when executed by the at least one processor. The slice group parameter set identification value and if the first slice group parameter set is not referenced by either a slice header or a slice group parameter set, the slice group for the second slice group parameter set Using the parameter set identification value is accomplished.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor when the apparatus further includes a synth of the slice group parameter set in a bitstream. Repeat the tax structure.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, the apparatus further comprising a bit of a syntax structure of the slice group parameter set. The syntax structure of the slice group parameter set is identified using a stream order and a predetermined numbering scheme.

In some embodiments of the apparatus, the at least one memory and code stored in the memory, when executed by the at least one processor, further include at least one of the following: Form a syntax element set for a set:
A syntax element indicating a scalable layer and / or other scalable characteristics;
A syntax element indicating a view and / or other multi-view characteristics;
Syntax elements associated with specific component types of multiview video;
Syntax elements associated with access unit identification information;
Syntax elements associated with the decoding order;
Syntax elements related to output order;
Syntax elements associated with other syntax elements that are invariant to all slices of the access unit;
A syntax element that is invariant across all slices of the view component;
Syntax elements related to reference picture list changes;
Syntax elements associated with the set of reference pictures used;
Syntax elements associated with decoding reference picture markings;
Syntax elements associated with the prediction weight table for weighted prediction;
A syntax element that controls deblocking filtering;
A syntax element that controls adaptive loop filtering;
A syntax element that controls the sample adaptive offset.

In some embodiments of the apparatus, when the at least one memory and the code stored in the memory are executed by the at least one processor, the apparatus further encodes the slice group parameter set when: Have one or more of the following:
Encoding the syntax element into a syntax structure of a slice group parameter set;
Including the syntax element set in the slice group parameter set by reference;
Indicating that the syntax element set is not in the slice group parameter set.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory, when executed by the at least one processor, further include a syntax element set associated with a scalable layer in the apparatus. Including in the slice group parameter set; including in the slice group parameter set syntax elements that are invariant in all slices of the view component.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, the apparatus further includes syntax associated with a reference picture list change. An element is included in the syntax structure of the slice group parameter set by reference or inclusion, or is excluded from the syntax structure of the slice group parameter set.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further executed by the at least one processor, the apparatus further comprising a slice group parameter set as a network abstraction layer unit. Encode.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, the apparatus further comprising a network abstraction of a slice group parameter set in a bitstream. A layer (NAL) unit is encoded with a coded slice NAL unit.

According to a third embodiment, a computer program product is provided that includes one or more sequences of one or more instructions. When the one or more sequences of the one or more instructions are executed by one or more processors, the device at least:
Classifying the syntax elements for the slices that the encoded picture contains into a first set and a second set;
Determining syntax element values for the first set and the second set;
• selectively encoding the first set in a first slice group parameter set or slice header, the following:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
* Encoding the first set of syntax element values;
* The encoding comprising any one of omitting both;
Selectively encoding the second set in a second slice group parameter set or slice header, the following:
* Providing indication information of inclusion of said related second set from another slice group parameter set;
* Encoding the second set of syntax element values;
* Performing the encoding, including any one of omitting both.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further includes at least one subset of syntax elements in the slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes the bitstream to infer the content or instance of the slice group parameter set from other syntax structures already encoded or decoded.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes the slice group parameter set to be formed from a slice header of the texture view component of the base view.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus is further caused to generate an identification value for the estimated slice group parameter set.

  In some embodiments of the computer program product, the slice group parameter set is valid within a particular access unit associated with it.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further includes the syntax structure of the slice group parameter set in a network abstraction layer unit sequence for a specific access unit. Here, the sequence is in the order of decoding or bitstream, and the slice group parameter set is valid from its appearance position to the end of the access unit.

  In some embodiments of the computer program product, the slice group parameter set is valid for a plurality of access units.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further encodes a plurality of slice group parameter sets for one access unit.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further determines whether at least one subset of the syntax element values in the slice header that are encoded is identical to that in the subsequent slice header; Encoding the slice group parameter set in a stream is performed.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further includes an identifier in the syntax structure of the slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes the identifier to be used to refer to a specific instance of the slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes the identifier to be used to reference the slice group parameter set from a slice header or another slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes a predetermined numbering space to be used for the identifier.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further uses a specific slice group parameter set identification value for the first slice group parameter set, and the first slice group parameter set is not referenced by either the slice header or the slice group parameter set. The slice group parameter set identification value is used for the second slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further repeats the syntax structure of the slice group parameter set in the bitstream.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further identifies the syntax structure of the slice group parameter set using a bitstream order of the syntax structure of the slice group parameter set and a predetermined numbering scheme.

In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes a syntax element set for the slice group parameter set to be formed from at least one of the following:
A syntax element indicating a scalable layer and / or other scalable characteristics;
A syntax element indicating a view and / or other multi-view characteristics;
Syntax elements associated with specific component types of multiview video;
Syntax elements associated with access unit identification information;
Syntax elements associated with the decoding order;
Syntax elements related to output order;
Syntax elements associated with other syntax elements that are invariant to all slices of the access unit;
A syntax element that is invariant across all slices of the view component;
Syntax elements related to reference picture list changes;
Syntax elements associated with the set of reference pictures used;
Syntax elements associated with decoding reference picture markings;
Syntax elements associated with the prediction weight table for weighted prediction;
A syntax element that controls deblocking filtering;
A syntax element that controls adaptive loop filtering;
A syntax element that controls the sample adaptive offset.

In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes the encoding of the slice group parameter set to perform one or more of the following:
Encoding the syntax element into a syntax structure of a slice group parameter set;
Including the syntax element set in the slice group parameter set by reference;
Indicating that the syntax element set is not in the slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes a set of syntax elements associated with the scalable layer to be included in the slice group parameter set; and a syntax element that is invariant in all slices of a view component is included in the slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus may further include, by reference or inclusion, a syntax element related to the change of the reference picture list in the syntax structure of the slice group parameter set, or to be absent from the syntax structure of the slice group parameter set. Let it be carried out.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further encodes the slice group parameter set as a network abstraction layer unit.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus further causes a network abstraction layer (NAL) unit of the slice group parameter set in the bitstream to be encoded along with the encoded slice NAL unit.

According to a fourth embodiment, the following device is provided, which is
Means for classifying syntax elements for slices included in the coded picture into a first set and a second set;
Means for determining syntax element values for the first set and the second set;
Means for selectively encoding the first set in a first slice group parameter set or slice header, wherein the means for encoding is:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
* Encoding the first set of syntax element values;
* The means for encoding comprising any one of omitting both;
Means for selectively encoding said second set in a second slice group parameter set or slice header, said means for encoding being:
* Providing indication information of inclusion of said related second set from another slice group parameter set;
* Encoding the second set of syntax element values;
* The means for encoding comprises any one of omitting both.

According to a fifth embodiment, the following method is provided, comprising decoding a coded slice of a coded picture, said decoding comprising:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first syntax element set and the second syntax element set to be used for decoding the encoded slice, and decoding the first and second sets The thing is:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

  In some embodiments, the element includes decoding the first set of syntax elements when the first indication information does not indicate the third slice group parameter set.

  In some embodiments, the element includes decoding the second set of syntax elements if the second indication information does not indicate the fourth slice group parameter set.

According to a sixth embodiment, an apparatus is provided comprising at least one processor and at least one memory containing computer program code. The at least one memory and the computer program code are stored in the device using the at least one processor:
Configured to decode an encoded slice of an encoded picture, said decoding comprising:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first and second set of sets, performed by decoding the first set of syntax elements and the second set of syntax elements to be used for decoding the coded slice To do:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

  In some embodiments of the apparatus, when the at least one memory and code stored in the memory are executed by the at least one processor, the apparatus further includes the first indication information as the third indication information. If no slice group parameter set is indicated, the first set of syntax elements is decoded.

  In some embodiments of the apparatus, when the at least one memory and code stored in the memory are executed by the at least one processor, the apparatus further includes the second indication information as the fourth indication information. If no slice group parameter set is indicated, the second set of syntax elements is decoded.

According to a seventh embodiment, a computer program product is provided that includes one or more sequences of one or more instructions. When the one or more sequences of the one or more instructions are executed by one or more processors, at least on the device,
Decoding an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first and second set of sets, performed by decoding the first set of syntax elements and the second set of syntax elements to be used for decoding the coded slice To do:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the If the first indication information does not indicate the third slice group parameter set, the apparatus causes the first set of syntax elements to be decoded.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the If the second indication information does not indicate the fourth slice group parameter set, the apparatus causes the second set of syntax elements to be decoded.

According to the eighth embodiment, the following method is provided, which comprises:
Decoding a coded slice of a coded picture, said decoding comprising:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first and second set of sets, performed by decoding the first set of syntax elements and the second set of syntax elements to be used for decoding the coded slice To do:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first slice group parameter set Decoding the syntax elements of the set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second slice group parameter set Decoding the syntax elements of the set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

  In some embodiments, the method includes decoding at least one subset of syntax elements from the slice group parameter set.

  In some embodiments, the method includes estimating the content or instance of the slice group parameter set from other syntax structures that are encoded or decoded or already in the bitstream.

  In some embodiments, the method includes decoding an identification value indicative of the estimated slice group parameter set.

  In some embodiments of the method, the slice group parameter set is valid within a particular access unit associated with it.

  In some embodiments of the method, the syntax structure of the slice group parameter set is included in a network abstraction layer unit sequence for a specific access unit, the sequence being in decoding order or bitstream order, the slice group parameter set. The set is valid from its appearance position to the end of the access unit.

  In some embodiments of the method, the slice group parameter set is valid for multiple access units.

  In some embodiments, the method includes decoding a plurality of slice group parameter sets for one access unit.

  In some embodiments of the method, the syntax structure of the slice group parameter set includes an identifier.

  In some embodiments, the method includes using the identifier to refer to a particular instance of a slice group parameter set.

  In some embodiments, the method includes using the identifier to reference the slice group parameter set from a slice header or another slice group parameter set.

  In some method embodiments, a predetermined numbering space is used for the identifier.

  In some embodiments, the method uses a specific slice group parameter set identification value for the first slice group parameter set, and the first slice group parameter set is a slice header or a slice group parameter set. If not referenced by any, it includes using the slice group parameter set identification value for a second slice group parameter set.

  In some embodiments, the method includes iterating from the bitstream to decode the syntax structure of the slice group parameter set.

  In some embodiments, the method includes identifying the syntax structure of the slice group parameter set using a bitstream order of the syntax structure of the slice group parameter set and a predetermined numbering scheme.

In some embodiments, the method includes decoding a syntax element set for the slice group parameter set to obtain at least one of the following:
A syntax element indicating a scalable layer and / or other scalable characteristics;
A syntax element indicating a view and / or other multi-view characteristics;
Syntax elements associated with specific component types of multiview video;
Syntax elements associated with access unit identification information;
Syntax elements associated with the decoding order;
Syntax elements related to output order;
Syntax elements associated with other syntax elements that are invariant to all slices of the access unit;
A syntax element that is invariant across all slices of the view component;
Syntax elements related to reference picture list changes;
Syntax elements associated with the set of reference pictures used;
Syntax elements associated with decoding reference picture markings;
Syntax elements associated with the prediction weight table for weighted prediction;
A syntax element that controls deblocking filtering;
A syntax element that controls adaptive loop filtering;
A syntax element that controls the sample adaptive offset.

In some embodiments, the elements include one or more of the following when decoding the slice group parameter set:
Decoding the syntax element from the syntax structure of the slice group parameter set;
Determining whether the syntax element set was included by reference to the slice group parameter set;
Determining whether the syntax element set is marked not in the slice group parameter set.

  In some embodiments, the method decodes a syntax element set associated with a scalable layer from the slice group parameter set; and decodes a syntax element that is invariant for all slices of a view component from the slice group parameter set. Including doing.

  In some embodiments, the method includes that a syntax element associated with a change of a reference picture list is included in the syntax structure of the slice group parameter set by reference or inclusion, or a syntax of the slice group parameter set. Determining whether it is not present in the tax structure.

  In some embodiments, the method includes decoding a slice group parameter set from a network abstraction layer unit.

  In some embodiments, the method includes decoding a network abstraction layer (NAL) unit of a slice group parameter set with a coded slice NAL unit from a bitstream.

According to a ninth embodiment, an apparatus is provided comprising at least one processor and at least one memory containing computer program code. The at least one memory and the computer program code are stored in the device using the at least one processor:
Decoding an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first and second set of sets, performed by decoding the first set of syntax elements and the second set of syntax elements to be used for decoding the coded slice To do:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first slice group parameter set Decoding the syntax elements of the set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second slice group parameter set Decoding the syntax elements of the set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, wherein the apparatus further includes a subset of syntax elements from the slice group parameter set. Decrypt at least one of

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further encoded or decoded to the apparatus when executed by the at least one processor, or the bit Let the stream infer the contents or instances of the slice group parameter set from other existing syntax structures.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further identified to the apparatus to indicate the estimated slice group parameter set when executed by the at least one processor. Decrypt the value.

  In some embodiments of the apparatus, the slice group parameter set is valid within the particular access unit associated with it.

  In some embodiments of the apparatus, the syntax structure of the slice group parameter set is included in a network abstraction layer unit sequence for a particular access unit, the sequence being in decoding order or bitstream order, the slice group parameter set. The set is valid from its appearance position to the end of the access unit.

  In some embodiments of the apparatus, the slice group parameter set is valid for multiple access units.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further executed by the at least one processor, the apparatus further comprising a plurality of slice groups for one access unit. Decode the parameter set.

  In some embodiments of the apparatus, the slice group parameter set syntax structure includes an identifier.

  In some embodiments of the device, the at least one memory and code stored in the memory further refer to the device for a particular instance of a slice group parameter set when executed by the at least one processor. The identifier is used for this purpose.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further executed by the at least one processor from the apparatus further from a slice header or another slice group parameter set. The identifier is used to refer to the slice group parameter set.

  In some embodiments of the device, a predetermined numbering space is used for the identifier.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further specified to the apparatus for a first slice group parameter set when executed by the at least one processor. The slice group parameter set identification value and if the first slice group parameter set is not referenced by either a slice header or a slice group parameter set, the slice group for the second slice group parameter set Using the parameter set identification value is accomplished.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further repeated from the bitstream upon execution of the slice group parameter set by the apparatus when executed by the at least one processor. Decrypt the syntax structure.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, the apparatus further comprising a bit of a syntax structure of the slice group parameter set. The syntax structure of the slice group parameter set is identified using a stream order and a predetermined numbering scheme.

In some embodiments of the device, the at least one memory and the code stored in the memory, when executed by the at least one processor, further obtain at least one of the following for the device: Decode the syntax element set for the slice group parameter set:
A syntax element indicating a scalable layer and / or other scalable characteristics;
A syntax element indicating a view and / or other multi-view characteristics;
Syntax elements associated with specific component types of multiview video;
Syntax elements associated with access unit identification information;
Syntax elements associated with the decoding order;
Syntax elements related to output order;
Syntax elements associated with other syntax elements that are invariant to all slices of the access unit;
A syntax element that is invariant across all slices of the view component;
Syntax elements related to reference picture list changes;
Syntax elements associated with the set of reference pictures used;
Syntax elements associated with decoding reference picture markings;
Syntax elements associated with the prediction weight table for weighted prediction;
A syntax element that controls deblocking filtering;
A syntax element that controls adaptive loop filtering;
A syntax element that controls the sample adaptive offset.

In some embodiments of the apparatus, when the at least one memory and code stored in the memory are executed by the at least one processor, the apparatus further includes: Have one or more of:
Decoding the syntax element from the syntax structure of the slice group parameter set;
Determining whether the syntax element set was included by reference to the slice group parameter set;
Determining whether the syntax element set is marked not in the slice group parameter set.

  In some embodiments of the apparatus, the at least one memory and code stored in the memory are further associated with a scalable layer from the slice group parameter set when executed by the at least one processor. Decoding a syntax element set; and decoding a syntax element that is invariant in all slices of the view component from the slice group parameter set.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor, the apparatus further includes syntax associated with a reference picture list change. It is determined whether an element is included in the syntax structure of the slice group parameter set by reference or inclusion or does not exist in the syntax structure of the slice group parameter set.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor when the apparatus further includes a slice group parameter set from a network abstraction layer unit. Decrypt.

  In some embodiments of the apparatus, the at least one memory and the code stored in the memory are further executed by the at least one processor when the apparatus further includes a network abstraction of slice group parameter sets from the bitstream. A layer (NAL) unit is decoded together with a coded slice NAL unit.

According to a tenth embodiment, a computer program product is provided that includes one or more sequences of one or more instructions. When the one or more sequences of the one or more instructions are executed by one or more processors, at least on the device,
Decoding an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set Identifying;
Decoding the first and second set of sets, performed by decoding the first set of syntax elements and the second set of syntax elements to be used for decoding the coded slice To do:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first slice group parameter set Decoding the syntax elements of the set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second slice group parameter set Decoding the syntax elements of the set;
* Decoding the encoded slice using the decoded first syntax element set and second syntax element set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the Let the apparatus decode at least one subset of syntax elements from the slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the Let the device estimate the content or instance of the slice group parameter set from other syntax structures that have been encoded or decoded, or existing in the bitstream.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus causes the identification value indicating the estimated slice group parameter set to be decoded.

  In some embodiments of the computer program product, the slice group parameter set is valid within a particular access unit associated with it.

  In some embodiments of the computer program product, the syntax structure of the slice group parameter set is included in a network abstraction layer unit sequence for a particular access unit, the sequence being in decoding order or bitstream order, the slice The group parameter set is valid from its appearance position to the end of the access unit.

  In some embodiments of the computer program product, the slice group parameter set is valid for a plurality of access units.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus causes a plurality of slice group parameter sets to be decoded for one access unit.

  In some embodiments of the computer program product, the slice group parameter set syntax structure includes an identifier.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the Let the device use the identifier to refer to a particular instance of a slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the Let the device use the identifier to reference the slice group parameter set from a slice header or another slice group parameter set.

  In some embodiments of the computer program product, a predetermined numbering space is used for the identifier.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The device uses a specific slice group parameter set identification value for the first slice group parameter set, and the first slice group parameter set is not referenced by either the slice header or the slice group parameter set , Using the slice group parameter set identification value for the second slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, the one or more sequences of the one or more instructions being executed by one or more processors, the bit The syntax structure of the slice group parameter set is decoded repeatedly from the stream.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus is configured to identify the syntax structure of the slice group parameter set using a bitstream order of the syntax structure of the slice group parameter set and a predetermined numbering scheme.

In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the Let the device decode a syntax element set for the slice group parameter set to obtain at least one of the following:
A syntax element indicating a scalable layer and / or other scalable characteristics;
A syntax element indicating a view and / or other multi-view characteristics;
Syntax elements associated with specific component types of multiview video;
Syntax elements associated with access unit identification information;
Syntax elements associated with the decoding order;
Syntax elements related to output order;
Syntax elements associated with other syntax elements that are invariant to all slices of the access unit;
A syntax element that is invariant across all slices of the view component;
Syntax elements related to reference picture list changes;
Syntax elements associated with the set of reference pictures used;
Syntax elements associated with decoding reference picture markings;
Syntax elements associated with the prediction weight table for weighted prediction;
A syntax element that controls deblocking filtering;
A syntax element that controls adaptive loop filtering;
A syntax element that controls the sample adaptive offset.

In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the Let the device perform one or more of the following when decoding the slice group parameter set:
Decoding the syntax element from the syntax structure of the slice group parameter set;
Determining whether the syntax element set was included by reference to the slice group parameter set;
Determining whether the syntax element set is marked not in the slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus performs decoding a syntax element set associated with a scalable layer from the slice group parameter set; and decoding syntax elements that are invariant in all slices of a view component from the slice group parameter set.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the Whether the apparatus includes a syntax element related to the change of the reference picture list by reference or inclusion in the syntax structure of the slice group parameter set or does not exist in the syntax structure of the slice group parameter set Let me decide.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the Let the device decode the slice group parameter set from the network abstraction layer unit.

  In some embodiments, the computer program product includes one or more sequences of one or more instructions, and when the one or more sequences of the one or more instructions are executed by one or more processors, the The apparatus causes a network abstraction layer (NAL) unit of a slice group parameter set to be decoded together with a coded slice NAL unit from the bitstream.

According to an eleventh embodiment, the following method is provided, the method comprising means for decoding a coded slice of a coded picture, said means for decoding comprising:
A first position in the first syntax element set and a second position in the second syntax element set used to decode the encoded slice into one of the slice headers of the slice group parameter set Means for identifying;
Means for decoding the first syntax element set and the second syntax element set to be used for decoding the coded slice, and decoding the first and second sets Means:
* Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and in response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from the third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first slice group parameter set Means for decoding the syntax elements of the set;
* Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from the fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second slice group parameter set Means for decoding the syntax elements of the set;
* Means for decoding the coded slice using the decoded first and second syntax element sets.

Claims (20)

A method comprising encoding an uncompressed picture into an encoded picture that includes a slice, said encoding comprising:
Classifying the syntax elements for the slice into a first set and a second set;
Determining syntax element values for the first set and the second set;
Selectively encoding the first set in a first slice group parameter set or slice header, wherein the encoding is:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
Encoding the first set of syntax element values;
The encoding, including any one of omitting both;
Selectively encoding the second set in a second slice group parameter set or slice header, wherein the encoding is:
Providing indication information of inclusion of the associated second set from another slice group parameter set;
Encoding the second set of syntax element values;
The method comprising the encoding, comprising any one of omitting both.   The method of claim 1, comprising estimating the content or instance of the slice group parameter set from other syntax structures that have been encoded or decoded or already in the bitstream.   The method according to claim 1 or 2, comprising forming the slice group parameter set from a slice header of a texture view component of a base view.   The syntax structure of the slice group parameter set is included in a network abstraction layer unit sequence for a specific access unit, the sequence in decoding order or bitstream order, and the slice group parameter set from the appearance position to the access 4. A method according to any of claims 1 to 3, which is valid until the end of the unit.   Determining whether the encoded subset of at least one subset of syntax element values in the slice header is the same as in a subsequent slice header; and if so, said slice in the bitstream 5. A method according to any preceding claim, comprising encoding a group parameter set.   Using a specific slice group parameter set identification value for the first slice group parameter set, and if the first slice group parameter set is not referenced by either a slice header or a slice group parameter set, 6. A method according to any of claims 1 to 5, comprising using the slice group parameter set identification value for two slice group parameter sets.   7. The syntax structure of the slice group parameter set, comprising identifying the syntax structure of the slice group parameter set using a bitstream order of the syntax structure of the slice group parameter set and a predetermined numbering scheme. Method. The method according to any of claims 1 to 7, comprising forming a syntax element set for the slice group parameter set from at least one of the following:
A syntax element indicating a scalable layer and / or other scalable characteristics;
A syntax element indicating a view and / or other multi-view characteristics;
Syntax elements associated with specific component types of multiview video;
Syntax elements associated with access unit identification information;
Syntax elements associated with the decoding order;
Syntax elements related to output order;
Syntax elements associated with other syntax elements that are invariant to all slices of the access unit;
A syntax element that is invariant across all slices of the view component;
Syntax elements related to reference picture list changes;
Syntax elements associated with the set of reference pictures used;
Syntax elements associated with decoding reference picture markings;
Syntax elements associated with the prediction weight table for weighted prediction;
A syntax element that controls deblocking filtering;
A syntax element that controls adaptive loop filtering;
A syntax element that controls the sample adaptive offset. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code together with the at least one processor in the apparatus:
Classifying syntax elements for slices included in the encoded picture into a first set and a second set;
Determining syntax element values for the first set and the second set;
Selectively encoding the first set in a first slice group parameter set or slice header, wherein the encoding is:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
Encoding the first set of syntax element values;
The encoding, including any one of omitting both;
Selectively encoding the second set in a second slice group parameter set or slice header, wherein the encoding is:
Providing indication information of inclusion of the associated second set from another slice group parameter set;
Encoding the second set of syntax element values;
An apparatus configured to perform the encoding, including any one of omitting both. A computer program product comprising one or more sequences of one or more instructions, when executed by one or more processors, the device at least:
Classifying syntax elements for slices included in the encoded picture into a first set and a second set;
Determining syntax element values for the first set and the second set;
Selectively encoding the first set in a first slice group parameter set or slice header, wherein the encoding is:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
Encoding the first set of syntax element values;
The encoding, including any one of omitting both;
Selectively encoding the second set in a second slice group parameter set or slice header, wherein the encoding is:
Providing indication information of inclusion of the associated second set from another slice group parameter set;
Encoding the second set of syntax element values;
A computer program product for performing the encoding, including any one of omitting both. Means for classifying syntax elements for slices included in the coded picture into a first set and a second set;
Means for determining syntax element values for the first set and the second set;
Means for selectively encoding the first set in a first slice group parameter set or slice header, comprising:
Providing indication information of inclusion of the associated first set from another slice group parameter set;
Encoding the first set of syntax element values;
Means for encoding, including any one of omitting both;
Means for selectively encoding the second set in a second slice group parameter set or slice header, comprising:
Providing indication information of inclusion of the associated second set from another slice group parameter set;
Encoding the second set of syntax element values;
An apparatus comprising the means for encoding, including any one of omitting both. Decoding a coded slice of a coded picture;
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set. To identify;
Decoding the first and second sets as performed for decoding the coded slice is performed by decoding the first syntax element set and the second syntax element set The thing is:
Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and as a response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from a third slice group parameter set;
Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from a fourth slice group parameter set;
Decoding the encoded slice using the decoded first syntax element set and second syntax element set. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code together with the at least one processor in the apparatus
Configured to decode an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set. To identify;
Decoding the first and second sets as performed for decoding the coded slice is performed by decoding the first syntax element set and the second syntax element set The thing is:
Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and as a response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from a third slice group parameter set;
Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from a fourth slice group parameter set;
An apparatus comprising: decoding the encoded slice using the decoded first syntax element set and second syntax element set. A computer program product comprising one or more sequences of one or more instructions, when executed by one or more processors, at least on a device,
Decoding an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set. To identify;
Decoding the first and second sets as performed for decoding the coded slice is performed by decoding the first syntax element set and the second syntax element set The thing is:
Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and as a response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from a third slice group parameter set;
Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from a fourth slice group parameter set;
A computer program product comprising decoding the encoded slice using the decoded first syntax element set and second syntax element set. A method comprising decoding a coded slice of a coded picture, said decoding comprising:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set. To identify;
Decoding the first syntax element set and the second syntax element set as used for decoding the encoded slice, and decoding the first and second sets. Is:
Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and as a response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from a third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first set Decoding the syntax elements of
Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from a fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second set Decoding the syntax elements of
Decoding the encoded slice using the decoded first syntax element set and second syntax element set.   16. The method of claim 15, comprising estimating the content or instance of the slice group parameter set from other syntax structures that are encoded or decoded or that are already in the bitstream.   The syntax structure of the slice group parameter set is included in a network abstraction layer unit sequence for a specific access unit, the sequence in decoding order or bitstream order, and the slice group parameter set from the appearance position to the access 17. A method according to any of claims 15 to 16, which is valid until the end of the unit. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code together with the at least one processor in the apparatus:
Decoding an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set. To identify;
Decoding the first syntax element set and the second syntax element set as used for decoding the encoded slice, and decoding the first and second sets. Is:
Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and as a response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from a third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first set Decoding the syntax elements of
Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from a fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second set Decoding the syntax elements of
An apparatus comprising: decoding the encoded slice using the decoded first syntax element set and second syntax element set. A computer program product comprising one or more sequences of one or more instructions, when executed by one or more processors, at least on a device,
Decoding an encoded slice of an encoded picture, wherein the decoding comprises:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set. To identify;
Decoding the first syntax element set and the second syntax element set as used for decoding the encoded slice, and decoding the first and second sets. Is:
Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and as a response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from a third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first set Decoding the syntax elements of
Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from a fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second set Decoding the syntax elements of
A computer program product comprising decoding the encoded slice using the decoded first syntax element set and second syntax element set. A method comprising means for decoding a coded slice of a coded picture, said means for decoding comprising:
A first position in a first syntax element set and a second position in a second syntax element set used to decode the encoded slice into either a slice header or a slice group parameter set. Means to identify;
Means for decoding the first syntax element set and the second syntax element set as used for decoding the encoded slice, and means for decoding the first and second sets Is:
Decoding first indication information indicating inclusion of the associated first set from a third slice group parameter set, and as a response to the first indication information indicating the third slice group parameter set, Decoding the associated first syntax element set from a third slice group parameter set, or if the first indication information does not indicate the third slice group parameter set, the first set Means for decoding syntax elements of
Decoding second indication information indicating inclusion of the related second set from a fourth slice group parameter set, and in response to the second indication information indicating the fourth slice group parameter set, Decoding the associated second syntax element set from a fourth slice group parameter set, or if the second indication information does not indicate the fourth slice group parameter set, the second set Means for decoding syntax elements of
A method comprising means for decoding the encoded slice using the decoded first syntax element set and second syntax element set.