JP2009543501A - Method and system for integrating multiple layers in a multi-layer bitstream - Google Patents

Abstract

  Embodiments of the present invention include systems and methods for managing and integrating multiple layers in a multi-layer bitstream.

Description

  Embodiments of the present invention include methods and systems for processing and processing management in multi-layer bitstreams. The present invention particularly relates to 1) a method and system for combining multiple layers in a multi-layer bitstream, and 2) conditional transform-domain residual accumulation. Method and system for 3) Method and system for scaling of residual layer 4) Method and system for image processing control based on characteristics of neighboring blocks 5) Maintenance of coded block pattern information and It relates to a method and system for use and 6) a method and system for selection and management of transformations.

  To reduce the bit rate of the encoder output, the scalable bitstream can include a form of inter-layer prediction. A typical system is the AVC | H. Includes inter-layer prediction in scalable video extensions for the H.264 video coding standard. The above extension is commonly known as SVC and the SVC system is T. Wiegand, G. Sullivan, J. Reichel, H. Schwarz and M. Wien, "Joint Draft 9 of SVC amendment (revision 2)", JVT-V201. , Marrakech, Morocco, January 13-19, 2007. In the SVC system, inter-layer prediction is realized by projecting motion and mode information from an enumerated lower layer to an enumerated upper layer. In addition, prediction residuals are projected from the enumerated lower layers to the enumerated upper layers. Thus, the above upper layer bitstream can include additional residuals to improve the quality of the decoded output.

  According to a first aspect of the present invention, there is provided a method for integrating a plurality of layers in a multi-layer bitstream, comprising: a) dequantizing the quantized transform coefficients of the first layer, whereby the first Creating a layer transform coefficient; b) scaling the first layer transform coefficient to match the characteristics of the second layer, thereby creating a scaled transform coefficient for the first layer C) dequantizing the quantized transform coefficients of the second layer, thereby creating second layer transform coefficients; and d) converting the scaled transform coefficients of the first layer to those of the second layer. A method is provided that includes integrating with the transform coefficients to form an integrated coefficient.

  The first layer may be a base layer.

  Dequantizing the quantized transform coefficients of the first layer may include using a first quantization parameter, and dequantizing the quantized transform coefficients of the second layer May include using a second quantization parameter.

  The first layer and the second layer may have different spatial resolutions.

  The second layer may be an enhancement layer.

  The method may further include the step of inverse transforming the integrated coefficients, thereby generating a spatial domain residual value.

  The method may further include integrating the spatial region residual value with the spatial region predicted value.

  The method comprises the steps of: a) dequantizing the third layer quantized transform coefficients, thereby creating third layer transform coefficients; b) matching the integrated coefficients with the characteristics of the third layer And c) further comprising the steps of: creating an integrated and scaled coefficient; and c) integrating the integrated and scaled coefficient with the third layer transform coefficient. Good.

  The method may further include generating an integrated bitstream that includes the integrated coefficients.

  The integrated bitstream may further include an intra prediction mode.

  The integrated bitstream may further include a motion vector.

  According to a second aspect of the present invention, a system for integrating a plurality of layers in a multi-layer bitstream comprising: a) dequantizing the quantized transform coefficients of the first layer, thereby A first dequantizer for creating first layer transform coefficients; b) scaling the first layer transform coefficients to match the characteristics of the second layer, thereby scaling the first layer A scaler for generating the transformed transform coefficients; c) a second inverse quantizer for dequantizing the second layer quantized transform coefficients, thereby creating a second layer transform coefficient; and d) A system is provided that includes a coefficient integrator for integrating the scaled transform coefficients of the first layer with the transform coefficients of the second layer to form an integrated coefficient.

  The system may further include a bitstream generator for generating an integrated bitstream that includes the integrated coefficients.

  The system inversely transforms the integrated coefficients, thereby generating a spatial domain residual value, and a first unit for integrating the spatial domain residual value with the spatial domain prediction value. 2 integrators may be further included.

  According to a third aspect of the present invention, there is provided a method for converting an SVC compliant bitstream into AVC compliant data, comprising a) prediction data, base layer residual data, and enhancement layer residual. Receiving a SVC-compliant bitstream including data; b) dequantizing the base layer residual data, thereby creating base layer transform coefficients; c) reversing the enhancement layer residual data Quantizing, thereby creating enhancement layer transform coefficients; d) scaling the base layer transform coefficients to match the enhancement layer quantization characteristics, thereby reducing the base layer scaled transform coefficients And e) integrating the scaled transform coefficients of the base layer with the transform coefficients of the enhancement layer. Method comprising the step of forming the coefficients is provided.

  The method may further include integrating the integrated coefficients with prediction data to form an AVC compliant bitstream.

  The prediction data may include an intra prediction mode index.

  The prediction data may include a motion vector.

  The method may further include the step of inverse transforming the integrated coefficients, thereby creating a spatial domain residual value.

  The method may further include obtaining a spatial region prediction value and integrating the spatial region prediction value with the spatial region residual value to form a decoded image.

  According to a fourth aspect of the present invention, there is provided a method for integrating a plurality of layers in a multi-layer bitstream, comprising: a) receiving a quantized transform coefficient of a first layer; b) a first layer Scaling the quantized transform coefficients of the second layer to match the characteristics of the second layer, thereby creating a quantized and scaled transform coefficient of the first layer; c) the quantized second layer Receiving a transform coefficient; and d) integrating the scaled transform coefficient of the first layer with the quantized transform coefficient of the second layer to form a quantized and consolidated coefficient. Is done.

  The method further includes the step of dequantizing the quantized and combined coefficients to generate the combined coefficients.

  The method further includes the step of inverse transforming the combined coefficients, thereby generating a spatial domain residual value.

  The method further includes integrating the spatial domain residual value with a spatial domain predicted value.

  The method further includes generating an integrated bitstream that includes quantized and integrated coefficients.

  According to a fifth aspect of the present invention, there is provided a method for conditionally integrating a plurality of layers in a multi-layer bitstream comprising the steps of: a) receiving a quantized transform coefficient of a first layer; b) Receiving a quantized transform coefficient of the second layer; c) receiving a layer integration indicator; d) a quantized transform of the first layer if the layer integration indicator indicates a transform domain accumulation Scaling the coefficients to match the characteristics of the second layer, thereby creating a quantized and scaled transform coefficient for the first layer; and e) the layer integration indicator indicates transform domain accumulation Integrating the scaled transform coefficients of the first layer with the quantized transform coefficients of the second layer, if any, to form quantized and consolidated coefficients The method comprising is provided.

  The layer integration indicator may be acquired from data in the bit stream of the second layer.

  The method may further include the step of invalidating the smoothed reference prediction when the layer integration indicator indicates transformation region accumulation.

  According to a sixth aspect of the present invention, there is provided a method for reconstructing an enhancement layer from a multi-layer bitstream comprising: a) receiving a first layer intra prediction mode; b) a first layer prediction mode being Receiving a second layer bitstream prediction indicator indicating that it is used for second layer prediction; c) using a first layer prediction mode, based on neighboring block data in the second layer; And d) integrating the second layer prediction with residual information, thereby creating a reconstructed second layer.

  According to a seventh aspect of the present invention, there is provided a method for integrating a plurality of layers in a multi-layer bitstream, comprising: a) determining a first spatial resolution of a first layer of a multi-layer image; b ) Determining a second spatial resolution of the second layer of the multi-layer image; c) comparing the first spatial resolution with the second spatial resolution; d) the first spatial resolution is Performing steps e) -f) if substantially equal to the second spatial resolution; e) scaling the transform coefficients of the first layer to match the characteristics of the second layer; Creating a layer of scaled transform coefficients; f) integrating the scaled transform coefficients of the first layer with the transform coefficients of the second layer to form a consolidated coefficient. G) performing steps h) to k) when the spatial resolution of the first layer is not substantially equal to the spatial resolution of the second layer; h) inverse transforming the transform coefficients of the first layer; Thereby generating a spatial region value of the first layer; i) inverse transforming the transform coefficients of the second layer, thereby generating a spatial region value of the second layer; j) a spatial region value of the first layer Scaling to match the resolution of the second layer, thereby generating a scaled spatial domain value of the first layer; and k) scaling the scaled spatial domain value of the first layer to that of the second layer A method is provided that includes integrating with a spatial domain value, thereby generating an integrated spatial domain residual value.

  The characteristic may include a quantization parameter.

  The transform coefficient may be a de-quantized transform coefficient.

  The transform coefficient may be a quantized transform coefficient, and the transform coefficient may be inversely quantized before the inverse transform.

  The scaling step comprises: a) determining a first layer quantization parameter; b) determining a second layer quantization parameter; and c) a first layer quantization parameter and a second layer quantum. Scaling the transform coefficients of the first layer based on the conversion parameter.

  The method further includes the step of inverse transforming the combined coefficients and thereby generating a spatial domain residual value when the first spatial resolution is substantially equal to the second spatial resolution. Also good.

  The method may further include integrating the spatial domain residual value with a spatial domain prediction value when the first spatial resolution is substantially equal to the second spatial resolution.

  The method integrates an integrated spatial domain residual value with a spatial domain prediction value when the first spatial resolution is not substantially equal to the second spatial resolution, thereby integrating the integrated space. The method may further include creating a region value.

  The method may further include generating an integrated bitstream that includes the integrated coefficients when the first spatial resolution is substantially equal to the second spatial resolution.

  The integrated bitstream may further include an intra prediction mode when the first spatial resolution is substantially equal to the second spatial resolution.

  The integrated bitstream may further include a motion vector if the first spatial resolution is substantially equal to the second spatial resolution.

  The method includes the step of transforming an integrated spatial domain value if the first spatial resolution is not substantially equal to the second spatial resolution, thereby creating an integrated transformed domain coefficient. Further, it may be included.

  The method may further include generating an integrated bitstream that includes the integrated transform domain coefficients when the first spatial resolution is not substantially equal to the second spatial resolution. Good. .

  The integrated bitstream may further include an intra prediction mode if the first spatial resolution is not substantially equal to the second spatial resolution.

  The integrated bitstream may further include a motion vector if the first spatial resolution is not substantially equal to the second spatial resolution.

  According to an eighth aspect of the present invention, there is a system for integrating a plurality of layers in a multi-layer bitstream, comprising: a) determining a first spatial resolution of a first layer of a multi-layer image; A resolution determiner for determining a second spatial resolution of a second layer of the multi-layer image; b) a comparator for comparing the first spatial resolution with the second spatial resolution; c) above A controller for performing steps e) to f) when the first spatial resolution is substantially equal to the second spatial resolution; d) transformation of the first layer to be consistent with the characteristics of the second layer A coefficient scaler for scaling the coefficients, thereby creating scaled transform coefficients for the first layer; e) the scaled transform coefficients for the first layer with the transform coefficients for the second layer And a coefficient integrator for forming an integrated coefficient; f) selectively steps g) to i) if the spatial resolution of the first layer is not substantially equal to the spatial resolution of the second layer. G) inverse transforming the first layer transform coefficients, thereby generating the first layer spatial domain values and inverse transforming the second layer transform coefficients, thereby creating the second layer space An inverse transformer for generating region values; h) scaling the spatial region values of the first layer to match the resolution of the second layer, thereby generating the scaled spatial region values of the first layer A spatial domain scaler for; and i) integrating the scaled spatial domain value of the first layer with the spatial domain value of the second layer, thereby generating an integrated spatial domain residual value System including a region integrating unit is provided.

  The inverse transformer may further inverse transform the integrated coefficients when the first spatial resolution is substantially equal to the second spatial resolution, thereby generating a spatial domain residual value. Good.

  The spatial domain integrator may further integrate the spatial domain residual value with the spatial domain prediction value when the first spatial resolution is substantially equal to the second spatial resolution.

  The spatial domain integrator further integrates an integrated spatial domain residual value with a spatial domain prediction value when the first spatial resolution is not substantially equal to the second spatial resolution, thereby An integrated spatial domain value may be created.

  The system further includes a bitstream generator for generating an integrated bitstream that includes integrated coefficients when the first spatial resolution is substantially equal to the second spatial resolution. May be.

  If the first spatial resolution is substantially equal to the second spatial resolution, the integrated bitstream may further include an intra prediction mode.

  The integrated bitstream may further include a motion vector if the first spatial resolution is substantially equal to the second spatial resolution.

  The system transforms an integrated spatial domain value if the first spatial resolution is not substantially equal to a second spatial resolution, thereby creating an integrated transformed domain coefficient A vessel may further be included.

  The system includes a bitstream generator for generating an integrated bitstream that includes integrated transform domain coefficients when the first spatial resolution is not substantially equal to the second spatial resolution. Further, it may be included.

  The integrated bitstream may further include an intra prediction mode if the first spatial resolution is not substantially equal to the second spatial resolution.

  The integrated bitstream may further include a motion vector if the first spatial resolution is not substantially equal to the second spatial resolution.

  According to a ninth aspect of the present invention, there is a method for integrating a plurality of layers in a multi-layer bitstream comprising: a) a non-quantized transform for a first layer having a first spatial resolution Receiving a coefficient; b) receiving an unquantized transform coefficient for a second layer having a first spatial resolution; c) scaling the transform coefficient of the first layer, thereby Creating a scaled transform coefficient; d) integrating the scaled transform coefficient of the first layer with the transform coefficient of the second layer, thereby creating an integrated transform coefficient; e) integrated Inverse transforming transform coefficients, thereby creating integrated spatial-domain residual values; f) for a third layer having a second spatial resolution Receiving a non-quantized transform coefficient; g) re-sampling the integrated spatial domain residual value to a second spatial resolution, thereby creating an integrated re-sampled spatial domain value; h) Inverse transforming the third layer transform coefficients, thereby creating a third layer spatial domain value; and i) integrating the integrated and resampled spatial domain value with the spatial domain value of the third layer. A method of including is provided.

  According to a tenth aspect of the present invention, there is provided a method for integrating a plurality of layers in a multi-layer bitstream, comprising: a) a quantized transform coefficient for a first layer having a first spatial resolution B) receiving the quantized transform coefficients for the second layer having the first spatial resolution; c) scaling the quantized transform coefficients of the first layer, whereby the first Creating a quantized scaled transform coefficient of the layer; d) integrating the quantized scaled transform coefficient of the first layer with the quantized transform coefficient of the second layer, thereby quantizing Creating an integrated transform coefficient; e) dequantizing the quantized and integrated transform coefficient, thereby creating an integrated transform coefficient; f) Inverse transforming the combined transform coefficients, thereby creating an integrated residual spatial domain value; g) receiving quantized transform coefficients for a third layer having a second spatial resolution H) resampling the integrated residual spatial domain value to a second spatial resolution, thereby creating an integrated resampled spatial domain value; i) a third layer quantized transform; Dequantizing the coefficients, thereby creating third layer transform coefficients; j) inverse transforming the third layer transform coefficients, thereby creating the third layer spatial domain values; and k) integrating There is provided a method comprising integrating the resampled spatial domain values with the spatial domain values of the third layer.

  According to an eleventh aspect of the present invention, a method for integrating a plurality of layers in a multi-layer bitstream, comprising: a) a non-quantized transform for a first layer having a first spatial resolution Receiving a coefficient; b) inverse transforming the unquantized transform coefficient of the first layer, thereby generating a spatial domain value of the first layer; c) a second higher than the first spatial resolution Receiving a non-quantized transform coefficient for a second layer having a spatial resolution; d) receiving a non-quantized transform coefficient for a third layer having a second spatial resolution; Up-sampling the spatial domain value of one layer to a second spatial resolution, thereby generating the up-sampled spatial domain value of the first layer; f) the unquantized second layer Integrating the transform coefficients with the non-quantized transform coefficients of the third layer, thereby creating an integrated transform coefficient; g) inverse transforming the merged transform coefficient and thereby the first integrated And h) integrating the up-sampled spatial region value of the first layer with the first integrated spatial region residual value.

  According to a twelfth aspect of the present invention, a method for integrating a plurality of layers in a multi-layer bitstream, comprising: a) a quantized transform coefficient for a first layer having a first spatial resolution B) receiving a quantized transform coefficient for the second layer having the first spatial resolution; c) a quantized transform coefficient for the third layer having the first spatial resolution. D) scaling the quantized transform coefficients of the first layer to match the properties of the second layer, thereby creating a quantized and scaled transform coefficient of the first layer E) integrating the quantized scaled transform coefficients of the first layer with the quantized transform coefficients of the second layer, thereby obtaining the quantized and unified transform coefficients F) dequantizing the quantized and integrated transform coefficients, thereby creating an integrated transform coefficient; g) dequantizing the third layer quantized transform coefficients; Creating a third layer non-quantized transform coefficient by: h) integrating the integrated transform coefficient with the third layer non-quantized transform coefficient, thereby integrating the three layer integrated transform A method is provided comprising: creating coefficients; and i) inverse transforming the combined transform coefficients of the three layers, thereby creating an integrated spatial domain value.

  According to a thirteenth aspect of the present invention, there is provided a method for integrating a plurality of layers in a multi-layer bitstream, i) determining whether a second layer of a multi-layer image uses residual prediction. Step; ii) performing the subsequent steps only if the second layer uses residual prediction; iii) determining the first spatial resolution of the first layer of the multi-layer image; iv) Determining a second spatial resolution of the second layer; v) comparing the first spatial resolution with the second spatial resolution; vi) the first spatial resolution substantially equal to the second spatial resolution; Vii) scaling the first layer transform coefficients to match the characteristics of the second layer, and thereby the first layer scaling. Creating a ringed transform coefficient; viii) integrating the scaled transform coefficients of the first layer with the transform coefficients of the second layer to form a consolidated coefficient; ix) spatial resolution of the first layer Performing steps x) to xiii) if x is not substantially equal to the spatial resolution of the second layer; x) inverse transforming the transform coefficients of the first layer, thereby converting the spatial domain value of the first layer Generating; xi) inverse transforming the transform coefficients of the second layer, thereby generating a spatial region value of the second layer; xii) matching the spatial region value of the first layer with the resolution of the second layer Generating a first layer scaled spatial domain value; and xiii) a first layer scaled spatial domain value to a second layer spatial A method is provided that includes the step of integrating with region values thereby generating an integrated spatial region value.

  According to a fourteenth aspect of the present invention, there is provided a method for scaling transform coefficients in a multi-layer bitstream, the step of determining a first-layer quantization parameter based on the multi-layer bitstream; Determining a second layer quantization parameter based on the multi-layer bitstream; and scaling the first layer transform coefficients based on the first layer quantization parameter and the second layer quantization parameter. A method of including is provided.

  The scaling may be performed according to the following relationship.

(Here, TSecondLayer and TFirstLayer are transform coefficients in the second layer and the first layer, respectively, k is an integer, and Qp_FirstLayer and Qp_SecondLayer are quantization parameters for the first layer and the second layer, respectively)
k may be equal to 6.

  The scaling may be performed according to the following relationship.

(Where // indicates integer division,% indicates modulo operation, M and ScaleMatrix are constants, TSecondLayer and TFirstLayer indicate transform coefficients in the second layer and the first layer, respectively, and k is an integer. Qp_FirstLayer and Qp_SecondLayer are quantization parameters for the first layer and the second layer, respectively)
If Qp_Diff is known to be less than 0, Qp_Diff may be reset to 0.

  ScaleMatrix may be equal to [512 573 642 719 806 902] and M may be equal to 512.

  ScaleMatrix may be equal to [8 9 10 11 13 14] and M may be equal to 8.

  The scaling may be performed according to the following relationship.

(Where // indicates integer division,% indicates modulo operation, M and ScaleMatrix are constants, TSecondLayer and TFirstLayer indicate transform coefficients in the second layer and the first layer, respectively, and k is an integer. Qp_FirstLayer and Qp_SecondLayer are quantization parameters for the first layer and the second layer, respectively)
If Qp_Diff is known to be less than 0, Qp_Diff may be reset to 0.

  ScaleMatrix may be equal to [291 325 364 408 457 512 573 642 719 806 902] and M may be equal to 512.

  The method may further include the step of integrating the scaled first layer transform coefficients with the second layer transform coefficients, thereby creating an integrated coefficient.

  The method may further include generating an integrated bitstream that includes the integrated coefficients.

The method further comprises determining the weighting factor S _F which depends on the conversion coefficient of the first layer and the weighting coefficient S _S which depends on the conversion coefficient of the second layer, the scaling is performed according to the relation It may be a thing.

(Here, TSecondLayer and TFirstLayer are transform coefficients in the second layer and the first layer, respectively, k is an integer, and Qp_FirstLayer and Qp_SecondLayer are quantization parameters for the first layer and the second layer, respectively)
The above method comprises determining the weighting factor S _F which depends on the conversion coefficient of the first layer; further comprising the step of determining the weighting factor S _S which depends on the transform coefficients and the second layer, the scaling, the following It may be performed according to the relationship.

(Where // indicates integer division,% indicates modulo operation, M and ScaleMatrix are constants, TSecondLayer and TFirstLayer respectively mean transform coefficients in the second layer and the first layer, and k is an integer. Qp_FirstLayer and Qp_SecondLayer are quantization parameters for the first layer and the second layer, respectively)
S _F and S _S may be explicitly present in the multi-layer bitstream.

S _F and S _S may be calculated from the multi-layer bitstream.

  The method further includes determining a weighting factor W1 that is multiplicative and depends on the transform coefficient, and determining a weighting factor W2 that is additive and depends on the transform coefficient, wherein the scaling is It may be performed according to the following relationship.

(Where // indicates integer division,% indicates modulo operation, M and ScaleMatrix are constants, TSecondLayer and TFirstLayer indicate transform coefficients in the second layer and the first layer, respectively, and k is an integer. Qp_FirstLayer and Qp_SecondLayer are quantization parameters for the first layer and the second layer, respectively)
W1 and W2 may be explicitly present in the multi-layer bitstream.

  W1 and W2 may be calculated from the multi-layer bitstream.

  The method may further include the step of dequantizing the combined coefficients, thereby generating combined and unquantized transform coefficients.

  The method may further include the step of inverse transforming the combined, unquantized transform coefficients, thereby generating a spatial domain residual value.

  The method may further include integrating the spatial domain residual value with a spatial domain prediction value.

  According to a fifteenth aspect of the present invention, there is provided a system for scaling transform coefficients in a multi-layer bitstream for determining a first-layer quantization parameter based on the multi-layer bitstream. A first parameter determiner; a second parameter determiner for determining a second layer quantization parameter based on the multi-layer bitstream; and a first layer quantization parameter and a second layer quantum A system is provided that includes a scaler for scaling the first layer transform coefficients based on the quantization parameter.

  According to a sixteenth aspect of the present invention, there is provided a method for controlling an entropy encoding process comprising the steps of: a) identifying a first neighboring macroblock adjacent to a target macroblock; Identifying a second neighboring macroblock adjacent to the macroblock; c) determining a first macroblock index indicating whether the first neighboring macroblock is encoded with reference to another layer; d) determining a second macroblock index indicating whether the second neighboring macroblock is encoded with reference to another layer; and e) a first macroblock index and a second macroblock index. A method is provided that includes determining an entropy encoding control value based on.

  The method may further include a step of encoding an intra prediction mode using the entropy encoding control value.

  The method may further include a step of decoding an intra prediction mode using the entropy encoding control value.

  The method may further include a step of encoding the target macroblock using the entropy encoding control value.

  The method may further include the step of decoding the target macroblock using the entropy encoding control value.

  The target macroblock may be a color difference macroblock.

  A macroblock may be determined to be encoded with reference to another layer when the macroblock is of type IntraBL.

  The entropy encoding control value may include a context.

  The context may be based on cumulative macroblock information.

  According to a seventeenth aspect of the present invention, there is provided a method for controlling an entropy encoding process comprising: a) identifying a first neighboring macroblock adjacent to a target macroblock; b) Identifying a neighboring second neighboring macroblock; c) determining whether the first neighboring macroblock is available; d) coding the first neighboring macroblock in inter-layer prediction mode E) determining whether the first neighboring macroblock is encoded in the spatial domain; f) whether the first neighboring macroblock is intra predicted in DC prediction mode G) determining whether the first neighboring macroblock is encoded with reference to another layer h) if any of steps c) to g) is true, setting a first adjacent block flag to 1; i) determining whether the second adjacent macroblock is available; j ) Determining whether the second adjacent macroblock is encoded in inter-layer prediction mode; k) determining whether the second adjacent macroblock is encoded in the spatial domain; l) above Determining whether the second neighboring macroblock is intra predicted in DC prediction mode; m) determining whether the second neighboring macroblock is encoded with reference to another layer; n) If any of steps i) to m) is true, setting a second adjacent block flag value to 1; and o) above the first adjacent block flag value Comprising the step of generating an entropy encoding control value by adding the second adjacent block flag value is provided.

  The method may further include a step of encoding the target macroblock using the entropy encoding control value.

  The method may further include the step of decoding the target macroblock using the entropy encoding control value.

  The target macroblock may be a color difference macroblock.

  A macroblock may be determined to be encoded with reference to another layer when the macroblock is of type IntraBL.

  A macroblock may be determined to be encoded in the spatial domain when the macroblock is type I_PCM.

  The entropy encoding control value may include a context.

  The context may be based on cumulative macroblock information.

  According to an eighteenth aspect of the present invention, there is provided a method for determining a prediction mode, comprising a) identifying a first adjacent macroblock adjacent to a target macroblock; b) adjacent to the target macroblock. Identifying a second neighboring macroblock; and c) the following conditions i) to vi): i) the first neighboring macroblock is available; ii) the first neighboring macroblock is in inter-layer prediction mode Iii) the first neighboring macroblock is coded with reference to another layer; iv) the second neighboring macroblock is available; v) the second neighboring macro The block is coded in inter-layer prediction mode; vi) the second neighboring macroblock is coded with reference to another layer; if either is true, the target block is estimated (est imated) a method is provided that includes setting the prediction mode to a predetermined mode.

  The predetermined mode may be a DC prediction mode.

  The method may further include the step of specifying the actual prediction mode of the target block based on the content of the target block.

  The method may further include comparing the estimated prediction mode with the actual prediction mode.

  The method is a step of encoding a message, wherein the decoder predicts a target block using the estimated prediction mode when the message is the same as the estimated prediction mode. The method may further include a step that instructs the above.

  The method is a step of decoding a message, wherein if the message is the same as the estimated prediction mode, the decoder is used to predict the target block using the estimated prediction mode. It may further include a step that is an instruction.

  The method is a step of decoding a message, instructing the decoder to predict the target block using the actual prediction mode if the message is not the same as the estimated prediction mode. It may further include a step that is a thing.

  The method is a step of encoding a message, instructing the decoder to predict the target block using the actual prediction mode when the message is not the same as the estimated prediction mode. It may further include a step that is to be performed.

  The estimated prediction mode of the target block may be a luminance prediction mode.

  According to a nineteenth aspect of the present invention, there is provided a system for controlling an entropy encoding process, wherein a) a first identifier for identifying a first neighboring macroblock adjacent to a target macroblock; b ) A second identifier for identifying a second adjacent macroblock adjacent to the target macroblock; c) a first macro indicating whether the first adjacent macroblock is encoded with reference to another layer A first index determiner for determining a block index; d) a second index for determining a second macroblock index indicating whether the second neighboring macroblock is coded with reference to another layer. And e) a system including a value determiner for determining an entropy coding control value based on the first macroblock index and the second macroblock index. It is.

  According to a twentieth aspect of the present invention, there is provided a method for integrating a plurality of layers in a multi-layer bitstream, comprising: a) a bitstream including encoded image coefficients and encoded block pattern (Cbp) information A step of receiving a bitstream identifying an area in which the Cbp information includes transform coefficients in the bitstream; b) a step of decoding the Cbp information; c) a bitstream area including transform coefficients using the Cbp information Analyzing the bitstream by identifying; d) scaling the transform coefficients of the first layer in the bitstream to match the characteristics of the second layer in the bitstream; e) of the first layer The scaled transform coefficients are added to the second layer transform coefficients to create an integrated ray Forming integrated coefficients therein; and f) calculating integrated Cbp information for the integrated layer, which is information identifying regions containing transform coefficients in the integrated layer. A method is provided.

  The above calculation may be performed only when the coefficients in the second layer are predicted from the first layer.

  The method may further include the step of inverse transforming the combined coefficients, thereby creating a spatial domain value.

  The method may further include selectively filtering regions of spatial region values based on the integrated Cbp information.

  The first layer and the second layer may have different spatial resolutions.

  The first layer and the second layer may have different bit depths.

  The first layer may be a basic layer.

  The second layer may be an enhancement layer.

  The step of calculating the integrated Cbp information may include a step of testing the integrated coefficient.

  The step of calculating the integrated Cbp information may include a step of calculating a binary OR of the first layer and the second layer.

  The step of calculating the integrated Cbp information may include scanning a coefficient list to identify regions having residual information.

  According to a twenty-first aspect of the present invention, there is provided a system for integrating a plurality of layers in a multi-layer bitstream, including g) encoded image coefficients and encoded block pattern (Cbp) information. A receiver for receiving a bitstream, wherein the Cbp information identifies an area including a transform coefficient in the bitstream; h) a decoder for decoding the Cbp information; i) Cbp information A parser for analyzing the bitstream by using to identify the bitstream region containing the transform coefficients; j) matching the transform coefficients of the first layer in the bitstream with the characteristics of the second layer in the bitstream K) the scaled transform coefficients of the first layer to the second layer An adder for adding to the transform coefficients to form a merged coefficient in the merged layer; and l) the merged layer, information identifying areas in the merged layer that contain transform coefficients A system including a calculator for calculating integrated Cbp information for is provided.

  The above calculation may be performed only when the coefficients in the second layer are predicted from the first layer.

  The method may further include an inverse transformer for inverse transforming the combined coefficients, thereby creating a spatial domain value.

  The system may further include a filter for selectively filtering regions of the spatial region value based on the integrated Cbp information.

  The first layer and the second layer may have different spatial resolutions.

  The first layer and the second layer may have different bit depths.

  The first layer may be a base layer.

  The second layer may be an enhancement layer.

  The calculation of the integrated Cbp information may include an examination of the integrated coefficient.

  The calculation of the integrated Cbp information may include a binary OR operation between the first layer and the second layer.

  The calculation of the integrated Cbp information may include scanning the coefficient list to identify regions having residual information.

  According to a twenty-second aspect of the present invention, there is provided a method for selecting a reconstructed transform size when the transform size is not indicated in the enhancement layer, and a) determining a transform size of a lower layer B) determining whether the transformation size of the lower layer is substantially similar to the predetermined transformation size; c) the transformation size of the lower layer is substantially similar to the predetermined transformation size; A step of selecting an inverse transform of a predetermined transform size as the reconstruction transform; and d) a default if the transform size of the lower layer is not substantially similar to the predetermined transform size A method is provided that includes selecting an inverse transform of the transform size as the reconstruction transform.

  The default transform size may be used to parse the bitstream regardless of whether it was selected for reconstruction transform.

  The method may further include a step of inversely transforming the enhancement layer coefficients by reconstruction transform.

  The predetermined conversion size may be 8 × 8.

  The predetermined conversion size may be 16 × 16.

  The method performs steps a) -d) only when determining a prediction mode for the enhancement layer and when the enhancement layer prediction mode indicates that the enhancement layer is predicted from a lower layer. A step.

  The method may further include extracting a plurality of enhancement layer coefficients formatted to a default transform size.

  The method may further include reformatting the plurality of extracted coefficients of the enhancement layer to a predetermined transform size.

  The method may further include extracting a plurality of quantized coefficients of the enhancement layer formatted to a default transform size.

  The method further comprises reformatting the plurality of quantized and extracted coefficients of the enhancement layer to a predetermined transform size, thereby creating a plurality of quantized and reformatted coefficients of the enhancement layer. May be included.

  The method includes: i) dequantizing a plurality of quantized transform coefficients in a lower layer, thereby creating a plurality of transform coefficients in the lower layer; ii) converting the plurality of transform coefficients in the lower layer into characteristics of the enhancement layer Scaling to match with, thereby creating a plurality of scaled transform coefficients for the lower layer; iii) dequantizing the quantized and reformatted coefficients of the enhancement layer, thereby transforming the enhancement layer transform coefficients And iv) integrating the scaled transform coefficients of the lower layer with the transform coefficients of the enhancement layer to form an integrated coefficient.

  The method may further include generating an integrated bitstream that includes the integrated coefficients.

  The integrated bitstream may further include an intra prediction mode.

  The integrated bitstream may further include a motion vector.

  The method may further include the step of inverse transforming the combined coefficients using a reconstruction transform, thereby generating a spatial domain residual value.

  The method may further include integrating the spatial region residual value with the spatial region predicted value.

  The method includes i) scaling the plurality of quantized transform coefficients of the lower layer to match the characteristics of the enhancement layer, thereby creating a plurality of quantized scaled transform coefficients of the lower layer; And ii) further comprising integrating the plurality of quantized scaled transform coefficients of the lower layer with the quantized reformatted coefficients of the enhancement layer to form a quantized integrated coefficient Good.

  The method may further include dequantizing the quantized and combined coefficients, thereby creating the integrated coefficients.

  The method may further include the step of generating an integrated bitstream that includes the integrated coefficients.

  The integrated bitstream may further include an intra prediction mode.

  The integrated bitstream may further include a motion vector.

  The method may further include the step of inverse transforming the combined coefficients using a reconstruction transform, thereby generating a spatial domain residual value.

  The method may further include integrating the spatial region residual value with the spatial region predicted value.

  According to a twenty-third aspect of the present invention, there is a system for selecting a reconstructed transform size when the transform size is not indicated in the enhancement layer, a) for determining the transform size of the lower layer B) a determination unit for determining whether the conversion size of the lower layer is substantially similar to the predetermined conversion size; c) the conversion size of the lower layer is set to the predetermined conversion size. A first selector for selecting an inverse transform of a predetermined transform size as a reconstruction transform if substantially similar; and d) the transform size of the lower layer is set to a predetermined transform size A system is provided that includes a second selector for selecting an inverse transform of the default transform size as the reconstruction transform if it is not substantially similar.

  Some embodiments of the present invention include methods and systems for processing and processing management in a multi-layer bitstream.

  The foregoing and other objects, features and advantages of the present invention will be more readily understood when the following detailed description of the invention is considered in conjunction with the accompanying drawings.

FIG. 6 illustrates an embodiment of the present invention that includes scaling of transform domain coefficients. FIG. 3 illustrates an embodiment of the present invention including accumulation of quantized transform coefficients and scaling of quantized transform domain coefficients. FIG. 6 illustrates an embodiment of the present invention including transform domain coefficient scaling and bitstream rewriting without reconfiguration. FIG. 6 illustrates an embodiment of the present invention including accumulation of quantized transform coefficients or transform indices and bitstream rewriting without reconstruction. FIG. 6 illustrates an embodiment of the present invention that includes selection of transform sizes. FIG. 6 illustrates an embodiment of the present invention including conditional transform size indication and selection. FIG. 4 illustrates an embodiment of the present invention including coefficient scaling based on quantization parameters. FIG. 6 illustrates an embodiment of the present invention including calculation of entropy encoder control values based on adjacent macroblock data. It is a figure which shows embodiment of this invention including determination of the entropy encoder control value based on the condition combination of an adjacent macroblock. FIG. 6 illustrates an embodiment of the present invention including determination of an estimated prediction mode and signaling of a prediction mode based on neighboring macroblock data. FIG. 6 illustrates an embodiment of the present invention including calculation of a unified layer coding block pattern. FIG. 6 illustrates an embodiment of the present invention that includes selective transform accumulation based on the spatial resolution of a layer. FIG. 6 is a block diagram illustrating an embodiment of the present invention that includes transform size selection. FIG. 3 is a block diagram illustrating an embodiment of the present invention that includes coefficient scaling based on quantization parameters. FIG. 6 illustrates an embodiment of the present invention including calculation of entropy encoder control values based on adjacent macroblock data. FIG. 6 illustrates an embodiment of the present invention including calculation of a unified layer coding block pattern. FIG. 6 is a block diagram illustrating an embodiment of the present invention that includes selective transform accumulation based on the spatial resolution of the layers.

  Embodiments of the present invention may be best understood with reference to the drawings. Like parts are indicated by like numerals throughout the drawings. Each figure listed above is expressly incorporated as part of this detailed description.

  It will be readily appreciated that the components of the present invention can be configured and designed in a wide variety of different configurations, as generally described herein and schematically illustrated in the drawings. Accordingly, the following more detailed description of the method and system embodiments of the present invention is not intended to limit the scope of the invention, but is merely representative of the presently preferred embodiments of the invention. .

  The components of the embodiments of the present invention can be realized by hardware, firmware, and / or software. The exemplary embodiments disclosed herein are only described for one of these forms, and those skilled in the art will implement these components in any of these forms as long as they are within the scope of the present invention. You will understand what you can do.

  Some embodiments of the present invention include methods and systems for residual accumulation for scalable video coding. Some embodiments include a method and system for decoding a scalable bitstream. The bitstream can be generated by an encoder and subsequently stored and / or transmitted to a decoder. The decoder can analyze the bitstream and convert the analyzed symbols into a sequence of decoded images.

  A scalable bitstream may contain different representations of original image sequences. In one embodiment, the first layer in the bitstream includes a low quality version of the image sequence, and the second layer in the bitstream includes a higher quality version of the image sequence. Yes. In a second example, the first layer in the bitstream includes a low resolution version of the image sequence, and the second layer in the bitstream includes a higher resolution version of the image sequence. Yes. More complex examples will be readily apparent to those skilled in the art. More complex examples may include multiple representations of image sequences and / or streams, including combinations of different qualities and different resolutions.

  To reduce the bit rate of the encoder output, the scalable bitstream can include a form of inter-layer prediction. Exemplary embodiments include AVC | H. It may include inter-layer prediction in scalable video extensions for the H.264 video coding standard. The above extension is commonly known as SVC, and the SVC system is T. Wiegand, G. Sullivan, J. Reichel, H. Schwarz and M. Wien, "Joint Draft 9 of SVC amendment (revision 2)", JVT-V201. , Marrakech, Morroco, January 13-19, 2007. The above references are hereby incorporated by reference. In the SVC system, inter-layer prediction is realized by projecting motion and mode information from the enumerated lower layers to the enumerated upper layers. In addition, prediction residuals are projected from the enumerated lower layers to the enumerated upper layers. Thus, the above upper layer bitstream can include additional residuals to improve the quality of the decoded output.

  ISO / IEC JTC1 / SC29 / WG11 Information Technology-Coding of Audio-Visual Objects-Part 10: Advanced Video Coding, ISO / IEC 14496-10, 2005 is also incorporated herein by reference.

  ITU-T Recommendation H.264: “Advanced video coding for generic audio visual services”, March 2003 is also incorporated herein by reference.

[Rewriting of bit stream from SVC to AVC]
Current SVC systems require transcoding that can support AVC devices at any layer in addition to the base layer. This limits the application range of SVC. Embodiments of the present invention are directed to coarse grain scalable layer syntax and semantics to enable fast rewriting of SVC bitstreams to AVC compliant bitstreams. Includes changes. In some embodiments, the network device can rewrite SVC data into an AVC bitstream without drift and without having to reconstruct the sequence. This can be achieved in some embodiments by merging multiple coarse-grained scalable layers.

  Some embodiments of the present invention include rewriting a bitstream from SVC to AVC. This process may include taking an SVC bitstream as input and generating an AVC bitstream as output. This is conceptually similar to transcoding. However, some embodiments take advantage of the single loop structure of SVC and allow SVC bitstreams to be mapped directly onto AVC syntax elements. Some embodiments can perform this function without introducing errors and without reconstructing the video sequence.

  Embodiments that allow fast rewriting of SVC to AVC bitstreams eliminate the need to incur additional costs incurred by SVC end-to-end. Therefore, it can be discarded when scalable functionality is no longer needed. These embodiments can greatly expand the application range of SVC. As a non-limiting example of an exemplary embodiment, consider a scenario where the final transmission link is a limited speed transmission link. This may be a wireless link to a portable device, or alternatively a wireless link to a high resolution display. In either case, we can intelligently adapt the speed at the transmitter using the scalability features of SVC. However, removing the SVC component from the bitstream is advantageous because the receiving device does not require SVC functionality. This improves the visual quality of the transmitted video as fewer bits are allocated to overhead and more bits are available for image data.

  As a second non-limiting example of bitstream rewriting, consider a system that supports a number of disparate devices. Devices connected by a low-speed transmission link receive an AVC base layer that is part of the SVC bitstream, and devices connected by a higher-speed transmission link receive additional SVC extensions in addition to the AVC base layer. receive. In order to see this extension data, these receivers must be able to decode and reconstruct the SVC sequence. This incurs significant expense for deploying SVC for applications that use many of these devices. A set top box (or other decoding hardware) must be deployed at each receiver. As a more cost effective solution, AVC data can be sent to all devices using a process of rewriting the bitstream from SVC to AVC in the network. This reduces SVC deployment costs.

  As a third non-limiting example of bitstream rewriting, consider an application that uses SVC to store content on a media server for final delivery to a client device. The SVC format is very attractive because it requires less storage space than storing a large number of AVC bitstreams on a server. However, it further requires that the transcoding operations in the server support AVC clients or support SVC capabilities on the clients. Enabling SVC-to-AVC bitstream rewriting allows the media server to use SVC to increase efficiency without requiring computationally demanding transcoding and / or SVC capabilities throughout the network. Allows encoding.

  As a fourth non-limiting example of bitstream rewriting, the process of rewriting a bitstream from SVC to AVC simplifies the hardware design of the SVC decoder. Currently, SVC decoders require modification throughout the AVC decoding and reconstruction logic. By enabling rewriting from SVC to AVC bitstream, the difference between AVC and SVC is aggregated into an entropy decoder and coefficient scaling operation. This simplifies the design of the SVC decoding process because the final reconstruction loop is identical to the AVC reconstruction process. Furthermore, the SVC reconstruction step is guaranteed to include only one prediction operation and only one inverse transformation operation per block. This is different from the current SVC operation and requires a large number of inverse transform operations and variable reference data for intra prediction.

  Some embodiments of the present invention include changes to the SVC coarse-grained scalability layer to allow direct mapping from SVC bitstream to AVC bitstream. These changes include a modified IntraBL mode and restrictions on the conversion of BLSkip blocks in an inter-coded (inter-frame coded) enhancement layer. In some embodiments, these changes can be made by flags sent on a sequence basis, and possibly both a sequence basis and a slice basis.

[Inter coding block]
Some embodiments include modifications for inter-coded blocks. These changes include the following: A block that is inferred from a base layer block must use the same transform as the base layer block. For example, if a block in a coarse-grained scalable layer has base_mode_flag equal to 1, and a base layer block in the same location uses 4x4 transform, the enhancement layer block also uses 4x4 transform There must be.

  Reconstruction of blocks that are estimated from base layer blocks and that use residual prediction must occur in the transform domain. Here, the base layer block will be reconstructed in the spatial domain and then the residual will be transmitted in the enhancement layer. In these embodiments, the base layer block transform coefficients are scaled at the decoder, refined with information in the enhancement layer, and then inverse transformed.

  When the avc_rewrite flag is 1, smoothed_reference_flag is 0.

[Intra coding block]
The intra coding (intra-screen coding) block causes an additional obstacle to the problem of rewriting from SVC to AVC. Within the CGS system, blocks in the enhancement layer may be encoded in IntraBL mode. This mode conveys that intra-coded blocks in the base layer should be decoded and used for prediction. The additional residual can then be communicated in the enhancement layer. In a system that rewrites from SVC to AVC, this creates a fault. This is because the reconstructed intra-coded block cannot be described as the sum of the spatial prediction in the vicinity thereof and the information-transferred residual. Therefore, intra-coded blocks must be transcoded from SVC to AVC. This requires additional computational complexity. In addition, it introduces coding errors that can be propagated through motion compensation.

  Some embodiments of the present invention will be described with reference to FIG. 1A. The decoder or rewriter (system) according to these embodiments includes a first inverse quantizer 5, a scaler 6, a second inverse quantizer 11, a first adder (coefficient integrator) 7, and an inverse transform. And a second adder (second integrator) 9. In these embodiments, base layer residuals (base layer quantized transform coefficients) 1, prediction mode data 2, and enhancement layer residuals (enhancement layer quantized transform coefficients) 3 are: Received by decoder or rewriter. Neighboring block data 4 is also known by the decoder / rewriter. The first dequantizer 5 can dequantize the residual data 1 of the base layer, thereby creating the transform coefficient of the base layer, and the scaler 6 can convert the transform coefficient to the characteristics of the enhancement layer. Scale to match, thereby creating a base layer scaled transform coefficient. In some embodiments, the matched characteristics may include quantization parameter characteristics. Furthermore, the enhancement layer residual 3 is dequantized by the second dequantizer 11 and the first adder 7 scales the base layer scaled residual coefficient (the base layer scaled). Conversion coefficients), thereby forming an integrated coefficient. Thereafter, the inverse transformer 10 inversely transforms the integrated coefficients to generate a spatial domain intensity value. In some embodiments, enhancement layer information can be ignored if not required. Prediction mode data 2 and neighboring block data 4 are used to determine a prediction block by intra prediction 8. Thereafter, the second adder 9 adds the prediction block to the spatial domain intensity values from the base layer and the enhancement layer to generate a decoded block 12.

  Some embodiments of the present invention are described with reference to FIG. 1B. In these embodiments, base layer residual 1, prediction mode 2, and enhancement layer residual 3 are received at a decoder or rewriter. Neighboring block data 135 is also known at the decoder / rewriter and can be used for prediction (134). In these embodiments, the base layer quantized transform coefficient 1 may be scaled 130 to match the characteristics of the enhancement layer, thereby creating a base layer scaled transform coefficient. In some embodiments, the matched characteristics may include quantization parameter characteristics. The quantized transform coefficients of the enhancement layer can be added (131) to the quantized scaled transform coefficient 3 of the base layer to create a quantized and integrated coefficient. The quantized and integrated coefficients can then be dequantized (132) to produce integrated and unquantized coefficients. The integrated and unquantized coefficients can then be inverse transformed (133) to generate integrated spatial domain values. These spatial region values can then be integrated with the prediction data (136) to form a reconstructed image 137.

  Some embodiments of the present invention are described with reference to FIG. 2A. In these embodiments, the bitstream is re-encoded without complete reconstruction of the image. In these embodiments, the base layer (BL) residual data 1 may be received at a decoder, transcoder, decoder portion of an encoder, or other device or module. The enhancement layer (EL) residual data 3 can also be received by the device or module described above. In these embodiments, the first inverse quantizer 5 can inverse quantize the residual 1 of BL to generate a transform coefficient of BL. The scaler 6 can then scale these BL transform coefficients to match the enhancement layer characteristics, thereby creating the BL transform coefficients. In some embodiments, the enhancement layer characteristic may be a quantization parameter, a resolution parameter, or some other parameter that associates the base layer with the enhancement layer. Furthermore, the second inverse quantizer 11 can inverse quantize the enhancement layer data 3 to generate the enhancement layer coefficient 18. A coefficient integrator 19 can then integrate the scaled BL coefficient 16 with the scaled BL coefficient to produce the integrated coefficient 17. A bitstream encoder (bitstream generator) 13 can then rewrite the combined coefficients into a reduced-layer or single layer bitstream. The bitstream encoder 13 can further write the prediction data 2 into the bitstream. The function of the bitstream encoder 13 may further include a quantization function, an entropy encoding function, and other functions.

  Some embodiments of the present invention will be described with reference to FIG. 2B. In these embodiments, the bitstream is re-encoded without complete reconstruction of the image and without inverse quantization. In these embodiments, base layer (BL) residual data 36 may be received at a decoder, transcoder, decoder portion of an encoder, or other device or module. Enhancement layer (EL) data 37 may also be received at the device or module. In these embodiments, the BL signal 36 and enhancement layer signal 37 may be entropy decoded to generate quantized coefficients or indices 21 and 23. The BL quantization index 21 can then be scaled (20) to match the enhancement layer characteristics, thereby creating a BL scaled index. In some embodiments, the enhancement layer characteristic may be a quantization parameter, a resolution parameter, or some other parameter that associates the base layer with the enhancement layer. The scaled BL index 26 can then be integrated (24) with the EL index 23 to generate an integrated index 27. The bitstream encoder 25 can then rewrite the combined coefficients into a reduced layer or single layer bitstream 28. The bitstream encoder 25 can further write prediction data 35 into the bitstream. The functions of the bitstream encoder 25 may also include a quantization function, an entropy encoding function, and other functions.

  In these embodiments, the base layer block need not be completely reconstructed. Instead, both intra prediction mode and residual data are mapped to the enhancement layer. Thereafter, additional residual data is added from the enhancement layer. Finally, the block is reconstructed. The advantage of this approach is that the extended block can be written in a single layer bitstream without loss and without having to completely decode the base layer.

  Some embodiments of the invention include propagating motion data between multiple layers in a CGS system without the use of residual prediction flags. These embodiments include an improved IntraBL method that propagates the intra prediction mode from the base layer to the enhancement layer. Thereafter, intra prediction is performed in the enhancement layer.

  In these embodiments, the transform type of the IntraBL block must be the same as the base layer block at the same location. For example, if the base layer block uses an 8x8 transform, the enhancement layer block must also use an 8x8 transform.

  In some embodiments, an 8x8 conversion flag can also be transmitted in the enhancement layer to allow independent processing of the bitstream.

  In some exemplary embodiments, blocks encoded by a 16x16 transform in the base layer are also encoded by a 16x16 transform in the enhancement layer. However, enhancement layer blocks are transmitted in a 4x4 scan pattern and method. That is, in some embodiments, the 16 × 16 block of DC and AC coefficients are not sent separately.

  Some embodiments of the present invention will be described with reference to FIGS. The system according to these embodiments includes a size determiner 201, a determiner 202, a first selector 203, and a second selector 204. In those embodiments involving multi-layer images, intra prediction modes and transform data can be estimated from one layer to another. In some embodiments, the size determiner 201 can determine the transform size of the first layer (30). The first layer may be a base layer or a layer that is a basis for predicting another layer. In these embodiments, a predetermined transform size is established. Thereafter, the conversion size of the first layer is compared with a predetermined (predetermined) conversion size. That is, the determiner 202 determines whether the conversion size of the lower layer is the same (substantially similar) to a predetermined conversion size. When the conversion size of the first layer is the same as the predetermined conversion size (31), the first selector 203 selects the predetermined conversion size for the inverse conversion operation (33). If the conversion size of the first layer is not the same as the predetermined conversion size (31), the second selector 204 selects the default conversion size for the inverse conversion operation (32). In some embodiments, the predetermined transform size can be 8 × 8 and the default transform size can be 4 × 4.

  In some embodiments, a predetermined transform size can also be associated with a particular scan pattern and method. In these embodiments, the relationship between the transform size of the first layer and a predetermined transform size can also trigger a special encoding method and pattern. For example, in some embodiments, the predetermined transform size may be 16 × 16, and the match between the predetermined 16 × 16 size and the actual lower layer size is “16 × 16 May be used, but the data is encoded with a 4 × 4 scan pattern and scan method in which AC and DC coefficients are transmitted together.

  Some embodiments of the present invention will be described with reference to FIG. In these embodiments, the multi-layer bitstream is analyzed (40) and processed to determine the transform size of the base layer and generate the BL coefficient values. The enhancement layer of the bitstream is also analyzed (41) to determine whether a conversion index exists. If the enhancement layer transform index is present in the bitstream (42), the indicated transform size can be used for the inverse transform of the EL coefficients (43). When the enhancement layer conversion index does not exist in the bitstream (42), it is determined whether the conversion size of the base layer is 8 × 8 (44). When the transform size of the base layer is 8 × 8, the enhancement layer is inversely transformed using the 8 × 8 transform size (46). If the base layer transform size is not 8 × 8, the enhancement layer can be inverse transformed using a default transform size such as 4 × 4 (45).

  In some embodiments of the invention, the intra prediction mode can be copied directly from the base layer by estimating the intra prediction mode from the base layer in the IntraBL block. In some alternative embodiments, it may be encoded differently from the base layer mode. In some embodiments, current methods for signaling intra prediction modes in AVC can be used. However, in these embodiments, the predicted mode (or most likely mode) is set equal to the base layer mode.

  In some embodiments, the 8x8 transform flag may be omitted from the enhancement layer bitstream and the transform may be estimated from the base layer mode.

  In some embodiments, 16 × 16 transform coefficients may be communicated in the same way both in the base layer and in the enhancement layer. The presence of a 16x16 transform may be signaled with an additional flag in the enhancement layer, and may be estimated from the base layer bitstream.

  Some embodiments of the present invention include a residual prediction flag for IntraBL blocks. These embodiments allow base layer residuals to be used adaptively to improve enhancement layer intra-predicted blocks.

  In some embodiments of the invention, the encoder can disable all modes in the SVC bitstream that cannot be mapped directly to the AVC bitstream. Information transfer for these embodiments can occur in the SVC bitstream. In some exemplary embodiments, this communication can occur in the sequence header, in the sequence parameter set, in the picture parameter set, in the slice header, or elsewhere. In some embodiments, this communication can occur in SEI messages. In an exemplary embodiment, this communication can occur in a spatial scalability SEI message. In some embodiments, this signaling may occur by other out-of-band methods and in some cases may not require standard changes to the SVC decoding operation.

  In some embodiments, once the encoder communicates this mode of operation, the decoder can assume that the encoder is generating a bitstream that can be translated into AVC. In some exemplary embodiments, the encoder may not use the IntraBL block mode or smoothing reference tool when operating in this mode. Furthermore, in these embodiments, the encoder can ensure that residual data can be incorporated by scaling the base layer transform coefficients and then adding the transmitted residuals. These embodiments may require the encoder to use the same conversion method in the base layer and in the enhancement layer.

[Bitstream rewriting from SVC to AVC for CGS: syntax]
F.7.3.2 Sequence Parameter Set SVC Extension Syntax

F.7.3.4 Slice header in scalable extensions

F.7.3.6.3 Residual syntax in scalable extensions

F.7.3.2 Sequence Parameter Set SVC Extension Semantics nal_unit_extension_flag equal to 0 indicates that the parameter specifying the mapping of simple_priority_id to (dependency_id, temporal_level, quality_level) follows in the sequence parameter set . Nal_unit_extension_flag equal to 1 indicates that there is no parameter that specifies mapping of simple_priority_id to (dependency_id, temporal_level, quality_level). When nal_unit_extension_flag does not exist, it is assumed that nal_unit_extension_flag is equal to 1. The NAL unit syntax element extension_flag of all NAL units with nal_unit_type equal to 20 and 21 that refer to the current sequence parameter set shall be equal to nal_unit_extension_flag.

Note: If profile_idc is not equal to 83, the syntax element extension_flag of all NAL units with nal_unit_type equal to 20 and 21 referring to the current sequence parameter set shall be equal to 1.
A value obtained by adding 1 to number_of_simple_priority_id_values_minus1 specifies the number of values of simple_priority_id that is specified by a parameter that is mapped to (dependency_id, temporal_level, quality_level) in the sequence parameter set. The value of number_of_simple_priority_id_values_minus1 is assumed to be in the range of 0-63.

  priority_id, dependency_id_list [priority_id], temporal_level_list [priority_id], and quality_level_list [priority_id] specify the process for estimating the syntax elements dependency_id, temporal_level, and quality_level shown in subordinate clause F.7.4.1. If dependency_list [priority_id], temporal_level_list [priority_id], and quality_level_list [priority_id] do not exist for all values of priority_id, dependency_list [priority_id], temporal_level_list [priority_id], and quality_level_list [priority_id] are estimated to be equal to 0 And

  extended_spatial_scalability indicates the presence of a syntax element associated with a geometric parameter for base layer upsampling. If extended_spatial_scalability is equal to 0, the geometric parameter is not in the bitstream. If extended_spatial_scalability is equal to 1, the geometric parameter is in the sequence parameter set. If extended_spatial_scalability is equal to 2, the geometric parameter is in slice_data_in_scalable_extension. A value of 3 is not used for extended_spatial_scalability. When extended_spatial_scalability does not exist, it is assumed that extended_spatial_scalability is equal to 0.

  scaled_base_left_offset specifies the horizontal offset between the upper left pixel of the up-sampled base layer picture and the upper left pixel of the current layer picture in units of two luminance samples. When scaled_base_left_offset does not exist, it is assumed that scaled_base_left_offset is equal to 0.

The variable ScaledBaseLeftOffset is specified as follows:
ScaledBaseLeftOffset = 2 * scaled_base_left_offset (F-40)
The variable ScaledBaseLeftOffsetC is defined as follows:
ScaledBaseLeftOffsetC = ScaledBaseLeftOffset / SubWidthC (F-41)
scaled_base_top_offset specifies the vertical offset between the upper left pixel of the upsampled base layer picture and the upper left pixel of the current layer picture in units of two luminance samples. When scaled_base_top_offset does not exist, it is assumed that scaled_base_top_offset is equal to 0.

  The variable ScaledBaseTopOffset is defined as follows.

ScaledBaseTopOffset = 2 * scaled_base_top_offset (F-42)
The variable ScaledBaseTopOffsetC is defined as follows.

ScaledBaseTopOffsetC = ScaledBaseTopOffset / SubHeightC (F-43)
scaled_base_right_offset specifies the horizontal offset between the lower right pixel of the upsampled base layer picture and the lower right pixel of the current layer picture in units of two luminance samples. When scaled_base_right_offset does not exist, it is assumed that scaled_base_right_offset is equal to 0.

  The variable ScaledBaseRightOffset is defined as follows.

ScaledBaseRightOffset = 2 * scaled_base_right_offset (F-44)
The variable ScaledBaseWidth is defined as follows:
ScaledBaseWidth
= PicWidthInMbs * 16-ScaledBaseLeftOffset-ScaledBaseRightOffset (F-45)
The variable ScaledBaseWidthC is defined as follows:
ScaledBaseWidthC = ScaledBaseWidth / SubWidthC (F-46)
scaled_base_bottom_offset specifies the vertical offset between the lower right pixel of the up-sampled base layer picture and the lower right pixel of the current layer picture in units of two luminance samples. When scaled_base_bottom_offset does not exist, it is assumed that scaled_base_bottom_offset is equal to 0.

  The variable ScaledBaseBottomOffset is specified as follows.

ScaledBaseBottomOffset = 2 * scaled_base_bottom_offset (F-47)
The variable ScaledBaseHeight is defined as follows.

ScaledBaseHeight
= PicHeightInMbs * 16-ScaledBaseTopOffset-ScaledBaseBottomOffset (F-48)
The variable ScaledBaseHeightC is defined as follows.

ScaledBaseHeightC = ScaledBaseHeight / SubHeightC (F-49)
chroma_phase_x_plus1 specifies the horizontal phase shift of the chrominance component in a unit of a quarter of the horizontal sampling space of the current layer picture. When chroma_phase_x_plus1 does not exist, it is assumed that chroma_phase_x_plus1 is equal to 0. chroma_phase_x_plus1 is in the range of 0 to 1, and the values 2 and 3 are unused.

  chroma_phase_y_plus1 specifies the vertical phase shift of the chrominance component in a unit of a quarter of the vertical sampling space of the current layer picture. When chroma_phase_y_plus1 does not exist, it is assumed that chroma_phase_y_plus1 is equal. chroma_phase_y_plus1 is in the range of 0 to 2, and the value of 3 is unused. Note: The chroma type specified in vui_parameters must match the chroma phase parameters chroma_phase_x_plus1 and chroma_phase_y_plus1 in the same sequence_parameter_set.

  avc_rewrite_flag indicates that the transmitted sequence can be rewritten as an AVC bit stream without deterioration only by decoding and encoding of the entropy code and scaling of the transform coefficient. An alternative method is used for IntraBL blocks, which limits the choice of transform size by the encoder.

  avc_adaptive_rewrite_flag indicates that avc_rewrite_flag is sent in the slice header.

  Some embodiments of the invention include a scaling process that maps quantized transform coefficients to either an “unquantized” version or an alternative quantization domain. In some embodiments, if the avc_rewrite_flag signals that these processes have been disabled as described above, the current H.264 standard. The decoded transform coefficients in all layers can be “dequantized” according to the process defined in the H.264 / AVC video coding standard. However, if the avc_rewrite_flag communicates that these embodiments are enabled, the decoded transform coefficient or index decoded in multiple layers preceding the desired enhancement layer is “non-quantum”. It's not. Instead, the quantized coefficient or index explicitly depends on the next higher layer (especially the previously mentioned lower layer) from the lower layer (especially the layer that depends on the desired enhancement layer). To a layer closer to the desired enhancement layer in the order of dependency.

  Some embodiments of the present invention will be described with reference to FIGS. The system according to these embodiments includes a first parameter determiner 211, a second parameter determiner 212, and a scaler 213. In these embodiments, the mapping process may operate as follows. First, the first parameter determiner 211 determines a quantization parameter, ie, a Qp value, in the lower layer bitstream (50). Next, the second parameter determiner 212 determines the quantization parameter in the upper layer, that is, the Qp value (51). The scaler 213 can then scale the lower layer coefficients (first layer transform coefficients) by a rate based on the quantization parameter (52).

  In some embodiments, the difference between the lower layer Qp value and the upper layer Qp value may be calculated. In some embodiments, the transform coefficients can be scaled by the following process.

(Here, THigherLayer and TLowerLayer are transform coefficients in the upper layer and the lower layer, respectively, n is an integer, and Qp_LowerLayer and Qp_HigherLayer are quantization parameters for the lower layer and the upper layer, respectively)
The calculation of the mapping process can be performed in a way that simplifies many calculations. For example, the following systems are equivalent:

(Where // indicates integer division,% indicates modulo operation, and M and ScaleMatrix are predetermined constants)
A specific example of these predetermined values is as follows.

ScaleMatrix = [512 573 642 719 806 902]
M = 512
However, it should be readily understood that other values for M and ScaleMatrix may be used.

  The simplified example above assumes that the value of Qp_Diff is always greater than zero. Thus, in some embodiments, the application can check the value of Qp_Diff before performing the scaling operation. If the value of Qp_Diff is less than 0, a value of 0 can be reassigned to it prior to further processing. In some embodiments, Qp_LowerLayer can be considered to be greater than or equal to Qp_HigherLayer.

  In some alternative embodiments, the following system may be implemented.

  In an exemplary embodiment, the predetermined value can be selected as follows.

ScaleMatrix = [291 325 364 408 457 512 573 642 719 806 902]
M = 512
In some embodiments, after mapping transform coefficients from lower layers to higher layers, the coefficients may be fine-tuned in some cases using the process described above. After fine tuning, a second scaling operation can be used. This scaling operation may “de-quantize” the transform coefficients.

  While some embodiments described above described only one lower layer and one upper layer, some embodiments may include more than two layers. For example, the typical three layers can function as follows. Initially, the lowest layer can be decoded. Thereafter, the transform coefficient can be mapped to the second layer by the method described above. Thereafter, the mapped transform coefficients can be fine tuned. These transform coefficients can then be mapped to the third layer using the method described above. These transform coefficients can then be fine tuned and the resulting coefficients are calculated as AVC / H. It can be “dequantized” by a scaling operation, such as a scaling operation defined by the H.264 video coding standard.

  Some embodiments of the present invention will be described with reference to FIGS. The system according to these embodiments includes a first discriminator 221, a second discriminator 222, a first indicator determiner 223, a second indicator determiner 224, and a value determiner 225. In these embodiments, it is possible to notify the encoding operation or the decoding operation for the target block or the macroblock using information associated with the adjacent macroblock. In some embodiments, the first adjacent macroblock is identified by the first identifier 221 (60) and the second adjacent macroblock is identified by the second identifier 222 (61). Thereafter, the first index determiner 223 determines the first adjacent macroblock index (62), and the second index determiner 224 determines the second adjacent macroblock index (63). The value determiner 225 can then determine entropy encoder control values based on these neighboring macroblock indices (64).

  Some embodiments of the present invention will be described with reference to FIG. In these embodiments, a first neighboring macroblock is identified (71) and a second neighboring macroblock is identified (72). Thereafter, the attributes of the first adjacent macroblock can be examined to determine whether the first macroblock satisfies a predefined condition (73). Furthermore, the condition of the second adjacent macroblock can also be checked to determine whether a predefined condition is satisfied (74). In some embodiments, these conditions may include: "A macroblock is available", "A macroblock is encoded in inter-layer prediction mode", "A macroblock is encoded in the spatial domain. Or “whether the macroblock is intra-predicted by DC prediction” and “whether the macroblock is coded with reference to another temporally matching layer”. If any of the above conditions are met for the first macroblock 75, the first macroblock flag is set to indicate compliance (80). If neither condition is met, the flag is set to indicate non-compliance (76). In some embodiments, if any condition is met (80), the flag is set to “0”, and if neither condition is met (76), the flag is set to “1”. be able to. The same process 74, 79 can continue for the second adjacent macroblock. In that process, if the condition is not met (78), the flag can be set to one value, and if the condition is met (81), the flag can be set to another value. If both neighboring macroblocks have been examined and the two associated flags are set, these flags can be added (82). Thereafter, the value of the addition result can be used as the entropy encoder control value.

  Some embodiments of the present invention will be described with reference to FIG. In these embodiments, the first neighboring macroblock is identified (90) and the second neighboring macroblock is identified (91). Thereafter, the attributes of the first adjacent macroblock and the second adjacent macroblock are checked to determine whether the macroblock satisfies a predetermined condition (92). In some embodiments, these conditions may include: “Are macroblocks available?”, “Is the macroblock encoded in inter-layer prediction mode”, and “A macroblock refers to another layer?” It is possible to include a condition of “is it encoded?”. If any of the above conditions is satisfied for any macroblock (94), the estimated prediction mode is set to a predetermined mode. In some embodiments, the predetermined mode may be a DC prediction mode.

  In these embodiments, the actual prediction mode can also be specified. The actual prediction mode can be based on the content of the image. The prediction mode can be determined using the method that produces the least error or reduced error. If the actual prediction mode is the same as the estimated prediction mode (94), the bitstream can be encoded to indicate use of the estimated prediction mode. On the decoder side, the same process can be continued to select the estimated mode when decoding the bitstream. If the actual prediction mode is not the same as the estimated prediction mode (94), a message can be sent to indicate the actual mode and its selection (95). Details regarding the information transmission of the estimated prediction mode and the actual prediction mode can be found in the JVT AVC standard. The JVT AVC standard is incorporated herein by reference.

  Some embodiments of the invention may include intra prediction mode encoding for luminance and color information in intra-coded blocks. Traditionally, these modes are signaled in a context-adaptive manner and encoded in a manner that depends on the prediction mode of spatial neighbors. In some embodiments of the invention, a conditional process may be used. In these embodiments, when the adjacent unit does not use inter-layer prediction, the prediction mode can be predicted from the adjacent unit. Blocks that utilize inter-layer prediction can be processed in one of the following ways. In some exemplary embodiments, a block can be processed as if it had the most likely prediction mode. H. In H.264 / AVC related embodiments, this can be a DC prediction mode (mode 2) for the case of luminance prediction.

  In some alternative embodiments, the block may be processed as if it were an inter-coded block and OUTSIDE (“external”) of the prediction region. In these embodiments, OUTSIDE has a specific context that includes the software used to test within the JVT SVC project group. This software is commonly known as JSVM.

  In some environments, prediction mode encoding and context selection to communicate the encoded mode may be separate processes. Different prediction methods may be used for these two processes. For example, for all intra-coded blocks, including blocks that use inter-layer prediction, the prediction mode may be encoded using the actual prediction mode. However, these same blocks may use another rule, such as one of the rules described above, to obtain the context for encoding the encoded value. For example, the context can assume that an intra block that utilizes inter-layer prediction has the most likely prediction mode. Some of these embodiments allow multiple bitstreams corresponding to different layers to be processed independently.

  Some embodiments of the present invention maintain the “encoded block pattern” information, ie Cbp, as defined in the JVT SVC standard (which is hereby incorporated by reference). Contains. This information defines a sub-region containing residual information in the image (or macroblock). In some cases, the bitstream decoder first decodes Cbp and then uses the information to analyze the rest of the bitstream, so that information may be needed to decode the bitstream. (For example, Cbp can specify the number of transform coefficient lists that can exist.) In many decoders, Cbp is also used to reconstruct the decoded frame. For example, if Cbp indicates residual information, the decoder need only calculate the inverse transform. In some embodiments, Cbp transmitted in the bitstream can be utilized by the analysis process to extract transform coefficients. However, it may no longer be useful for the reconstruction process. This is because the sub-region can include residual information from the previous layer.

  Therefore, the decoder according to the embodiment of the present invention may (1) not use the Cbp information in the reconstruction process, or (2) do not recalculate Cbp after analyzing the bitstream. Good. An example of a recalculation process is to scan the entire coefficient list to identify sub-regions containing residual information, or alternatively, transmitted Cbp and Cbp (which reconstruct lower layer data) Generating a new Cbp by calculation of a binary logical OR operation. In this case, “lower layer data” means a layer used during the inter-layer prediction process.

  Some embodiments of the present invention will be described with reference to FIGS. The system according to these embodiments includes a receiver 231, a decoder 232, an analyzer 233, a scaler 234, an adder 235, and a calculator 236. In these embodiments, receiver 231 receives a bitstream that includes Cbp information and encoded image data (100). The decoder 232 can decode the Cbp information (101), and use the Cbp information to determine which part of the bitstream includes transform coefficient data. The analyzer 233 can then analyze the bitstream using the Cbp information to identify quantized indexes or non-quantized transform coefficients in the base layer and all enhancement layers (102). ). Scaler 234 may then scale the base layer or lower layer index or coefficient to match the enhancement layer (103). An adder 235 may then add the scaled index or coefficient to the enhancement layer (ie, merge with the enhancement layer) to form a unified layer (104). The calculator 236 can then recalculate or update the Cbp information to reflect the change in coefficient position between the original base layer or lower layers and the new integrated layer (105). The new merged Cbp information can then be used to process subsequent merged layers or resulting reconstructed images. In some embodiments, the integrated Cbp information can be used for loop filter operations defined in the AVC standard.

  Some embodiments of the present invention include a method and system for processing a flag that enables an 8 × 8 transform. These embodiments may relate to the JVT SVC standard. In these embodiments, it is not necessary to transmit this flag if the block is intra-coded with inter-layer prediction and does not contain residual data. In some embodiments, the flag need not be transmitted if the inter-frame prediction utilizes a block that is smaller than a specified size (eg, 8 × 8, etc.). These embodiments may copy the conversion flag transmitted in one or more lower layers and use this flag during the reconstruction process.

  Some embodiments of the present invention include an alternative method and system for processing a flag that enables an 8x8 transform. In these embodiments, it is not necessary to transmit this flag if the block does not contain residual data. If this case occurs in the lower layer used for inter-layer prediction, the upper layer can choose to enable the 8 × 8 transform when sending residual data. This may be the default value of the flag that is not transmitted but disables the 8x8 conversion. In some embodiments, in this special case, the decoder may allow lower and upper layers to use different transforms.

  Some embodiments of the present invention include methods and systems for processing quantization matrices. Quantization matrices are also known to those skilled in the art as weight matrices or scaling matrices. These matrices may change the “non-quantization” process and allow the encoder and decoder to apply frequency-dependent (or transform coefficient-dependent) quantization. In these embodiments, the presence of the scaling matrix alters the scaling process described in the mapping process described above. In some embodiments, the mapping procedure can be written as follows:

(Here, THigherLayer and TLowerLayer are transform coefficients in the upper layer and the lower layer, n is an integer, Qp_LowerLayer and Qp_HigherLayer are quantization parameters for the lower layer and the upper layer, respectively, and S_L and S_H are respectively (Scaling rate for lower and upper layers)
To adjust the weight matrix, some embodiments can utilize a modified version of the algorithm shown in the mapping process. A modified version of this algorithm can be defined as follows with reference to the above description.

(Here, S_L [n] and S_H [n] may be explicitly present, or may be obtained from the bitstream instead)
In an alternative embodiment of adjusting the weight matrix, an additional weight matrix can be sent in the bitstream. This additional weight matrix can explicitly define the frequency weights necessary to predict a layer from a lower layer. For example, the weight matrix can be used as follows.

(W1 and W2 are weight matrices included in the bit stream)
In some embodiments, either W1 or W2 may not be transmitted. In these embodiments, a non-transmitted matrix can be considered as having elements equal to zero.

  Embodiments of the present invention include methods and systems for modifying, creating and / or applying scalable video codecs. Some embodiments allow for fast conversion of a multi-layer bitstream to a bitstream with fewer layers. Some embodiments include conversion from a multi-layer bitstream to a single-layer bitstream. Some exemplary embodiments include conversion from an AVC bitstream to an SVC bitstream.

  Embodiments of the present invention relate to residual prediction. These embodiments may include a residual prediction process that operates in both the transform domain and the spatial domain. In an exemplary embodiment, if an upper layer in the bitstream refers to a lower layer in the bitstream, and both layers contain the same spatial resolution, the residual prediction process extracts the residual transform coefficients from the lower layer. Mapping to higher layers can be included. This mapping process is useful for scaled transform coefficients or levels of (unscaled) transform coefficients. In some embodiments, the scaled transform coefficient residual prediction process may be defined as in the next paragraph.

A.8.11.4.1 Residual accumulation process for scaled transform coefficients The input to this process is
The variable fieldMb that indicates whether the macroblock is a field or frame macroblock
The variable lumaTrafo indicating the type of luminance conversion
List of scaled transform coefficient values with elements 256 + 2 * MbWidthC * MbHeightC sTCoeff
And
The output of this process includes a modified version of the scaled transform coefficient value sTCoeff.

  The progressive fine-tuning process for scaled transform coefficients described in subsection G.8.11.3 (as defined in the SVC standard, which is included herein) is fieldMb, You can start with lumaTrafo and sTCoeff as inputs and a modified version of sTCoeff as output.

  Conversely, in some embodiments, the enhancement layer may perform a residual prediction process in the spatial domain when a lower layer that includes a different spatial resolution is utilized for inter-layer prediction. In these embodiments, the residual from the referenced layer is reconstructed in the intensity domain and interpolated to the enhancement layer resolution. In an alternative scenario, the residual from the reference layer is added to the prediction derived from the reference layer in the spatial domain. Thereafter, the result of this addition is interpolated into the enhancement layer.

  Some embodiments of the present invention will be described with reference to FIGS. The system according to these embodiments includes a resolution determiner 241, a comparator 242, a controller 243, a coefficient scaler 244, a coefficient integrator 245, an inverse transformer 246, and a spatial domain integrator 247. In these embodiments, the current layer may be examined to determine if the current layer is using residual prediction (110). If residual prediction is not used, no accumulation is required (111). If residual prediction is used (110), the resolution determiner 241 determines the spatial resolution of the current layer and the reference layer (112, 113). Thereafter, the comparator 242 compares the spatial resolution of the current layer with the spatial resolution of the reference layer (114). Controller 243 selectively allows coefficient scaler 244 and coefficient integrator 245 to perform steps 116 and 117 when these spatial resolutions are the same. That is, if these spatial resolutions are the same (114), the coefficient or index of the reference layer (from which the current layer will be predicted) is scaled by the coefficient scaler 244 (116) and the coefficient integrator 245 It can be integrated (117) with an index or coefficient. The controller 243 selectively allows the inverse transformer 246 and the spatial domain integrator 247 to perform steps 115, 118, and 120 if these spatial resolutions are not the same. That is, if the spatial resolutions are not the same (114), the current layer and reference layer indices can be dequantized and the resulting coefficients can be inverse transformed (115, 118). A spatial domain integrator 247 can then integrate the obtained spatial domain values in the current layer and the reference layer to form a reconstructed image (120).

  As can be readily seen from the above description, the method of residual prediction depends on the resolution of the enumerated upper layers and enumerated lower layers referenced for prediction. Unfortunately, this is a problem because the accumulation of residual information in the spatial domain may not be equal to the accumulation of residuals in the transform domain and subsequent conversion to the spatial domain. In the case of a standardized decoding process, this can lead to errors between the encoder and decoder and a reduction in coding efficiency.

  Current SVC systems attempt to solve this problem by performing residual prediction only in the spatial domain. However, some embodiments of the invention include a decoding process that performs residual prediction in both regions. More specifically, if residual prediction is enabled and the enhancement layer and the layer referenced for inter-layer prediction are the same resolution, the residual is accumulated in the transform domain. However, if residual prediction is enabled and the enhancement layer and the layer referenced for inter-layer prediction have different resolutions, the residual is accumulated in the spatial domain.

  A typical decoding process is described as follows.

  Although not explicitly described in the above pseudocode, other exemplary embodiments include other extensions to the defined decoding process. In some embodiments, intra layer prediction may be performed at multiple layers in a scalable bitstream. If this is allowed in the video coding standard, the function GenerateIntraLayerPrediction may be called prior to all processing. The output of this function can be added to the array rYCC. Furthermore, in some embodiments, the function GenerateIntraLayerPrediction is not called in the above pseudo code. Instead, the line outYCC = GeneateIntraLayerPrediction (layerID) + rYCC will be replaced with outYCC = rYCC.

  In some embodiments of the present invention, the residual accumulation process may be performed on unscaled transform coefficients. In this case, an inter-layer prediction process can be performed before constructing the scaled transform coefficients. An aspect of some embodiments is entitled “Methods and Systems for Image Scalability” and is filed on July 10, 2006, C.I. It is described in US Provisional Patent Application No. 60 / 806,930, invented by C. Andrew Segall. An aspect of some embodiments is entitled “Systems and Methods for Bit-Stream Rewriting for Coarse Grain Scalability”, 2006. Filed on October 6th, C.I. U.S. Provisional Patent Application No. 60 / 828,618 invented by Andrew Segal.

  The pseudo code for a typical procedure is as follows:

  Some embodiments of the present invention include a decoder that takes a scalable bitstream as input and generates a reconstructed image sequence. The scalable bitstream uses an inter-layer prediction process to project information from the enumerated lower layers of the bitstream to the enumerated upper layers of the bitstream.

  Some embodiments of the present invention include a decoding process that accumulates residual information both in the transform domain and in the spatial domain. If the enumerated layers in the bitstream describe image sequences having the same resolution, accumulation within the transform domain is performed between the enumerated layers in the bitstream.

  Some embodiments of the present invention provide a decoding process that transforms the accumulated transform coefficients into the spatial domain only when processing a current layer that has a different spatial resolution than the layer used for inter-layer prediction. Contains. The transform coefficients are transformed into the spatial domain and subsequently upsampled (ie, interpolated). Thereafter, the transform coefficient list is set equal to zero.

  Some embodiments of the present invention may decode such that the residual accumulates in the transform domain until the resolution is different between the current decoding layer and the layer used for inter-layer prediction. Includes processes. Thereafter, the transform coefficient list is set to 0, and in the processing of the layer referring to the subsequent layer having the same spatial resolution, accumulation is performed in the transform region.

  Some embodiments of the present invention perform intra layer prediction, perform inverse transform calculations based on scaled transform coefficients, and perform inverse transform operations on residual signals that may be non-zero. And a decoding process, such as generating an output bitstream by adding the result of the previous addition with the output of the intra layer prediction process.

  Some embodiments of the present invention include a decoding process that further enables inter-layer prediction based on unscaled transform coefficients or transform coefficient levels.

  Some embodiments of the invention include a decoding process that further enables intra-layer prediction within a layer of the bitstream that is not reconstructed for output. The result of this intra layer prediction is added to the accumulated spatial residual.

  Some embodiments of the present invention include a decoding process that performs clipping during the residual prediction process.

  The terms and expressions used in the foregoing specification have been used herein as descriptive terms, not as restrictive terms, and the use of such terms and expressions includes the features and portions thereof illustrated and described. It should be recognized that the equivalents of the invention are not intended to be excluded and the scope of the present invention is defined and limited only by the claims that follow.

  Each element of the system of the embodiment of the present invention may be realized by software using a CPU as follows.

  In other words, the above system includes a CPU (Central Processing Unit) that executes instructions of a control program that realizes each function, a ROM (Read Only Memory) that stores the program, and a program that is executed on the CPU It may include components such as a RAM (Random Access Memory), a storage device (recording medium) such as a memory for storing the program and various data. The object of the present invention is to record the program code (for example, an executable code program, an intermediate code program, a source program, etc.) of the control program of the system, which is software for realizing each function, in a computer-readable form. It can also be achieved by recording on a medium, supplying the recording medium to the system, and reading the program code from the recording medium by the computer (or CPU or MPU) and executing the program. .

  Examples of such recording media are: tapes such as magnetic tape and cassette tape; magnetic disks such as flexible disks and hard disks; disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R; IC cards ( Cards such as memory cards); mask ROM, EPROM (Erasable Programmable Read Only Memory; and erasable programmable ROM), EEPROM (Electrically Erasable Programmable Read Only Memory), flash ROM, etc. Includes semiconductor memory.

  Alternatively, the system may be connected to a communication network so that the program code can be supplied via the communication network. Non-limiting examples of communication networks include the Internet, Intranet, Extranet, LAN, ISDN, VAN, CATV network, virtual private network, telephone line network, mobile communication network, and satellite communication network . Non-limiting examples of transmission media that make up a communication network include: wired media such as IEEE 1394, USB, power line communications, cable TV lines, telephone lines, and ADSL lines; infrared such as IrDA and remote controllers; Bluetooth® , IEEE802.11, HDR, mobile phone network, satellite line and terrestrial digital broadcasting network. Note that the present invention can be realized by a carrier wave or a data signal sequence embodied by electronic transmission of a program code.

Claims (20)

A method for integrating multiple layers in a multi-layer bitstream comprising:
a) dequantizing the quantized transform coefficients of the first layer, thereby creating the transform coefficients of the first layer;
b) scaling the first layer transform coefficients to match the characteristics of the second layer, thereby creating scaled transform coefficients for the first layer;
c) dequantizing the quantized transform coefficients of the second layer, thereby creating a transform coefficient of the second layer; and d) converting the scaled transform coefficients of the first layer with the transform coefficients of the second layer. Integrating the method to form an integrated coefficient. A system for integrating multiple layers in a multi-layer bitstream,
a) a first dequantizer for dequantizing the quantized transform coefficients of the first layer, thereby creating the transform coefficients of the first layer;
b) a scaler for scaling the first layer transform coefficients to match the characteristics of the second layer, thereby creating scaled transform coefficients for the first layer;
c) a second inverse quantizer for inversely quantizing the second layer quantized transform coefficients, thereby creating a second layer transform coefficient; and d) a scaled transform coefficient for the first layer. Including a coefficient integrator for integrating the second layer transform coefficients to form an integrated coefficient. A method for converting an SVC compliant bitstream into AVC compliant data,
a) receiving an SVC-compliant bitstream including prediction data, base layer residual data, and enhancement layer residual data;
b) dequantizing the base layer residual data, thereby creating base layer transform coefficients;
c) dequantizing the enhancement layer residual data, thereby creating enhancement layer transform coefficients;
d) scaling the base layer transform coefficients to match the enhancement layer quantization characteristics, thereby creating base layer scaled transform coefficients; and e) base layer scaled transform coefficients. Integrating the enhancement layer transform coefficients to form an integrated coefficient. A method for integrating multiple layers in a multi-layer bitstream comprising:
a) receiving the quantized transform coefficients of the first layer;
b) scaling the quantized transform coefficients of the first layer to match the characteristics of the second layer, thereby creating a quantized scaled transform coefficient of the first layer;
c) receiving the quantized transform coefficients of the second layer; and d) integrating the scaled transform coefficients of the first layer with the quantized transform coefficients of the second layer to quantize and integrate A method comprising the step of forming a coefficient. A method for conditionally integrating multiple layers in a multi-layer bitstream,
a) receiving the quantized transform coefficients of the first layer;
b) receiving the quantized transform coefficients of the second layer;
c) receiving a layer integration indicator;
d) when the layer integration indicator indicates transform domain accumulation, the quantized transform coefficients of the first layer are scaled to match the characteristics of the second layer, and thereby the first layer quantum Creating a normalized and scaled transform coefficient; and e) a quantized transform of the second layer to the scaled transform coefficient of the first layer if the layer integration indicator indicates transform domain accumulation Integrating the coefficients to form a quantized integrated coefficient. A method for reconstructing an enhancement layer from a multi-layer bitstream, comprising:
a) receiving a first layer intra prediction mode;
b) receiving a second layer bitstream prediction indicator indicating that the first layer prediction mode is used for second layer prediction;
c) configuring a second layer prediction based on adjacent block data in the second layer using the prediction mode of the first layer; and d) integrating the second layer prediction with residual information; Creating a reconstructed second layer by: A method for integrating multiple layers in a multi-layer bitstream comprising:
a) determining a first spatial resolution of the first layer of the multi-layer image;
b) determining a second spatial resolution of the second layer of the multi-layer image;
c) comparing the first spatial resolution with the second spatial resolution;
d) performing steps e) to f) when the first spatial resolution is substantially equal to the second spatial resolution;
e) scaling the first layer transform coefficients to match the characteristics of the second layer, thereby creating scaled transform coefficients for the first layer;
f) integrating the scaled transform coefficients of the first layer with the transform coefficients of the second layer to form an integrated coefficient;
g) performing steps h) to k) if the spatial resolution of the first layer is not substantially equal to the spatial resolution of the second layer;
h) inverse transforming the transform coefficients of the first layer, thereby generating a spatial domain value of the first layer;
i) inverse transforming the transform coefficients of the second layer, thereby generating a spatial domain value of the second layer;
j) scaling the spatial domain value of the first layer to match the resolution of the second layer, thereby generating a scaled spatial domain value of the first layer; and k) scaled of the first layer. Integrating the spatial region value with the spatial region value of the second layer, thereby generating an integrated spatial region residual value. A system for integrating multiple layers in a multi-layer bitstream,
a) a resolution determiner for determining a first spatial resolution of the first layer of the multi-layer image and for determining a second spatial resolution of the second layer of the multi-layer image;
b) a comparator for comparing the first spatial resolution with the second spatial resolution;
c) a controller for performing steps e) to f) when the first spatial resolution is substantially equal to the second spatial resolution;
d) a coefficient scaler for scaling the first layer transform coefficients to match the characteristics of the second layer, thereby creating a scaled transform coefficient for the first layer;
e) a coefficient integrator for integrating the scaled transform coefficients of the first layer with the transform coefficients of the second layer to form an integrated coefficient;
f) a controller that selectively performs steps g) -i) when the spatial resolution of the first layer is not substantially equal to the spatial resolution of the second layer;
g) Inversely transform the first layer transform coefficients, thereby generating the first layer spatial domain values, and inverse transform the second layer transform coefficients, thereby generating the second layer spatial domain values Inverse converter for;
h) a spatial domain scaler for scaling the spatial domain value of the first layer to match the resolution of the second layer, thereby generating a scaled spatial domain value of the first layer; and i) first A system including a spatial domain integrator for integrating the scaled spatial domain value of a layer with the spatial domain value of a second layer, thereby generating an integrated spatial domain residual value. A method for integrating multiple layers in a multi-layer bitstream comprising:
a) receiving unquantized transform coefficients for a first layer having a first spatial resolution;
b) receiving unquantized transform coefficients for the second layer having the first spatial resolution;
c) scaling the first layer transform coefficients, thereby creating the first layer scaled transform coefficients;
d) integrating the scaled transform coefficients of the first layer with the transform coefficients of the second layer, thereby creating an integrated transform coefficient;
e) inverse transforming the integrated transform coefficients, thereby creating an integrated spatial domain residual value;
f) receiving unquantized transform coefficients for a third layer having a second spatial resolution;
g) resampling the merged spatial domain residual value to a second spatial resolution, thereby creating a consolidated and resampled spatial domain value;
h) inverse transforming the third layer transform coefficients, thereby creating a third layer spatial domain value; and i) integrating the integrated and resampled spatial domain value with the third layer spatial domain value. A method comprising steps. A method for integrating multiple layers in a multi-layer bitstream comprising:
a) receiving quantized transform coefficients for a first layer having a first spatial resolution;
b) receiving quantized transform coefficients for a second layer having the first spatial resolution;
c) scaling the first layer quantized transform coefficients, thereby creating the first layer quantized scaled transform coefficients;
d) integrating the quantized scaled transform coefficients of the first layer with the quantized transform coefficients of the second layer, thereby creating a quantized and unified transform coefficient;
e) dequantizing the quantized and integrated transform coefficients, thereby creating an integrated transform coefficient;
f) inverse transforming the integrated transform coefficients, thereby creating an integrated residual space domain value;
g) receiving quantized transform coefficients for a third layer having a second spatial resolution;
h) resampling the integrated residual spatial domain value to a second spatial resolution, thereby creating an integrated resampled spatial domain value;
i) dequantizing the quantized transform coefficients of the third layer, thereby creating the transform coefficients of the third layer;
j) inverse transforming the third layer transform coefficients, thereby creating a third layer spatial domain value; and k) integrating the integrated and resampled spatial domain value with the third layer spatial domain value. A method comprising steps. A method for integrating multiple layers in a multi-layer bitstream comprising:
a) receiving unquantized transform coefficients for a first layer having a first spatial resolution;
b) inverse transforming the unquantized transform coefficients of the first layer, thereby generating a spatial domain value of the first layer;
c) receiving unquantized transform coefficients for a second layer having a second spatial resolution higher than the first spatial resolution;
d) receiving unquantized transform coefficients for a third layer having a second spatial resolution;
e) up-sampling the spatial domain value of the first layer to a second spatial resolution, thereby generating the up-sampled spatial domain value of the first layer;
f) integrating the non-quantized transform coefficients of the second layer with the non-quantized transform coefficients of the third layer, thereby creating an integrated transform coefficient;
g) inverse transforming the integrated transform coefficients, thereby creating a first integrated spatial domain residual value; and h) a first integrated upsampled spatial domain value of the first layer. Including integrating with a spatial domain residual value. A method for integrating multiple layers in a multi-layer bitstream comprising:
a) receiving quantized transform coefficients for a first layer having a first spatial resolution;
b) receiving quantized transform coefficients for a second layer having a first spatial resolution;
c) receiving quantized transform coefficients for a third layer having a first spatial resolution;
d) scaling the quantized transform coefficients of the first layer to match the characteristics of the second layer, thereby creating a quantized scaled transform coefficient of the first layer;
e) integrating the quantized scaled transform coefficients of the first layer with the quantized transform coefficients of the second layer, thereby creating a quantized and unified transform coefficient;
f) dequantizing the quantized and integrated transform coefficients, thereby creating an integrated transform coefficient;
g) dequantizing the quantized transform coefficients of the third layer, thereby creating a non-quantized transform coefficient of the third layer;
h) integrating the integrated transform coefficients with the non-quantized transform coefficients of the third layer, thereby creating an integrated transform coefficient of the three layers; and i) an integrated transform of the three layers A method comprising the step of inverse transforming the coefficients, thereby creating an integrated spatial domain value. A method for integrating multiple layers in a multi-layer bitstream comprising:
i) determining whether the second layer of the multi-layer image uses residual prediction;
ii) performing the subsequent steps only if the second layer uses residual prediction;
iii) determining a first spatial resolution of the first layer of the multi-layer image;
iv) determining a second spatial resolution of the second layer;
v) comparing the first spatial resolution with the second spatial resolution;
vi) performing steps vii) -viii) if the first spatial resolution is substantially equal to the second spatial resolution;
vii) scaling the first layer transform coefficients to match the characteristics of the second layer, thereby creating scaled transform coefficients for the first layer;
viii) integrating the scaled transform coefficients of the first layer with the transform coefficients of the second layer to form an integrated coefficient;
ix) performing steps x) to xiii) if the spatial resolution of the first layer is not substantially equal to the spatial resolution of the second layer;
x) inverse transforming the transform coefficients of the first layer, thereby generating the spatial domain values of the first layer;
xi) inverse transforming the transform coefficients of the second layer, thereby generating a spatial domain value of the second layer;
xii) scaling the spatial domain value of the first layer to match the resolution of the second layer, thereby generating a scaled spatial domain value of the first layer;
xiii) integrating the scaled spatial domain value of the first layer with the spatial domain value of the second layer, thereby generating an integrated spatial domain value. A method for scaling transform coefficients in a multi-layer bitstream, comprising:
Determining a first layer quantization parameter based on the multi-layer bitstream;
Determining a second layer quantization parameter based on the multi-layer bitstream; and scaling a first layer transform coefficient based on the first layer quantization parameter and the second layer quantization parameter. Including methods. A method for controlling an entropy encoding process comprising:
a) identifying a first neighboring macroblock adjacent to the target macroblock;
b) identifying a second adjacent macroblock adjacent to the target macroblock;
c) determining a first macroblock index indicating whether the first neighboring macroblock is encoded with reference to another layer;
d) determining a second macroblock index indicating whether the second neighboring macroblock is coded with reference to another layer; and e) a first macroblock index and a second macroblock index. Determining an entropy encoding control value based on the method. A method for controlling an entropy encoding process comprising:
a) identifying a first neighboring macroblock adjacent to the target macroblock;
b) identifying a second adjacent macroblock adjacent to the target macroblock;
c) determining whether the first neighboring macroblock is available;
d) determining whether the first neighboring macroblock is encoded in inter-layer prediction mode;
e) determining whether the first neighboring macroblock is encoded in the spatial domain;
f) determining whether the first neighboring macroblock is intra predicted in DC prediction mode;
g) determining whether the first neighboring macroblock is encoded with reference to another layer;
h) if any of steps c) -g) is true, setting the first adjacent block flag to 1;
i) determining whether the second neighboring macroblock is available;
j) determining whether the second adjacent macroblock is encoded in inter-layer prediction mode;
k) determining whether the second neighboring macroblock is encoded in the spatial domain;
l) determining whether the second neighboring macroblock is intra predicted in DC prediction mode;
m) determining whether the second neighboring macroblock is encoded with reference to another layer;
n) if any of steps i) to m) is true, setting a second adjacent block flag value to 1; and o) the first adjacent block flag value and the second adjacent Adding the block flag value to generate an entropy encoding control value. A method for determining a prediction mode, comprising:
a) identifying a first neighboring macroblock adjacent to the target macroblock;
b) identifying a second neighboring macroblock adjacent to the target macroblock; and c) the following conditions i) to vi):
i) the first neighboring macroblock is available;
ii) the first neighboring macroblock is coded in inter-layer prediction mode;
iii) the first neighboring macroblock is encoded with reference to another layer;
iv) a second neighboring macroblock is available;
v) the second neighboring macroblock is coded in inter-layer prediction mode;
vi) the second neighboring macroblock is encoded with reference to another layer;
If any of the above is true, the method includes the step of setting the estimated prediction mode of the target block to a predetermined mode. A method for integrating multiple layers in a multi-layer bitstream comprising:
a) receiving a bitstream that includes encoded image coefficients and encoded block pattern (Cbp) information, wherein the Cbp information identifies an area that includes transform coefficients in the bitstream;
b) decoding the Cbp information;
c) analyzing the bitstream by identifying the bitstream region containing the transform coefficients using the Cbp information;
d) scaling the transform coefficients of the first layer in the bitstream to match the characteristics of the second layer in the bitstream;
e) adding the scaled transform coefficients of the first layer to the transform coefficients of the second layer to form the merged coefficients in the merged layer; and f) transform coefficients in the merged layer A method comprising calculating integrated Cbp information for an integrated layer, which is information identifying a region to include. A system for integrating multiple layers in a multi-layer bitstream,
g) for receiving a bitstream including encoded image coefficients and encoded block pattern (Cbp) information, wherein the Cbp information identifies an area including a transform coefficient in the bitstream. Receiving machine;
h) a decoder for decoding Cbp information;
i) a parser for analyzing the bitstream by identifying the bitstream region containing the transform coefficients using Cbp information;
j) a scaler for scaling the first layer transform coefficients in the bitstream to match the characteristics of the second layer in the bitstream;
k) an adder for adding the scaled transform coefficients of the first layer to the transform coefficients of the second layer to form a merged coefficient in the merged layer; and l) in the merged layer A system including a calculator for calculating integrated Cbp information for an integrated layer, which is information identifying a region containing transform coefficients. A method for selecting a reconstructed transform size when the transform size is not indicated in the enhancement layer,
a) determining the transform size of the lower layer;
b) determining whether the transformation size of the lower layer is substantially similar to a predetermined transformation size;
c) selecting the inverse transform of the predetermined transform size as the reconstruction transform if the transform size of the lower layer is substantially similar to the predetermined transform size; and d) transforming the lower layer Selecting the inverse transform of the default transform size as the reconstruction transform if the size is not substantially similar to the predetermined transform size.

Images