JP2008517499A - Apparatus and method for generating an encoded video sequence using intermediate layer residual value prediction and decoding the encoded video sequence - Google Patents

Abstract

The video coder determines motion data for the enhancement layer using motion data from the base layer and / or motions in the enhancement layer using a sequence of residual pictures from the base layer via an intermediate layer predictor (1018). In order to predict the sequence of residual pictures after compensated prediction, motion compensated prediction (1906, 1012, 1014, 1016) is performed in both the base layer (1002) and enhancement layer (1004). On the decoder side, an intermediate layer combiner is used to cancel this intermediate layer prediction. Thereby, the data rate is improved compared to the scalability scheme without intermediate layer prediction with the same picture quality.
[Selection] Figure 1a

Description

  The present invention relates to a video coding system, and more particularly to a video coding standard H.264. The present invention relates to a scalable video coding system used in connection with H.264 / AVC or a new MPEG video coding system.

  H. The H.264 / AVC standard is the result of a video standardization project between the ITU-T video coding expert group VCEG and the ISO / IEC video coding expert group (MPEG). The main goal of this standardization project is to provide a clear video coding concept that has very good compression characteristics and at the same time generates a video representation that is easy to use on the network, Includes both “conversational” applications and non-conversational applications (save, broadcast, stream transfer).

  Apart from the aforementioned standard ISO / IEC 14496-10, there are several announcements regarding the standard. By way of example only, references include “New H.264-AVC Standard (Emerging H.264-AVC)” by Ralph Schaefer, Thomas Wiegand, and Heiko Schwartz. standard) ”, EBU Technical Review, January 2003. In addition, Thomas Wiegand, Gary J. et al. Gary J. Sullivan, Gesle Bjontegaard and Ajay Lothra, a specialized publication “Overview of H.264 / AVC Video Coding Standard (Overview of the H.264 / AVC) Video Coding Standard), IEEE Transactions on Circuits and Systems for Video Technology (July 2003), and Detlev Marp (D) And Thomas Wiegant (Thomas Wiega) d) Special Publication “Context-based adaptive Binary Arithmetic Coding in the H.264 / AVC Video Compression Standard circuit and video technology in the H.264 / AVC video compression standard”. IEEE Transactions on Systems and Systems for Video Technology (September 2003) for the system gives a detailed overview of the various features of the video coding standard.

  However, for better understanding, an overview of the video encoding / decoding algorithm will be described with reference to FIGS.

  FIG. 9 shows the overall structure of a video coder, which generally consists of two different stages. In general, in the first stage, general video-related operations are performed to generate output data, and this data is entropy encoded in the second stage as indicated by 80 in FIG. This data consists of data 81a, quantized transform coefficients 81b, and motion data 81c. These data 81a, 81b, 81c are sent to the entropy coder 80, and a video signal encoded at the output terminal of the entropy coder 80 is transmitted. Generated.

Specifically, each input video signal is segmented and divided into macroblocks, and each macroblock has 16 × 16 pixels. Next, the macroblock slice group and the association with the slice are selected, whereby each macroblock of each slice is processed for each net of the operation block, as shown in FIG. It should be noted that when various slices exist in a video picture, it is possible to efficiently process macroblocks in parallel. The association of macroblocks with slice groups and slices is performed via the block coder control 82 of FIG. There are various slices, defined as follows:
I slice: An I slice is a slice in which all macroblocks of the slice are encoded using intra prediction.
P slice: A specific macroblock of a P slice can be encoded using inter prediction with at least one motion compensated prediction signal for each prediction block in addition to the I slice encoding type.
B slice: Specific macroblocks of a B slice can be encoded using inter prediction with two motion compensated prediction signals per prediction block in addition to the coder types available in P slices.

The above three coder types are very similar to the previous standard types, except that they use reference pictures as described below. The next two coder types are H.264 and H.264. This is new in the H.264 / AVC standard.
SP slice: It is also referred to as a switch P slice and is coded so that it can be efficiently switched between different pre-coded pictures.
SI slice: The SI slice, also referred to as the switch I slice, allows precise adaptation of macroblocks in the SP slice for direct random access and error recovery purposes.

  Generally speaking, a slice is a sequence of macroblocks, which are processed in raster scan order when the flexible macroblock order FMO feature is not used, and is also defined in the standard. As shown in FIG. 11, a picture can be partitioned into one or several slices. Thus, a picture is a collection of one or several slices. The syntax element of the slice can be analyzed from the bitstream (syntax analysis). If the reference pictures used for the coder and the decoder are the same, the slice can be analyzed without requiring data from other slices. In this sense, the slices are independent of each other because the sample values can be correctly decoded within the range of the picture to be represented. However, since a deblocking filter between slice boundaries is used, specific information from other slices may be required.

  The FMO function changes the method of partitioning a picture into slices and macroblocks using the concept of slice groups. Each slice group is a set of macroblocks defined by mapping from macroblocks to slice groups, and is specified by a picture parameter set and specific information from the slice header. This mapping of macroblocks to slice groups is made up of slice group identification numbers for each macroblock in the picture, and this identifies the slice group to which the associated macroblock belongs. Each slice group can be partitioned into one or several slices, where a slice is a sequence of macroblocks within the same slice group and in the order of raster sampling within the set of macroblocks of individual slice groups. It is processed.

Each macroblock may be transmitted in one or several coder types depending on the slice coder type. In all slice coder types, a type of intra coding called intra- ₄ × ₄ or intra- ₁₆ × ₁₆ is supported, and in addition, chroma prediction mode and I 2 _-PCM prediction mode are supported.

The Intra- ₄ × ₄ mode is based on individual prediction of each 4 × 4 chroma block and is suitable for encoding a distinctly detailed picture portion. On the other hand, the intra- ₁₆ × ₁₆ mode is more suitable for performing prediction of the entire 16 × 16 chroma block and coding the “soft” region of the picture.

In addition to these two chroma prediction types, another chroma prediction is performed. As the type of alternative to intra _-4 × ₄ and intra _-16 × _16, I in _-4 × ₄ coder type coder prediction and transform coding simply skips, send the value of the coded samples instead directly can do. The I- _PCM mode has the following purposes. That is, the mode allows the coder to accurately represent sample values. The mode provides a way to accurately represent very irregular screen content values without enlarging the data. In addition, the mode allows the coder to determine the strict boundary for the number of bits it needs to handle a macroblock without compromising coding efficiency.

  In contrast to previous video coding standards in which intra prediction was performed in the transform domain (ie H.263 plus and MPEG-4 visuals). Intra prediction in H.264 / AVC is always performed in the spatial domain with reference to adjacent samples of already coded blocks located respectively on the left and top of the predicted block (FIG. 10). In certain situations, such as when transmission and reception errors occur, this can lead to error propagation, which is caused by motion compensation in intra-coded macroblocks. As described above, there is a limit to signalization by the intra coding mode, and it is only possible to predict adjacent macroblocks that are intra-coded.

When using the intra- ₄ × ₄ mode, 4 × 4 blocks of each spatially adjacent sample are predicted. Using the previously decoded samples that are in adjacent blocks, 16 samples of a 4 × 4 block are predicted. Any one of nine prediction modes can be used for each 4 × 4 block. In addition to “DC prediction” (which predicts the entire 4 × 4 block using a certain value), 8-way prediction modes are specified. These modes are suitable for predicting directional structures in pictures such as edges at different angles.

  In addition to the intra macroblock coder type, various prediction or motion compensation coder types are identified as P macroblock types. Any type of P macroblock corresponds to a particular partition that partitions the macroblock into block forms, which are used for motion compensated prediction. Partitions of samples with 16 × 16, 16 × 8, 8 × 8 or 8 × 16 luma block sizes are supported in the syntax. For 8 × 8 sample partitions, an additional syntax element is transmitted for each 8 × 8 partition. This syntax element specifies whether each 8 × 8 partition is further partitioned into 8 × 4, 4 × 8 or 4 × 4 luma samples and corresponding chroma sample partitions.

  A prediction signal for each predictive-coded M × M luma block is obtained by shifting the area of each reference picture specified by the translational motion vector and the picture reference index. Therefore, if a macroblock is encoded using four 8x8 partitions, and if each 8x8 partition is partitioned into four 4x4 partitions, the maximum amount for a single P macroblock Sixteen motion vectors can be transmitted in a so-called motion field.

  H. In H.264 / AVC, the quantization of the transform coefficient is calculated using the quantization parameter slice QP. This parameter can take 52 values. These values are processed so that an increase of 1 for the quantization parameter results in an increase of about 12% of the quantization step width. This means that if the quantization parameter is increased by 6, the quantization step width is exactly doubled. Note that a step size change of about 12% also means that the bit rate is reduced by about 12%.

  The quantized transform coefficients of the block are generally sampled in a zigzag path and processed using an entropy coding method. The 2 × 2 DC coefficient of the chroma component is sampled in a raster scan sequence and All inverse transform operations within H.264 / AVC can be performed using only 16-bit integer value addition and shift operations.

  Referring to FIG. 9, the input signal is first partitioned into 16 × 16 pixel macroblocks for each picture of the video sequence for every picture. Next, each picture is sent to a subtractor 84, which subtracts the original picture supplied from the decoder 85 built in the coder from the picture. The result of the subtraction is the residual signal in the spatial domain, which is then transformed, scaled and further quantized (block 86) to obtain quantized transform coefficients on line 81b. In order to generate a subtraction signal that is supplied to the subtractor 874, the quantized transform coefficients are first rescaled and further inverse transformed (block 87) and sent to the adder 88, whose output is deblocking filter 89. To be supplied. This output video signal is decoded, for example by a decoder, and can be monitored at the output of the deblocking filter, for example for control (output 90).

  Motion estimation is performed in block 91 using the decoded output signal at output 90. As seen in FIG. 9, for motion estimation in block 90, a picture of the original video signal is provided. This standard allows two different motion estimations: forward motion estimation and backward motion estimation. In forward motion estimation, the current picture motion is estimated with respect to the previous picture. On the other hand, in the backward motion estimation, the motion of the current picture is estimated using a future picture. The result of the motion estimation (block 91) is sent to the motion compensation block 92, and specifically, when the switch 93 is switched to the inter prediction mode as in FIG. 9, the motion compensation inter prediction is performed. Executed. On the other hand, when the switch 93 is switched to intra frame prediction, intra frame prediction is executed using the block 490. Therefore, no motion data is required because motion compensation is not performed in intra-frame prediction.

  The motion estimation block 91 generates motion data and a motion field, respectively. The motion data and the motion field are composed of motion vectors, which are transmitted to a decoder, respectively, and corresponding inverse predictions, that is, transform coefficients and Reconstruction using motion data can be performed. In the case of forward prediction, a motion vector can be calculated from the immediately preceding picture and several preceding pictures. In the case of backward prediction, the current picture can be calculated using the future picture immediately adjacent to it and, of course, the future picture.

  A disadvantage of the video coding concept shown in FIG. 9 is that it does not have simple scalability. As is known in the art, the term “scalability” refers to a coder / decoder concept in which a coder transmits a scaled data stream. The scaled data stream includes a base scaling layer and one or several enhanced scaling layers. The base scaling layer typically includes a representation of the encoded signal that is of low quality but also has a low data rate. The enhanced scaling layer includes a further representation of the video signal, and usually provides an improved quality representation for the base scaling layer in conjunction with the representation of the video signal in the base scaling layer. On the other hand, the extended scaling layer naturally has individual bit requirements, and the number of bits representing a signal to be encoded increases each time the layer is expanded.

  Depending on design and performance, the decoder decodes only the base scaling layer and provides a relatively poor quality representation of the picture signal represented by the encoded signal. However, as each additional scaling layer is "added", the decoder improves the signal quality in stages (at the expense of bit rate).

  Depending on the implementation and the transmission path from the coder to the decoder, at least only the base scaling layer is transmitted, because the bit rate of the base scaling layer is usually very low, so a rather limited transmission path is sufficient. If the transmission path does not have any more bandwidth for the application, only the base scaling layer will be transmitted and the enhanced scaling layer will not be transmitted. As a result, the decoder can only generate a low quality representation of the picture signal. Compared to the case where the scaling is not performed and the data rate is too high for the transmission system to support, the low quality representation has some advantages. In the case of a transmission path that allows transmission of one or several enhancement layers, the coder will send one or several enhancement layers to the decoder, step by step of the output video signal as needed. Quality can be improved.

  A distinction can be made between two different scalings for the coding of video sequences. One scaling is time scaling, which does not transmit all video frames of the video sequence, but only transmits every second, every third, every fourth, etc., to reduce the data rate.

  Another scaling is SNR scalability (SNR = signal-to-noise ratio) where each scaling layer, eg, base scaling layer, and first, second, third,. Time information, but the quality is different. Thus, the base scaling layer has a low data rate but a low signal to noise ratio, and this signal to noise ratio can be improved by adding an extended scaling layer step by step to it.

  The coder concept shown in FIG. 9 is problematic in that it is based on the fact that only the residual value is generated by the subtractor 84 and then processed. These residual values are calculated based on the prediction algorithm in the apparatus shown in FIG. 9, which forms a closed loop using blocks 86, 87, 88, 89, 93, 94 and 84. , The quantization parameters are input to this closed loop, i.e. to blocks 86,87. Thus, for example, when simple SNR scalability is implemented, every prediction residual signal is first quantized with a coarse quantization step width, and then with an enhancement layer, stepwise finer quantization step width is achieved. Quantization will result in the following results: Due to the dequantization and prediction, in particular with respect to motion estimation (block 91) and motion compensation (block 92) performed using the original picture on the one hand and the quantized picture on the other hand, both the coder and the decoder "Divergence" of the step width of the generator occurs. This creates a serious problem in generating the extended scaling layer on the coder side. Further, the processing of the extended scaling layer on the decoder side is at least H.264. This is not possible for elements defined in the H.264 / AVC standard. The reason for this lies in the closed loop including the quantization in the video coder shown in FIG. 9 as described above.

  Standardized document JVT of Heiko Schwartz, Detlev Marpe and Thomas Wiegand presented at the 9th JVT meeting in San Diego, December 2-5, 2003 -I 032 t1, the title “H.264 / AVC SNR Scalable Extension of SNR-Scalable Extension of H.264 / AVC”. A scalable extension to H.264 / AVC has been announced, which includes scalability in terms of both time and signal-to-noise ratio (with comparable or different temporal accuracy). There, a lifting representation of temporal subband partitions is introduced, and a known method can be used for motion compensation prediction.

  J. et al. -R. Ohm, "Complexity and delay analysis of MCTF interframe wave structure", ISO / IECJTC1 / WG11 Doc. In M8520, July 2002, wavelets based on video coder algorithms are described, and lifting implementations are used for wavelet analysis and wavelet synthesis. D. Taubman's “Sequential Improvement of Video: Basic Issues, Previous Efforts, and New Directions (Successive refinements of video: past effects and new directions)”, SPIE (3), Proe. There is also a comment on scalability in the year 5150, pages 649-663, but it is stated that significant changes to the coder are necessary. According to the present invention, a coder / decoder concept is realized which, on the one hand, retains scalability possibilities and on the other hand can be based on elements that conform to the standard, in particular for motion compensation.

  Before referring to further details of the coder / decoder with respect to FIG. 3, first the basic lifting scheme on the coder side and the inverse lifting scheme on the decoder side will be described respectively with respect to FIG. For a detailed description of the background of the combination of lifting schemes and wavelet transforms, see W.W. Sweldens, “A custom design of construction of biological wavelets”, J. Org. Appl. Comp. Harm. Anal. 1996, Volume 3 (No. 2), pages 186-200, and I.C. Daubechies and W. Swelldens, “Factoring wavelet transforms into lifting Steps”, J. Am. Fourier Anal. Appl. 1998, Vol. 4 (No. 3), pages 247-269. In general, a lifting scheme is composed of three steps: a multiphase decomposition step, a prediction step, and an update step.

The disassembling step includes partitioning the input data stream into the same first copy for the lower branch 40a and the same copy for the upper branch 40b. Further, identical copies of the upper branch 40b is delayed by a time stage (z ^-1), and the sample S _2k with sample S _{2k + 1} and the even index having odd indices k, likewise, the respective decimator and down Pass through the samplers 42a and 42b, respectively. Decimators 42a and 42b reduce the number of samples in the upper and lower branches 40b and 40a, respectively, by eliminating every second sample.

The second region II is related to the prediction step and includes a prediction operator 43 and a subtracter 44. The third region is an update step and includes an update operator 45 and an adder 46. On the output side, there are two normalizers 47 and 48, which normalize the high-pass signal h _k (normalizer 47) and normalize the low-pass signal l _k through the normalizer 48.

となる。 Specifically, in multiphase decomposition, the predetermined signal s [k] is partitioned into even and odd samples. Since the correlation structure usually exhibits local features, even and odd polyphase components are highly correlated. Thus, in the final step, odd sample prediction (P) is performed using integer samples. The prediction operator (P) corresponding to each odd sample s _odd [k] = s [2k + 1] is a linear combination of adjacent even samples s _even [k] = s [2k], ie

It becomes.

となる。 As a result of the prediction step, odd samples are replaced with their respective prediction residual values,

It becomes.

  In addition, I.I. Daubechies and W. Swelldens, “Factoring wavelet transforms into lifting steps”, J. Am. Fourier Anal. Appl. 1998, Volume 4 (No. 3), pages 247-269, the prediction step is equivalent to performing high-pass filtering with a two-channel filter bank.

により与えられる。 In the third step of the lifting scheme, low-pass filtering is performed by replacing the even sample s _even [k] with a linear combination of predicted residual values h [k]. Each update operator U is given by

Given by.

で置き換えることによって、所定の信号ｓ［ｋ］を、最終的にｌ（ｋ）およびｈ（ｋ）で表すことができ、各信号は、半分のサンプルレートを有する。更新ステップおよび予測ステップの双方は完全に可逆的なので、対応する変換は、厳密にサンプリングされた完全な再構成フィルタバンクと解釈することができる。実際に、ウェーブレットフィルタの一切の双直交ファミリを、１つまたはいくつかの予測ステップおよび１つまたはいくつかの更新ステップによって、実現することができる。前記で説明したように、ローパスおよびハイパス成分を正規化するため、正規化器４７および４８には、適切に選択されたスケーリングファクタＦ_lおよびＦ_hが供給される。 The even number sample is

, The given signal s [k] can ultimately be represented by l (k) and h (k), each signal having a half sample rate. Since both the update and prediction steps are completely reversible, the corresponding transform can be interpreted as a perfectly sampled fully reconstructed filter bank. In fact, any bi-orthogonal family of wavelet filters can be realized with one or several prediction steps and one or several update steps. As explained above, the normalizers 47 and 48 are supplied with appropriately selected scaling factors F _l and F _h to normalize the low-pass and high-pass components.

The inverse lifting scheme corresponds to a synthesis filter bank and is shown on the right side of FIG. It consists simply of the application of prediction and update operators in reverse order in reverse order, followed by reconstruction with even and odd polyphase components. Specifically, the decoder on the right side of FIG. 4 again includes a first decoder region I, a second decoder region II, and a third decoder region III. In the first decoder area, the action of the update operator 45 is cancelled. This is accomplished by supplying the update operator 45 with a high pass signal that has been normalized by a further normalizer 50. Next, the output signal of the update operator 45 on the decoder side is supplied to a subtracter 52 as opposed to the adder 46 in FIG. Similarly, the output signal of the predictor 43 is processed, and the output signal is then supplied to the adder 53 instead of the subtracter as on the coder side. Next, upsampling of the signal by factor 2 is performed in each branch (blocks 54a, 54b). The upper branch is then shifted forward by one sample, which is the same as delaying the lower branch, with the adder 55 performing the addition of the data streams on the upper and lower branches. And a reconstructed signal _sk is obtained at the output of the synthesis filter bank.

ここで、

であり、これらの式は、それぞれ、ヘアーフィルタの非正規化ハイパスおよびローパス（分析）出力信号に対応する。 Several wavelets can be implemented by the predictor 43 and the update operator 45, respectively. When implementing the so-called hair wavelet, the prediction and update operators are given by

here,

And these equations correspond to the unnormalized high pass and low pass (analysis) output signals of the hair filter, respectively.

で与えられる。 In the case of a 5/3 biorthogonal spline wavelet, the low pass and high pass analysis filters of this wavelet have 5 and 3 filter taps, respectively, and the corresponding scaling function is a quadratic B spline. In a coder application for still pictures, such as JPEG2000, this wavelet is used in a temporal subband coder scheme. In the lifting environment, the corresponding 5/3 transformation prediction and update operators are:

Given in.

FIG. 3 is a block diagram of a coder / decoder structure in which both the coder side and the decoder side have typical four filter levels. As for the coder, it can be seen from FIG. 3 that the first filter level, the second filter level, the third filter level, and the fourth filter level are the same. The filter levels are the same for the decoder. On the coder side, each filter level includes a backward predictor M _i0 and a forward predictor M _i1 61 as central elements. The backward predictor 60 corresponds in principle to the predictor 43 in FIG. 4, and the forward predictor 61 corresponds to the update operator in FIG.

  Note that, unlike FIG. 4, FIG. 4 relates to a stream of samples, where one sample has an odd index 2k + 1 and another sample has an even index 2k. However, as already described with respect to FIG. 1, the notation of FIG. 3 relates to a group of pictures, not a group of samples. If a picture has, for example, several samples and pictures respectively, the entire picture is sent. Next, the next picture is also sent, and so on. Thus there are no more odd and even samples, but there are odd and even pictures. According to the present invention, the lifting scheme described for odd and even samples is applied to odd and even pictures, respectively, each of these pictures having multiple samples. 4 is the backward motion compensated prediction 60, and the update operator 45 for each sample is the forward motion compensated prediction 61 for each picture.

Note that the motion filter is composed of motion vectors and represents the coefficients for blocks 60 and 61, is calculated for two consecutive related pictures, and is further transmitted as side information from the coder to the decoder. However, as described with reference to FIG. The ability to easily calculate the motion field M _i0 and motion field M _i1 using the elements 91 and 92 standardized in the H.264 / AVC standard is a major advantage of the inventive concept. Thus, the concept of the present invention does not require the use of new predictor / update operators, but replaces existing algorithms that have been verified and verified for functionality and efficiency and described in the video standard either forward or backward. Can be used for motion compensation.

Specifically, the general structure of the used filter bank shown in FIG. 3 shows the temporal decomposition of the video signal of the group of 16 pictures sent to the input 64. This decomposition is a binary temporal decomposition of the video signal, which is shown in FIG. 3 and in the embodiment with ^four levels 2 ⁴ = 16 pictures, ie the group size embodiment of 16 pictures. It is necessary to achieve a representation with a temporal resolution of i.e. signals from the outputs 28a and 28b. Thus, grouping 16 pictures leads to a delay of 16 pictures, which causes various problems for interactive applications with the four level concept shown in FIG. Thus, if targeting an interactive application, it is desirable to form smaller groups of pictures, such as groups of 4 or 8 pictures. Secondly, the delay is reduced accordingly and it can be used for interactive applications. If bi-directionality is not required, such as for storage purposes, the number of pictures in the group, ie the group size, can be increased to a number such as 32, 64, etc. accordingly.

Thus, an interactive application of a hair-based motion compensation lifting scheme is used, which is The method further includes an updating step comprising backward motion compensation prediction (M _i0 ) as in H.264 / AVC and including forward motion compensation (M _i1 ). Both the prediction step and the update step are H.264. As shown in H.264 / AVC, motion compensation processing is used. Further, not only motion compensation but also a deblocking filter 89 indicated by reference numeral 89 in FIG. 9 is used.

  Similarly, the second filter level adds down samplers 66a and 66b, a subtractor 69, a backward predictor 67, a forward predictor 68 and an adder 70, and first and second high pass pictures of the second level. The second level first and second low-pass pictures are output to the output terminal of the adder 70, including the additional processing means for outputting to the output of the processing means.

  Further, the coder of FIG. 3 includes a third level and a fourth level, and a group of 16 pictures is sent to the input 64 of the fourth level. At the fourth level, eight high-pass pictures quantized by the quantization parameter Q and processed in accordance with the quantization parameter Q are output from the high-pass output 72, also referred to as HP4. Similarly, the eight low-pass pictures are output to the low-pass output end 73 of the fourth filter level and sent to the input end 74 of the third filter level. This level is also useful for generating four high-pass pictures at the high-pass output 75, also referred to as HP3, and for generating four low-pass pictures at the low-pass output 76, which is the input of the second filter level. It is sent to the end 10 and disassembled.

  Here, the group of pictures processed at a certain filter level does not necessarily need to be a video picture directly derived from the original video sequence, and is output from the low-pass output terminal of the filter level by the filter level one stage above. It can be a low-pass picture.

  Furthermore, the coder concept for the 16 pictures shown in FIG. 3 can be easily reduced to 8 pictures by simply deleting the fourth filter level and sending the group of pictures to the input terminal 74. Similarly, the concept shown in FIG. 3 is added to the fifth filter level, the 16 high pass picture is output from the high pass output end of the fifth filter level, and the 16 low pass picture at the output end of the fifth filter level is output to the fourth filter level. Can be expanded to a group of 32 pictures.

  A coder-side tree-like concept is applied to the decoder side, but this time it is not from the high level to the lower level as in the coder side, but the decoder side performs processing from the low level to the higher level. Become. For this reason, the data stream is received from a transmission medium generally called the network abstraction layer 100, but the received bit stream is first subjected to the reverse addition processing using the reverse addition processing means, as shown in FIG. A reconstructed version of the first level low pass picture is obtained at the output end of the means 30a and a first level low pass picture at the output end of the block 30b of FIG. Next, in a manner similar to the right half of FIG. 4, first the forward motion compensated prediction is inverse processed through the predictor 61, and then the output signal of the predictor 61 is from the reconstructed version of the low pass signal. Subtracted (subtractor 101).

The output signal of the subtractor 101 is fed into the backward compensated predictor 60 where a prediction result is generated and added by the adder 102 to the reconstructed version of the high pass picture. Next, both signals in the lower branches 103a, 103b are brought to a double sample rate using the upsamplers 104a and 104b, where the signals of the upper branch are either delayed or [Accelerated] Note that the upsampling is performed by the bridges 104a and 104b simply by inserting a number of zeros corresponding to the number of samples per picture. In the upper branch 103b, since the picture is delayed and shifted by the element indicated by z ⁻¹ with respect to the lower branch 103a, the addition of the adder 106 causes two second-level low-pass pictures to be output on the output side of the adder 106. It occurs continuously.

  Next, the reconstructed versions of the first and second second level low pass pictures are fed into a second level inverse filter on the decoder side and transmitted by the same implementation inverse filter bank. And a sequence of four third level low pass pictures at the second level output 101. The four third level low pass pictures are then combined with the transmitted third level high pass picture by the third level inverse filter level and connected to the output 110 of the third level inverse filter in succession. Obtain eight fourth level low pass pictures of the format. These eight third level low pass pictures are then combined with the eight fourth level high pass pictures received from the transmission medium 100 via the input end HP4 by a fourth level inverse filter as well. As described with respect to the first level, a group of 16 reconstructed pictures is obtained at the output 112 of the fourth level inverse filter.

  In this way, for each stage of the analysis filter bank, any two pictures out of the plurality of original pictures or the plurality of pictures representing the low-pass signal generated at the level one higher are the low-pass signal and the high-pass signal. Broken down into signals. The low-pass signal can be considered to express a feature common to the input pictures, and the high-pass signal can be considered to express a difference between the input pictures. At the corresponding stage of the synthesis filter bank, the two input pictures are reconstructed again using the low pass signal and the high pass signal. In the synthesis step, the inverse operation of the analysis step is performed and a complete reconstruction is guaranteed by the analysis / synthesis filter bank (which naturally does not include quantization).

  The loss that occurs is only due to quantization in the additional processing means such as 26a, 26b, 18. With very fine quantization, a good signal to noise ratio is achieved. Conversely, if the quantization is performed very roughly, the signal to noise ratio will be relatively poor, but on the other hand, a low bit rate or low bit requirement will be achieved.

  In the concept shown in FIG. 3, time scaling control can be performed without SNR scalability. A time scaling control 120 is then used, which takes on its input the respective outputs of the high pass and the low pass, as well as the respective outputs from the additional processing means (26a, 26b, 18...) The data stream TP1, HP1, HP2, HP3, HP4 is formed to generate a scaled data stream. The partial data stream has a processed version of the first low pass picture and the first high pass picture in the base scaling layer. The processed version of the second high pass picture can then be adapted to the first enhancement scaling layer. The processed version of the third level high pass picture can be adapted to the second enhancement scaling layer, and the processed version of the fourth level high pass picture is introduced to the third enhancement scaling layer. This makes it possible for the decoder to first generate a sequence of low-pass pictures of low quality and low level, i.e. two first level low pass pictures for each group of pictures, based only on the base scaling layer. . With each enhancement scaling layer added, the number of reconstructed pictures per group is always doubled. Typically, the decoder functionality is controlled by scaling control, which controls how many scaling layers are included in the data stream and how many scaling layers the decoder needs to consider when decoding. Each is formed to detect.

  JVT of Heiko Schwarz, Detlev Marpe and Thomas Wiegand announced at the 10th JVT Conference in Waikoloa, Hawaii, December 8-12, 2003 Document JVT-J 035, entitled “SNR-Scalable Extension of H.264 / AVC”, describes the SNR scalable extension of the temporal decomposition scheme illustrated in FIG. 3 and FIG. is doing. Specifically, the time-scaling layer is partitioned into separate “SNR scaling sublayers”, and the SNR base layer is quantized with a first coarser quantization step width to form the SNR base layer. To get. Next, dequantization is first performed, and the difference signal is obtained by subtracting the signal resulting from the dequantization from the original signal, and then this is quantized with a finer quantization step width so that the second scaling layer is can get. However, the second scaling layer is re-quantized with its finer quantization step width, and the signal obtained after re-quantization is subtracted from the original signal to obtain a further difference signal, which again is more The second SNR scaling layer and the SNR enhancement layer are each expressed by quantization with a fine quantization step width.

  Thus, it can be seen that the above-described scalability scheme based on motion compensated temporal filtering (MCTF) has high flexibility with respect to temporal scalability and SNR scalability. However, when coding a picture with the highest quality, the problem remains that the total bit rate of these multiple scaling layers far exceeds the bit rate achievable without scalability. Due to side information for these multiple scaling layers, the scalable coder may not be able to achieve the bit rate at all when not scaled. However, the bit rate of a data stream with multiple scaling layers should be as close as possible to the bit rate without scaling.

  In addition, the scalability concept should provide a high degree of flexibility for all scalability types, ie both time and space and SNR.

  Low resolution pictures are already sufficient, but high flexibility is particularly important when higher temporal resolution is required. This situation arises, for example, due to fast changes occurring in pictures such as team sports videos where many people move in addition to the ball.

"Emerging H.264-AVC standard", EBU technical review (Eal technical T) (EBU technical review), EBU technical review (EBU technical T) Review), January 2003 Thomas Wiegand, Gary J. Gary J. Sullivan, Gesle Bjontegaard and Ajay Rothra, a specialized publication “Overview of H.264 / AVC Video Coding Standard (Overview of the H.264 / AVC) Video Coding Standard), IEEE Transactions on Circuits and Systems for Video Technology, July 2003. Special publication of Detlev Marpe, Heiko Schwartz, and Thomas Wiegand, “Context-based adaptive binary arithmetic coding in the H.264 / AVC video compression standard. Binary Arithmetic Coding in the H.264 / AVC Video Compression Standard), IEEE Transactions on Circuits and Systems for 3 Years, "H.264 / AVC's SNR Scalable Extension 9 (SNR-Scalable Extension of AV.264 / Hecto Schwarz), Detlev Marpe and Thomas Wiegand" JVT-I 032 t1, Standards Document for the 2nd JVT Conference, December 2-5, 2003, San Diego J. et al. -R. Ohm, "Complexity and delay analysis of MCTF interframe wave structure", ISO / IECJTC1 / WG11 Doc. M8520, July 2002 D. Taubman's “Sequential Improvement of Video: Basic Issues, Previous Efforts, and New Directions (Successive refinements of video: past effects and new directions)”, SPIE (VIP3, Pro 03) Year, 5150, 649-663 W. Sweldens, “A custom design of construction of biological wavelets”, J. Org. Appl. Comp. Harm. Anal. 1996, Volume 3 (No. 2), pages 186-200. I. Daubechies and W. Swelldens, “Factoring wavelet transforms into lifting Steps”, J. Am. Fourier Anal. Appl. 1998, Volume 4 (No. 3), pages 247-269. "H.264 / AVC's SNR Scalable Extension (SNR-Scalable Extension of AV.264 / HNR 264 / AVC)" by Heiko Schwarz, Detlev Marpe and Thomas Wiegand. JVT Document JVT-J 035 at the 8th JVT Conference, December 8-12, 2003, Hawaii, Waikoloa

  The object of the present invention is to provide a concept for flexible encoding / decoding that provides the lowest possible bit rate, despite the fact that it is a scalable concept.

  The object is to provide an apparatus for generating an encoded video sequence according to claim 1, a method for generating an encoded video sequence according to claim 15, and a decoding of an encoded video sequence according to claim 16. An apparatus for converting, a method for decoding an encoded video sequence according to claim 26, a computer program according to claim 27 or a computer readable medium according to claim 28.

  The present invention not only provides the bit rate reduction by motion compensated prediction performed in the scaling layer, but also allows further bit rate reduction with constant picture quality to be achieved for lower layers such as the base layer, for enhancement layers. This is based on the knowledge that it is obtained by performing intermediate scaling layer prediction of residual pictures after motion compensated prediction to such higher layers.

  Different scaling layers considered individually, which are preferably scaled with respect to resolution or signal to noise ratio, but these residual values are between the residual values after motion compensated prediction within the same temporal scaling layer It was discovered that there is a correlation. In the present invention, by providing an intermediate layer predictor corresponding to the intermediate layer combiner on the decoder side on the coder side with respect to the enhanced scaling layer, these correlations are advantageously utilized. Desirably, this intermediate layer predictor is adaptively designed, for example, for each macroblock, whether intermediate layer prediction is valuable, or does prediction lead to an increase in bit rate rather? It is to decide whether. The latter case is when the prediction residual signal is larger than the original motion compensation residual signal of the enhancement layer with respect to the subsequent entropy coder. However, this situation does not occur in many cases and the intermediate layer predictor is activated resulting in significant bit rate reduction.

  Furthermore, in a preferred embodiment of the present invention, prediction of enhancement layer motion data is also performed. Thus, it relates to bit rate reduction according to the present invention by providing motion data predictors in which motion fields in different scaling layers are correlated with each other in different quality scaling layers, as for SNR or resolution. It is further shown that it can be used advantageously. In this implementation, the prediction can be performed in that separate motion data is not calculated for the enhancement layer, but the base layer motion data is ultimately transmitted after upsampling. However, this results in the fact that the motion compensation residual signal in the enhancement layer is larger than if motion data is calculated specifically for the enhancement layer. However, this disadvantage does not make a difference if the savings due to motion data saved for the enhancement layer during transmission is greater than the bit rate increase possibly caused by a larger residual value.

  However, a separate motion field can be calculated for the enhancement layer in the implementation, and the base layer motion field is incorporated into the calculation or used as a predictor that transmits only the motion field residual values. This implementation has the effect that the motion data correlation of the two scaling layers is fully utilized and the residual value of the motion data is as small as possible after motion data prediction. However, a disadvantage of this concept is the fact that additional motion data residual values must be transmitted.

  In a preferred embodiment of the present invention, SNR scalability is further used. This means that the quantization is performed at the base layer with coarser quantization parameters than at the enhancement layer. The residual value of the base motion prediction quantized and reconstructed with a coarser quantizer step width is thereby used as a prediction signal for the intermediate layer predictor. For pure SNR scalability it may be sufficient to calculate one motion field for all scaling layers on the coder side. For enhancement layer motion data, this also does not require transmission of further enhancement motion data, but the enhancement motion data from the base layer can be fully used on the coder side for inverse motion compensation for the enhancement layer. Means that. However, when motion data computation is used, different quantization parameters result in different motion fields in which the quantization parameters are introduced.

  When spatial scalability is used, it means that the base scaling layer has a coarser spatial resolution than the enhanced scaling layer, meaning that it is converted from the lower spatial resolution of the enhanced scaling layer and then provided to the intermediate layer predictor It is preferable to interpolate the residual value of the base motion prediction.

  Furthermore, it is preferable to perform separate calculation of motion information for each scaling layer. However, in a preferred embodiment of the invention, motion data prediction is used here for data rate reduction, which is to send only motion data residual values that require a lower data rate than unpredicted motion data. In full transmission of motion data of the lower scaling layer (after scaling), or use a lower scaling layer upsampled motion vector to predict a higher scaling layer motion vector It can consist of things. In this case, it is preferable to adaptively design both the intermediate layer predictor and the extended motion data predictor.

  In a preferred embodiment of the invention, the base scaling layer and the enhanced scaling layer are combined in that they differ in spatial resolution and in the used quantization parameter, which means the used quantizer step width. Scalability is used. In this case, for example, starting from the previous quantization parameter for the base scaling layer with Lagrange optimization, the combination of the quantization parameter for the base layer, the distortion for the base layer motion data and the bit requirement is calculated. The residual values obtained after motion compensated prediction and the base motion data used are thereby used for the prediction of the respective data of the higher scaling layer, and finer scaling parameters for the higher scaling layer. Beginning again, motion data bit requirements, quantization parameter and distortion combinations, and extended motion data can be calculated.

Preferred embodiments of the present invention are described below with reference to the accompanying drawings, which include:
FIG. 1a is a preferred embodiment of the coder of the present invention,
FIG. 1b is a detailed representation of the base picture coder of FIG.
FIG. 1c is an illustration of the function of the intermediate layer prediction flag,
FIG. 1d is an illustration of the motion data flag,
FIG. 1e is a preferred implementation of the extended motion compensator 1014 of FIG.
FIG. 1f is a preferred implementation of the extended motion data determining means 1078 of FIG.
FIG. 1g is a schematic representation of three preferred embodiments for calculating extended motion data and, if necessary, processing extended motion data for signaling and residual data transmission,
FIG. 2 is a preferred embodiment of the decoder of the present invention,
FIG. 3 is a block diagram of a decoder having four levels,
FIG. 4 is a block diagram for explaining lifting decomposition of a temporal subband filter bank,
FIG. 5a is a functional representation of the lifting scheme shown in FIG.
FIG. 5b is a representation of two preferred lifting specifications with unidirectional prediction (hair wavelet) and bi-directional prediction (5/3 transform),
FIG. 5c is a preferred embodiment of a prediction and update operator with motion compensation and reference index for arbitrary selection of two pictures processed by a lifting scheme;
FIG. 5d is a representation of an intra mode in which original picture information can be inserted into a high-pass picture for each macroblock,
FIG. 6a is a schematic representation for signaling the macroblock mode,
FIG. 6b is a schematic representation for upsampling motion data in spatial scalability according to a preferred embodiment of the present invention;
FIG. 6c is a schematic representation of the data stream syntax for motion vector differences;
FIG. 6d is a schematic representation of a residual value syntax extension according to a preferred embodiment of the present invention,
FIG. 7 is a schematic diagram for explaining a time shift of a group of 8 pictures, for example.
FIG. 8 is a preferred temporal arrangement of low-pass pictures for a group of 16 pictures,
FIG. 2 is a schematic block diagram for explaining a basic coder structure for a coder according to the H.264 / AVC standard;
FIG. 10 is a context array composed of two pixel elements A and B adjacent to the left and top of the current syntax element C, respectively.
FIG. 11 is a representation of a partition into a slice of a picture.

  FIG. 1a shows a preferred embodiment of an apparatus for generating an encoded video sequence having a base scaling layer and an enhanced scaling layer. The original video sequence with 8, 16 or any number of groups of pictures is fed via input 1000. On the output side, the encoded video sequence includes a base scaling layer 1002 and an enhanced scaling layer 1004. The extended scaling layer 1004 and the base scaling layer 1002 are supplied to a bitstream multiplexer that generates a single scalable bitstream on the output side. However, depending on the implementation, separate transmissions of the two scaling layers are possible and are useful in some cases. FIG. 1a shows a coder for generating two scaling layers: a base scaling layer and an enhanced scaling layer. In order to obtain a coder that generates one or several further enhancement layers, if necessary, the enhancement scaling layer functions such that the enhancement scaling layer 1004 shown in FIG. Are repeated, and data is always supplied to the higher-order extended scaling layer by the extended scaling layer one level below.

  Before referring to the contents of various scaling types such as SNR scalability, or spatial scalability, or scalability combining spatial and SNR scalability, the basic principle of the present invention will be described first. Initially, the coder includes a base motion compensator or base motion estimator 1006 for calculating base motion data, which is related to another picture in the group of pictures that the base motion compensator 1006 gets from the input side. Shows how the macroblocks in the current picture have moved. Techniques for calculating motion data, specifically, techniques for calculating motion vectors for macroblocks that are basically regions of pixels in a digital video picture are known. Preferably, the video coding standard H.264 is used. Motion compensation calculations are used as standardized in H.264 / AVC. This takes into account the macroblock of the later picture and determines how the macroblock “moved” compared to the previous picture. This motion (in the xy direction) is indicated by a two-dimensional motion vector, is calculated for every macroblock by block 1006 and is further fed to base picture coder 1010 via motion data line 1008. Next, for the next picture, it is calculated how the macroblock has moved from the previous picture to the next picture.

  In one implementation, this new motion vector represents the motion from the second to the third picture in a sense and can also be transmitted as a two-dimensional vector. However, for efficiency reasons, it is desirable to transmit only the motion vector difference, which is the difference between the macroblock motion vector from the second to the third picture and the macroblock from the first to the second picture. It is the difference from the motion vector. Instead of this, it is also possible to refer to not only the previous picture but also a preceding picture and use the difference between them.

  The motion data calculated by block 1006 is then provided to a base motion predictor 1012, which is designed to calculate a base sequence of residual pictures to use the motion data and group of pictures. Has been. Thus, the base motion estimator performs motion compensation, which in a sense is prepared by the motion compensator and the motion estimator, respectively. This base sequence of residual pictures is then supplied to a base picture coder. The base picture coder is formed to output a base scaling layer at its output end.

  In addition, the coder of the present invention includes an extended motion compensator or extended motion estimator 1014 for detecting extended motion data. These extended motion data are supplied to an extended motion predictor 1016, which generates an extended sequence of residual pictures on the output side, and further supplies them to a downstream intermediate layer predictor 1018. Thus, the extended motion estimator performs motion compensation, which in a sense is prepared by a motion compensator and a motion estimator, respectively.

  The intermediate layer predictor is configured to calculate an extended prediction residual picture on the output side. In some implementations, the intermediate layer predictor uses the base sequence of residual pictures supplied from block 1012 via dotted bypass line 1020 in addition to the data obtained from block 1016, ie, the extended sequence of residual pictures. . Alternatively, block 1018 can also use an interpolated sequence of residual pictures supplied from the output of block 1012 and interpolated by interpolator 1022. Similarly, alternatively, the intermediate layer predictor can also provide a reconstructed base sequence of residual pictures as provided from the output 1024 of the base picture coder 1010. As can be seen from FIG. 1a, this reconstructed base sequence of residual pictures can be interpolated (1022) or not interpolated (1020). Thus, in general, the intermediate layer predictor operates using a base sequence of residual pictures, and the information at the input 1026 of the intermediate layer predictor is, for example, the residual from the output of block 1012 Derived by reconstruction or interpolation of the base sequence of pictures.

  Downstream of the intermediate layer predictor 1018 is an enhanced picture coder 1028, which is configured to encode an enhanced prediction residual picture to obtain an encoded enhanced scaling layer 1004.

  In the preferred embodiment of the present invention, the intermediate layer predictor subtracts the signal at its output 1026 from each signal that the intermediate layer predictor 1018 obtains from the enhanced motion predictor 1016 for each macroblock and for each picture. Is formed. The signal obtained as a result of this subtraction represents a macroblock in the picture of the extended prediction residual picture.

  In the preferred embodiment of the invention, the intermediate layer predictor is adaptively formed. For every macroblock, an intermediate layer prediction flag 1030 is provided, which indicates to the intermediate layer predictor that a prediction needs to be performed, or in other display states. This indicates that the prediction does not need to be performed, but the corresponding macroblock at the output end of the extended motion estimator 1016 is supplied as it is to the extended picture coder 1028 without further prediction. This adaptive implementation has the effect that it is performed only when the intermediate layer prediction is useful, compared to the case where the intermediate layer prediction is not performed but the output data of the extended motion predictor 1016 is directly encoded. The prediction residual signal has a low output picture rate.

  For spatial scalability, a decimator 1032 is provided between the enhancement scaling layer and the base scaling layer to convert a video sequence having a specific spatial resolution at the input end to a video sequence having a lower resolution at the output end. Is formed. If pure SNR scalability is used, that is, if base picture coders 1010 and 1028 for the two scaling layers operate with different quantization parameters 1034 and 1036, respectively, decimator 1032 is not provided. This is schematically illustrated by the bypass line 1038 of FIG.

  Further, in the case of spatial scalability, an interpolator 1022 needs to be provided. In the case of pure SNR scalability, no interpolator 1022 is provided. Instead, a bypass line 1020 is employed as shown in FIG.

  In one implementation, the enhanced motion compensator 1014 either calculates all the individual motion fields or uploads the motion fields calculated by the base motion compensator 1006 directly (bypass line 1040) or by the upsampler 1042. It is formed to be used after sampling. For spatial scalability, it is necessary to have an upsampler 1042 to upsample, eg scale, the motion vectors of the base motion data to a higher resolution. For example, if the extended resolution is twice the height and width of the base resolution, a macroblock (16 × 16 luminance samples) in the extended layer corresponds to a sub macroblock (8 × 8 luminance samples) in the base layer The picture area to be covered is covered.

  In this way, the base motion vector is doubled in its x and y components, i.e. scaled twice, to allow the use of the base motion vector for macroblocks in the enhanced scaling layer. This will be described in more detail with reference to FIG.

  On the other hand, in the case of SNR scalability only, the motion field is the same for all scaling layers. Thus, this only needs to be calculated once, and any higher scaling layer can directly use what the lower scaling layer has calculated.

  The signal at the output end of the base motion predictor 1012 can also be used for intermediate layer prediction. Alternatively, the reconstructed signal on line 1024 can be used. Selection of which of these two signals is used for prediction is performed by the switch 1044. The signal on line 1024 differs from the signal at the output of block 1012 in that it has already undergone a quantization process. This means that the signal on line 1024 has a quantization error when compared to the signal at the output of block 1012. The choice to use the signal on line 1024 for intermediate layer prediction is particularly advantageous when SNR scalability is used alone or in combination with spatial scalability, because in this case the base picture coder The quantization error generated by 1010 is “carried” to a higher scaling layer, ie, the output signal of block 1018 contains the quantization error generated by the first scaling layer, which in turn is input. This is because the edge 1036 is normally quantized by an extended picture coder having a finer quantization step width and a different quantization parameter 2 and written to the extended scaling layer 1004.

  Similar to the intermediate layer prediction flag 1030, the motion data flag 1048 is fed into the picture coder and the corresponding information about the motion data is included in the enhanced scaling layer 1004 and will be used later in the decoder, This will be described with reference to FIG.

  If pure spatial scalability is used, the output signal of base motion estimator 1012, ie the base sequence of the residual picture, is used instead of the signal on line 1024, ie instead of the reconstructed sequence of the base residual picture. be able to.

  Depending on the implementation, control of this switch can be performed manually or based on a predicted profit function.

  Desirably, all predictions, that is, motion prediction, extended motion data prediction, and intermediate layer residual value prediction are designed adaptively. Specifically, for example, a motion data prediction residual value does not necessarily have to exist for every macroblock or sub-macroblock in a picture of a base sequence of residual pictures. In this way, the pictures of the base sequence of residual pictures can each include unpredicted macroblocks and sub-macroblocks even though they are called “residual pictures”. This situation will occur, for example, when it is detected that a new object has occurred in the picture. Here, motion compensated prediction is not useful, because the prediction residual signal will be larger than the original signal in the picture. In the extended motion prediction at block 1016, in such a case, both the prediction operator and ultimately the update operator for this block (eg, macroblock or sub-macroblock) are deactivated.

  Here, for clarification, for example, the base sequence of the residual picture is only one residual picture of the base sequence of the residual picture having one block that actually includes the motion prediction residual signal. Even so, this is referred to as a base sequence of residual pictures. However, for a typical application, in practice every residual picture has a large number of blocks with motion prediction residual data.

  In the sense of the present invention, this also applies to an extended sequence of residual pictures. As such, the state in the enhancement layer is the same as the state in the base layer. Thus, in the sense of the present invention, an extended sequence of residual pictures is a sequence of pictures, and in extreme cases only a single block of a single “residual picture” is a motion prediction residual. In all other blocks in this picture, as well as in all other “residual pictures”, with motion-compensated prediction and motion if necessary for all other pictures / blocks Since the compensation update is deactivated, there may be no residual.

  In the present invention, this also applies to an intermediate layer predictor that calculates an extended prediction residual picture. Typically, the enhanced prediction residual picture will exist as a sequence. On the other hand, the intermediate layer predictor is preferably formed adaptively as well. For example, base layer residual data prediction from base layer to enhancement layer is only useful for a single block of a single “residual picture”, for all other blocks of this picture, and If necessary, for all other pictures in the sequence of enhanced prediction residual pictures, if it is detected that middle layer residual data prediction is deactivated, in the context of the present invention, Therefore, this sequence is also referred to as an extended prediction residual picture. In this connection, the intermediate layer predictor can predict the residual data because the motion compensation residual value is already calculated in the corresponding block of the residual picture in the base layer, and the block corresponding to this block ( For example, motion compensated prediction for the same x, y position) is also already performed in the residual picture of the extended sequence, and only if the residual value is present in the enhancement layer by motion compensated prediction in this block. The intermediate layer predictor is preferably activated only if the actual motion compensated prediction residual value is present in both considered blocks, and the residual value block in the base layer picture is replaced in the enhancement layer picture. As a predictor for a block of residual values, only the residual value of this prediction, ie, only the extended prediction residual data in this block of the considered picture, is sent to the extended picture coder.

  In the following, detailed examples of the base picture coder 1010, the extended picture coder 1028 and other picture coders will be described with reference to FIG. 1b. The picture coder receives the group of residual pictures at the input side and supplies them to the converter 1050 for each macroblock. The transformed macroblock is then scaled at block 1052 and further quantized using quantization parameters 1034, 1036,. The output end of the block 1052 outputs the quantization parameter used, that is, the quantization step width used for the macroblock and the quantization index for the spectral value of the macroblock. This information is then supplied to an entropy coder stage, not shown in FIG. 1b, which includes a Huffman coder, preferably an arithmetic coder. It operates on the well-known CABAC concept according to H.264 / AVC. The output signal of means 1052 is also provided to block 1054, where inverse scaling and requantization are performed to convert the quantization index into a numerical value along with the quantization parameter, which is then the inverse transform of block 1056. To obtain a reconstructed group of residual pictures. This has a quantization error at the input of the transform block 1050 compared to the original group of residual pictures, and the quantization error depends on the quantization parameter and the quantization step width, respectively. To perform the residual value prediction of the present invention, one signal or another signal is supplied to the interpolator 1022 or directly to the intermediate layer predictor 1018 under the control of the switch 1044.

  A simple implementation of the intermediate layer predictor flag 1030 is shown in FIG. If the intermediate layer prediction flag is set, the intermediate layer predictor 1018 is activated. However, if the flag is not set, the intermediate layer predictor is deactivated and a simultaneous cast operation is performed on this macroblock or sub-macroblocks subordinate to it. As described above, this is because the predicted coder gain actually becomes a coder loss, that is, when the corresponding macroblock is transmitted at the output end of the block 1016, the predicted residual value is used. In later entropy decoding, better coder gain is provided.

  A simple implementation of the motion data flag 1030 is shown in FIG. If the flag is set, the enhancement layer motion data is derived from the base layer upsampled motion data. For SNR scalability, the upsampler 1042 is not necessary. Here, when the flag 1048 is set, the motion data of the enhancement layer can be directly derived from the base motion data. The motion data “derivation” may be a direct citation of motion data, or the motion vector obtained from the base layer by the block 1014 may be used as an extended scaling layer calculated by the block 1014 in order to obtain a motion data prediction value. On the other hand, it may be an actual prediction that is subtracted from the corresponding motion vector. The enhancement layer motion data (if no type of prediction has been performed) or the residual value of the prediction (if the actual prediction has been performed) is sent to the enhanced picture coder 1028 via the output shown in FIG. 1a. And will eventually be included in the enhanced scaling layer bitstream 1004. However, with or without scaling, if a full takeover of motion data is performed from the base scaling layer, they need not be written into the enhanced scaling layer bitstream 1004. It is sufficient to signal that by the motion data flag 1048 in the enhanced scaling layer bitstream.

  FIG. 2 shows an apparatus for decoding an encoded video sequence that includes a base scaling layer 1002 and an enhanced scaling layer 1004. The enhanced scaling layer 1004 and the base scaling layer 1002 can be extracted from the bitstream demultiplexer, which extracts both the scaling layers to extract both the base scaling layer 1002 and the enhanced scaling layer 1004 from the common bitstream. Separate scalable bitstreams. The base scaling layer 1002 is provided to a base picture decoder 1060, which is configured to decode the base scaling layer to obtain a base sequence and base motion data for residual pictures, which are output lines 1062 Applies to The output signal on line 1062 is then provided to a base motion combiner 1064, which in the coder of block 1012 outputs the decoded picture of the first scaling layer on the output side. Cancel the base motion predictor introduced in. Further, the decoder of the present invention includes an extended picture decoder 1066 for decoding the extended scaling layer 1004 to obtain an extended prediction residual picture on the output line 1068. Further, the output line 1068 includes motion data flag 1070 or motion data information such as extended motion data or extended motion data residual values in the case where the extended motion data or the extended motion data residual value actually exists in the extended scaling layer 1004. On the other hand, the decoded base sequence on line 1062 is interpolated by the interpolator 1070 or remains unchanged (line 1072) to cancel the intermediate layer prediction performed by the intermediate layer predictor 1018 of FIG. ) Will be supplied to the intermediate layer combiner 1074. In this way, the intermediate layer combiner is able to obtain an extended prediction residual picture and an interpolated (1070) or not (1072) decoded base on line 1062 to obtain an extended sequence of residual pictures. It is configured to combine information about the sequence. This extended sequence is finally provided to an extended motion combiner 1076, which cancels the motion compensation being performed in the enhancement layer, similar to the base motion combiner 1064. Extended motion combiner 1076 couples to motion data calculation means 1078 to provide motion data for the motion combination in block 1076. This motion data may actually be all the extended motion data for the enhancement layer provided by the enhancement picture decoder from the output terminal 1068. Alternatively, the extended motion data can be used as a motion data residual value. In both cases, the corresponding data is supplied to the motion data determination means 1078 via the extended motion data line 1080. However, if the motion data flag 1070 signals that the individual enhancement motion data has not been transmitted to the enhancement layer, the required motion data is used from the base layer via the line 1082 to the scalability used. Depending on whether it is taken directly (line 1084) or after upsampling by upsampler 1086.

  Furthermore, in the case of intra-block intermediate layer prediction, there is no residual motion data value, and the corresponding connection between the extended motion combiner 1076 and the base motion combiner 1064 is provided on the decoder side, and the decoder has spatial scalability. In some cases, it has an interpolator 1090 or a bypass line if only SNR scalability is used. In the case of selective intra block prediction between two layers, only the prediction residual signal is transmitted to the enhancement layer for that intra block, and the intra block is indicated by the corresponding signaling information in the bitstream. In this case, the extended motion combiner also performs addition on this macroblock in addition to the functions described below, i.e., combines the macroblock residual value with the macroblock value from the lower scaling layer, Further, the obtained macroblock is supplied to the actual inverse motion compensation process.

  In the following, preferred embodiments of the base motion predictor 1012 or the extended motion predictor 1016 and the inverse processing elements, ie the extended motion combiner 1076 or the base motion compensator 1064, will be described with reference to FIGS. explain.

  Basically, any motion compensation prediction algorithm can be used, ie, the motion compensation algorithm shown at 92 in FIG. Thus, the conventional motion compensation algorithm also follows the system shown in FIG. 1, but the update operator U, indicated by reference numeral 45 in FIG. 4, is deactivated. Thereby, the group of pictures is converted into an original picture, a residual picture, a prediction residual signal, or a residual picture depending on the picture. However, if the update operator as shown in FIG. 4 is active and the extension is performed with a known motion compensation scheme, eg calculated as illustrated in FIGS. 5a-5d, standard motion compensation The prediction calculation is a so-called MCTF process, also called motion compensation time filtering. Here, since the original picture is combined with the prediction residual signal weighted by the update operator, the conventional motion compensation standard picture and intra picture become low-pass pictures through the update operation, respectively.

  In the preferred embodiment of the present invention, as already described with respect to FIGS. 1a and 2, such MCTF processing is performed for every scaling layer, and MCTF processing is preferably performed in FIGS. And in the manner described with reference to FIGS.

  Hereinafter, a preferred embodiment of the motion compensation prediction filter will be described with reference to FIG. 4 and the subsequent FIGS. 5a to 5d. As already explained, the motion compensated temporal filter (MCTF) is composed of a general lifting scheme with three steps: polyphase decomposition, prediction and update. The corresponding analysis / synthesis filter bank structure is shown in FIG. On the analysis side, odd samples of a given signal are filtered to a prediction residual value by linear combination of even samples using the prediction operator P and the high pass signal H. The corresponding low-pass signal l is formed by adding a linear combination of predicted residual values h to the even samples of the input signal s using an update operator. The mathematical relationship between variables h and l is shown in FIG. 4, and a basic embodiment of operators P and U is shown in FIG. 5a.

Since both the prediction and update steps can be completely reversed, the corresponding transform can be considered as a strictly sampled complete reconstruction filter bank. The synthesis filter bank includes an application of inverse sign prediction and update operators arranged in reverse order in the summation process, where even and odd polyphase components are used. For normalization of the high-pass / low-pass components, corresponding scaling factors F _l and F _h are used. These scaling factors are not necessarily used, but can be used in selecting the quantization step size during the encoding process.

f [x, k] indicates a spatial coordinate x = (x, y) ^T , and k is a time coordinate. The prediction operator P and the update operator U for temporal decomposition using the lifting expression of the hair wavelet are given by the equations shown on the left side of FIG. For the 5/3 transformation, the corresponding operator is obtained as shown on the right side of FIG. 5b. Extensions to motion compensated temporal filtering are obtained by modifying the prediction and update operators as shown in FIG. 5c. Specifically, a reference index of r> 0 is referred to, thereby enabling general picture adaptive motion compensation filtering. Through these reference indices, in the scenario shown in FIG. 4, not only can the two temporally succeeding pictures be decomposed into a high-pass picture and a low-pass picture, but for example the first picture in a motion compensated manner. The third picture can be filtered. Alternatively, by appropriately selecting the reference index, for example, the same one picture of a sequence in a sequence group can be used as a base for a motion vector. This means that a single index as a result of processing these 8 pictures with the filter scheme of FIG. 4 in a sequence of 8 pictures in which all motion vectors are associated with the 4th picture of the sequence by means of the reference index. A low-pass picture can be obtained, and all the motion vectors associated with 7 high-pass pictures (enhanced pictures) and the same one picture of the original sequence can be obtained, with one extended picture being associated with every motion vector.

Thus, if the same single picture of a sequence is used as a reference for filtering several additional pictures, this leads to temporal resolution scaling independent of a factor of 2, which is advantageous for certain applications. It is. At all times, the same picture, ie the fourth picture of a sequence of eg 8 pictures, is fed into the lower branch of the analysis filter bank of FIG. The low-pass picture is the same in all filtering, in other words, the single low-pass picture that is ultimately required for the sequence of pictures. If the update parameter is zero, the base picture is “passed” through the lower branch. On the other hand, a high-pass picture always depends on other pictures corresponding to the original sequence and a prediction operator, and a motion vector associated with the input picture is used for prediction. Thus, in this case, the finally obtained low pass picture is associated with a particular picture of the original sequence of pictures, and each high pass picture is also associated with a picture of the original sequence. Is exactly corresponding to the sequence (motion compensation) from the selected base picture of the sequence (which is fed to the lower branch of the analysis filter bank of FIG. 4). If any update parameters M ₀₁ , M ₁₁ , M ₂₁ and M ₃₁ are equal to zero, it means that the picture sent into the lower branch 73 of the fourth level is “passed” towards the bottom. In a sense, the low-pass picture TP1 is fed “repeatedly” to the filter bank, while the other pictures are controlled by reference indices and introduced one by one at the input 64 of FIG.

  As can be seen from the previous equation, the prediction and update operators for motion compensation filtering each provide different predictions for two different wavelets. When hair wavelets are used, one-way motion compensated prediction is achieved. However, when 5/3 spline wavelets are used, the two operators specify bi-directional motion compensated prediction.

  Bidirectional compensated prediction typically reduces the energy of the prediction residual value compared to unidirectional prediction, but increases the motion vector rate, so switching between unidirectional and bidirectional prediction can be performed dynamically. Desirably, this means that it is possible to switch between lifting representations of 5/3 spline wavelets and hair wavelets that depend on picture-dependent control signals. The inventive concept allows this macroblock-by-macro switching between two wavelets without using a closed feedback loop for temporal filtering, and also supports flexibility and especially data rate savings, signal adaptation Can be performed optimally in a traditional manner.

In order to ideally represent the motion field or generally the motion prediction data fields M _P and M _U , The existing syntax of B slices in H.264 / AVC can be used.

By cascading the pairwise picture decomposition stages, a binomial decomposition tree structure is obtained, whereby the group of 2 ⁿ pictures is represented by 2 ^n-1 residuals, as shown in FIG. It is decomposed into a picture and a single low-pass (or intra) picture. Specifically, FIG. 7 shows a first level high pass picture HP1 at the output end 22 of the first level filter and a first level low pass picture at the output end 24 of the first level filter. Two low-pass pictures TP2 at the output 16 of the second level filter and a high pass picture obtained from the second level are shown in FIG. 7 as a second level picture. The third level low pass picture is applied to the output 76 of the third level filter and the third level high pass picture is applied to the output 75 in the processed form. A group of 8 pictures can be originally composed of 8 pictures, so the decoder of FIG. 3 is used without a fourth level filter. However, when the group of 8 pictures is a group of 8 low-pass pictures as used at the output end 73 of the fourth level filter, the MCTF decomposition of the present invention is further used as a base motion predictor and an extended motion predictor. Can be used as a base motion combiner or an extended motion combiner, respectively.

Thus, in this decomposition, a group of 2 ⁿ pictures, a (2 ^{n + 1-2} ) motion field description, a (2 ^n-1 ) residual picture and a single low-pass (or intra) picture are transmitted.

  Both the base motion compensator and the extended motion compensator are preferably controlled by the base control parameter and the extended control parameter, respectively, and an optimal combination of the quantization parameter (1034 or 1036) and the motion information is calculated, It has been fixed to a specific rate. This is performed according to the following method in order to obtain an optimal ratio for a particular maximum bit rate. Thus, it can be seen that at low bit rates, meaning relatively coarse quantization parameters, motion vectors are more important than for higher order scaling layers where relatively fine quantization parameters are taken. Thus, for coarse quantization and thus lower bit rates, less motion data is calculated than for higher scaling layers. Therefore, the higher scaling layers have better quality and optimal state at higher bit rates than the lower bit rates where motion data is proportionally important with respect to residual data compared to higher scaling layers. Therefore, it is desirable to shift to the sub macroblock mode in order to calculate a certain amount of motion data. This will be described below.

は、ラグランジュ関数

を最小化することによって計算され、ここで、劣化数式項は次式

で与えられる。
ここで、Ｓは、参照ピクチャＡ内の動きベクトル検索領域を特定する。Ｐは、考慮されたマクロブロックパーティションまたはサブマクロブロックパーティションによってカバーされる領域である。Ｒ（ｉ，ｍ）は、動きベクトルｍの全ての成分を送信するために必要なビット数を特定し、λは固定ラグランジュ乗数である。 Pictures A and B are given, which are the original picture or the picture representing the low pass signal generated in the previous analysis stage. In addition, corresponding sequences of luma samples a [] and b [] are provided. The motion description M _i0 is estimated for each macroblock as follows.
For all possible macroblocks and sub-macroblock partitions of macroblock i in picture B, the associated motion vector

Lagrange function

, Where the degradation formula term is

Given in.
Here, S specifies a motion vector search area in the reference picture A. P is the area covered by the considered macroblock partition or sub-macroblock partition. R (i, m) specifies the number of bits required to transmit all the components of the motion vector m, and λ is a fixed Lagrange multiplier.

は、補間演算子として解釈される。 Initially, the motion search proceeds through the exact motion vector of all integer samples in a given search region S. Next, using the best integer motion vector, the eight surrounding half-sample accurate motion vectors are tested. Finally, using the best half-sample accurate motion vector, the eight surrounding quarter-sample accurate motion vectors are tested. Mathematical terms for half and quarter accurate motion vector improvements

Is interpreted as an interpolation operator.

劣化数式項は、次式

で与えられ、ここで、Ｐは、マクロブロックまたはサブマクロブロック領域を特定し、ｍ［ｐ，ｘ，ｙ］は、マクロブロックまたはサブマクロブロックのモードｐおよびルマ位置（ｘ，ｙ）を含むパーティションまたはサブマクロブロックパーティションに関連付けられる動きベクトルである。 In general, mode determination for macroblock mode and sub-macroblock follows the same approach. The mode p _i that minimizes the next Lagrangian function is selected from a predetermined set of possible macroblock or sub-macroblock modes S _mode .

Degradation formula term is

Where P identifies the macroblock or sub-macroblock region and m [p, x, y] includes the mode p and luma position (x, y) of the macroblock or sub-macroblock A motion vector associated with a partition or sub-macroblock partition.

  The rate formula term R (i, p) represents the number of bits associated with the selection of the coder mode p. In motion compensation coder mode, the mode is a bit for macroblock mode (if applicable), sub-macroblock mode and its multiple modes (if applicable), and motion vector and each of its multiple vectors. including. In intra mode, the mode includes bits for macroblock mode and an array of quantized luma and chroma transform coefficient levels.

で与えられる。 The set of possible sub-macroblock modes is

Given in.

で与えられ、ここで、ＩＮＴＲＡモードは、予測ステップで用いられるモーションフィールド記述Ｍ_i0が推定される場合にだけ用いられる。 The set of possible macroblock modes is

Where the INTRA mode is used only when the motion field description M _i0 used in the prediction step is estimated.

に従って設定される。 The Lagrangian multiplier λ is based on the base layer quantization parameter for the high-pass picture or picture QP _Hi of the decomposition stage where the motion field is estimated:

Set according to.

  In accordance with the present invention, the decomposition scheme shown in FIG. 8 is used, which is assumed to be a sensible compromise between time scalability and coder efficiency. The sequence of original pictures is handled as a sequence of input pictures A, B, A, B, A, B,. Thus, this scheme provides a stage with optimal temporal scalability (equal distance between low-pass pictures). The low-pass picture sequence is used as an input signal for all subsequent decomposition stages, but is treated as a sequence of input pictures B, A, A, B, B, A,. In the two-channel analysis scheme, the space between the low-pass pictures to be decomposed is kept small, as can be seen in FIG.

  In the following, reference is made to FIGS. 6a to 6d for the preferred implementation of both motion data intermediate layer prediction and residual data intermediate layer prediction. In order to obtain spatial and SNR scalability, respectively, basically the motion data and texture data of the lower scaling layer are used for the prediction of the higher scaling layer. Here, particularly in spatial scalability, upsampling of the motion data is required before the motion data is used as a prediction for spatial enhancement layer decoding. Base layer representation motion prediction data is transmitted using a subset of the existing B-slice syntax of AVC. Preferably, two additional macroblock modes are introduced for enhancement layer motion field encoding.

  The first macroblock mode is “base_layer_mode”, and the second mode is “qpel_refinement_mode”. To signal these two additional macroblock modes, two flags, BLFflag and QrefFlag, are added to the macroblock layer syntax prior to the syntax element mb_mode, as shown in FIG. Thus, the first flag BLFlag 1098 signals the base layer mode, and the other flag 1100 symbolizes the Qpel refinement mode. When such a flag is set, its value is 1 and the data stream is as shown in FIG. 6a. Thus, when the flag 1098 is a value of 1, the flag 1100 and the syntax element macroblock mode 1102 have no further significance. On the other hand, if the value of the flag 1098 is zero, the flag is not set, the flag 1100 is used, and if the flag 1100 is set, the element 1102 is bridged again. However, if the values of both flags 1098 and 1100 are zero, i.e., both are not set, then the macroblock mode is evaluated in syntax element 1102.

  If BLFlag = 1, the base layer mode is used for the corresponding macroblock and no further information is used. This macroblock mode indicates that motion prediction information including the macroblock partition of the corresponding macroblock of the base layer is thus directly used for the enhancement layer. It should be noted that throughout this part and throughout this specification, the term “base layer” refers to a layer one level lower with respect to the currently considered enhancement layer. If the base layer represents a layer with half spatial resolution, the motion vector field, ie the field of the motion vector containing the macroblock partition, is scaled accordingly, as shown in FIG. 6b. In this case, the current macroblock includes the same area as the 8 × 8 sub-macroblock of the base layer motion field. Thus, if the corresponding base layer macroblock is directly encoded in 16 × 16, 16 × 8 or 8 × 16 mode, or if the corresponding base layer sub-macroblock is 8 × If it is encoded in 8 mode or directly in 8x8 mode, the 16x16 mode is used for the current macroblock. On the other hand, if the base layer sub-macroblock is encoded in 8 × 4, 4 × 8 or 4 × 4 mode, the mode for the current macroblock is macroblock mode = 16 × 8, 8 × 16 or 8 × 8 (however, all sub macroblock modes = 8 × 8). If the base layer macroblock represents an INTRA macroblock, the current macroblock is set to INTRA_BASE, ie, the macroblock with prediction from the base layer. For the macroblock partition of the current macroblock, the same reference index is used as for the corresponding macroblock / sub-macroblock partition of the base layer block. The associated motion vector is doubled. This scaling factor is applied in the situation shown in FIG. 6b, ie, the base layer 1102 includes half the pixel area and the number of pixels, respectively, of the enhancement layer 104. If the ratio of base layer spatial resolution to enhancement layer spatial resolution is not ½, the corresponding scaling factor is used for the motion vector.

  However, if the flag 1098 is equal to zero and the flag 1100 is equal to 1, the macroblock mode qpel_refinement_mode is signaled. Flag 1100 is preferably present only if the base layer represents a layer having a spatial resolution that is half that of the current layer. Otherwise, the macroblock mode (qpel_refinement_mode) is not included in the set of possible macroblock modes. The macroblock in this case is the same as in the base layer mode. Macroblock partitions, reference indices and motion vectors are derived as in the base layer mode. However, for each motion vector, there is an additional quarter-sample motion vector refinement -1.0 or +1 for every motion vector component, which is additionally transmitted to the derived motion vector Added.

When the flag 1098 = 0 and the flag 1100 = 0, that is, when the flag 1100 does not exist, the macroblock mode, the corresponding reference index, and the motion vector difference are specified as usual. This means that the entire set of motion data is transmitted to the enhancement layer in the same way as was done to the base layer. However, the present invention provides the possibility to use a base layer motion vector as a predictor for the current enhancement layer motion vector (instead of a spatial motion vector predictor). In this way, the reference index list of the considered motion vector is specified by the list X (X is a value between 0 and 1). If all subsequent states are true, as shown in FIG. 6c, the following flag MvPrdFlag is transmitted for all motion vector differences.
-Base layer macroblocks containing the current macroblock / sub-macroblock partition are not encoded in INTRA macroblock mode.
The base layer macroblock / submacroblock partition covering the upper left sample of the current macroblock / submacroblock partition uses list X or bi-prediction.
The list X reference index of the base layer macroblock / submacroblock partition containing the upper left sample of the current macroblock / submacroblock partition is equal to the list X reference index of the current macroblock / submacroblock partition.

  When the flag 1106 of FIG. 6c does not exist or when this flag 1106 = 0, a spatial motion vector predictor is specified as in the AVC standard. Otherwise, if the flag 1106 is present and = 1, the corresponding base layer vector is used as the motion vector predictor. In this case, the current macroblock / sub-macroblock partition list X motion vector (X = 0 or 1) was sent to the base layer macroblock / sub-macroblock partition scaled list X motion vector. It is obtained by adding the list X motion vector difference.

  Thus, flags 1098, 1100, and 1106 together represent a manner in which motion data flag 1048 and motion data control signal 1048, as schematically illustrated in FIG. 1a, respectively, can be implemented. Of course, there are other different ways of signaling, and of course the signaling information can be reduced between the transmitter and receiver using a certain convention.

  Further details regarding the specific implementation of the extended motion compensator 1014 of FIG. 1 and the extended motion data determination means 1078 of FIG. 2 will now be described with respect to FIGS.

  Referring to FIG. 1e, it can be seen that the extended motion compensator 1014 basically has to do two things. That is, it must first calculate the extended motion data, typically the entire motion vector, and provide it to the extended motion predictor 1016, so that it is usually block-by-block in the prior art. These vectors in uncoded form can be used to obtain an extended sequence of residual pictures that is adaptively performed on The other matter is to process the extended motion data, that is, to write the motion data used for motion compensated prediction into the bitstream as compressed as possible. In order to write something to the bitstream, it is necessary to send the respective data to the extended picture coder 1028 as shown in FIG. As described above, the extended motion data processing unit 1014b has a function for reducing the redundancy included in the extended motion data determined by the extended motion data calculating unit 1014a as much as possible with respect to the base layer.

  According to the present invention, for the base motion data or the upsampled base motion data, the extended motion data calculation means 1014a can be used to calculate the actually used extended motion data, or the extended motion data can be processed. That is, it can be used only for compression of extended motion data, but these are not important for the calculation of extended motion data. In FIG. ) And 2. 2) shows an embodiment in which base motion data and upsampled base motion data are used in the calculation of the extended motion data, as shown in 3.b of FIG. ) Shows a case where the information about the base motion data is not used for the calculation of the extended motion data, but is used only for encoding and capturing the residual data.

  FIG. 1f shows the decoder side implementation of the extended motion data determining means 1078, which has a control module 1078a for performing block-by-block control, which includes signals from the bitstream and from the extended picture decoder 1066. Each includes information transmission. Further, the extended motion data determination means 1078 includes extended motion data reconstruction means 1078b, which is decoded using only decoded base motion data or decoded upsampling base motion data. The motion vector of the extended motion data field is actually determined by combining the information about the base motion data and the information from the residual data extracted from the extended scaling layer 1004 by the extended motion decoder 1066; The motion vectors can be used by an extended motion combiner 1076 that can be formed as a common combiner for inverse processing of coder side motion compensated prediction.

  In the following, reference is made to the various embodiments outlined in FIG. As already described with respect to FIG. 6a, the BLFlag 1098 signals a full takeover of the upscaled base motion data for extended motion prediction. In this case, the means 1014a takes over the base motion data entirely, and in the case of different resolutions from different layers, takes over the motion data in an upscaled form and transmits them to the means 1016, respectively. It is formed as follows. However, information about motion fields or motion vectors is not sent to the extended picture coder. Instead, only a separate flag 1098 is sent for each block, either macroblock or sub-macroblock.

  On the decoder side, means 1078a of FIG. 1f decodes flag 1098 for one block and, if it is active, decoded base motion data in the base layer, or decoded upsample Using the generated base motion data, the extended motion data is calculated and then provided to block 1076. In this case, the means 1078 does not require motion vector residual data.

の最小値を検索することによって実行される。 In the second embodiment of the invention, signaled by the flag QrefFlag 1100, the base motion vector is incorporated into the extended motion data calculation performed by the means 1014a. Part of FIG. ) And as described above, motion data calculation and motion vector m

This is done by searching for the minimum value of.

  The difference between the current picture B block and the preceding and / or following picture block shifted by a particular potential motion vector is introduced into the distortion formula term D. The quantization parameter of the extended picture coder shown at 1036 in FIG. 1a is introduced into the factor λ. The formula term R provides information about the number of bits used to encode the potential motion vector.

  Usually, a test is performed between the various potential motion vectors, the distortion formula term D is calculated for every new motion vector, the rate formula term R is calculated, preferably constant but varies. An extended quantization parameter 1036 that can also be considered. The above summation formula term is evaluated for various potential motion vectors and the motion vector that provides the minimum sum of the results is used.

  Then, according to the present invention, the base motion vector of the corresponding block from the base layer is also incorporated into this interactive search. If the vector meets the search criteria, only flag 1100 must be sent as before, but no residual value or anything else for this block. Thus, if the base motion vector meets the criteria for the block (minimization of the previous mathematical term), means 1014a uses the motion vector to send it to means 1016. However, only the flag 1100 is transmitted to the extended picture coder.

  On the decoder side, if means 1078b decodes flag 1100 to determine the motion vector for this block from the base motion data, means 1078a controls means 1078b because the extended picture decoder uses the residual data It is because it has not transmitted.

  In the modification of the second embodiment, not only the base motion vector but also a plurality of base motion vectors derived from the base motion vector and having (slightly) differences are incorporated in the search. Depending on the implementation, any component of the motion vector can be individually incremented or decremented by one, or can remain the same. This increment can represent a particular granularity of the motion vector, eg, a resolution step, a half resolution step or a quarter resolution step. When such a different base motion vector satisfies the search criterion, the difference, that is, the increment value of +1, 0, or −1 is added to the flag 1100 as “residual data” and transmitted.

  Activated by flag 1100, the decoder looks for increments in the data stream, recovers base motion vectors or upsampled base motion vectors, and obtains motion vectors for corresponding blocks in the enhancement layer. At 1078, the increment and the corresponding base motion vector are combined.

  In the third embodiment, signaled by the flag 1106, the motion vector determination can basically be performed arbitrarily. With regard to this overall flexibility, the means 1014a can determine the extended motion data, for example according to the minimized object described in connection with the second embodiment. Next, the determined motion vector is used for motion compensated prediction on the coder side without considering information from the base layer. However, in this case, the extended motion data processing 1014a is formed so as to incorporate a base motion vector into motion vector processing for redundancy reduction before actual arithmetic coding.

  In this way, H.C. According to the H.264 / AVC standard, motion vector difference transmission is performed, and the difference between adjacent blocks is determined in a picture. In this implementation, this difference can be formed between various adjacent blocks and the smallest difference can be selected. Next, according to the present invention, the base motion vector for the corresponding block in the picture is incorporated into this search for the most suitable predictor for the motion vector difference. If the base motion vector meets the criterion of providing the smallest residual value as a predictor, it is signaled by flag 1106 and only that residual value is sent to block 1028. If the base motion vector does not meet this criterion, the flag 1106 is not set and a spatial motion vector difference calculation is performed.

  However, in a simpler coder implementation, instead of interactive search, simply use each base motion vector and its upsampled version as a predictor for always adaptively determined blocks. Can do.

  According to the present invention, intermediate layer prediction of residual data is also performed. This will be described below. When motion information is changed from one layer to the next, it may be advantageous or disadvantageous to predict the residual information. In the case of MCTF decomposition, the high-pass information of the enhancement layer from the base layer Each prediction is so. If the motion vector for a block of the current layer is similar to the motion vector of the corresponding base layer, and for each macroblock, the motion vector of the corresponding base layer is similar, the encoded base layer If the residual signal (high pass signal) is used for prediction of the extended residual signal (extended high pass signal), and only the difference between the extended residual signal and the base layer reconstruction (line 1024 in FIG. 1a) is encoded, The coder efficiency is likely to increase. However, if the motion vectors are not similar, the prediction of the residual signal is very unlikely to improve coder efficiency. As a result, an adaptive approach is used to predict the residual signal and the high-pass signal, respectively. This adaptive approach, i.e., an adaptive approach based on whether the intermediate layer predictor is active, can be performed by actual calculation of the benefits based on the differential signal, or the motion vector of the base scaling layer for the macroblock is extended It can be performed based on an estimate of how different it is for the corresponding macroblock in the scaling layer. If this difference is less than a certain threshold, the intermediate layer predictor is activated via control line 130. However, if this difference is greater than a certain threshold, the intermediate layer predictor for this macroblock is deactivated.

  A flag ResPrdFlag 1108 is transmitted. When this flag 1108 = 1, the reconstructed residual signal of the base layer is used for prediction of the residual signal of the current macroblock of the enhancement layer and the current residual signal of the enhancement layer and its base layer re- Only approximate values of the difference between the configurations are encoded. Otherwise, flag 1108 does not exist or is equal to zero. Here, the residual signal of the current macroblock in the enhancement layer is encoded without prediction from the base layer.

  If the base layer represents a layer having a spatial resolution half that of the enhancement layer, the residual signal is upsampled using an interpolation filter, and then the base layer upsampled residual signal is used as the prediction signal. This filter is an interpolation filter with 6 taps, with values from the surroundings being as good as possible in order to interpolate higher spatial resolution values of the enhancement layer that were not present in the lower resolution base layer Used to obtain interpolation data.

  However, if the value of the edge of the transform block is interpolated and the interpolation filter can only use the value of another transform block for interpolation, stop this, but synthesize the value of the interpolation filter outside the considered block Thus, it is desirable to perform interpolation with as few artifacts as possible.

  Based on so-called core experiments, it has been found that middle layer prediction of motion and residual values significantly improves coder efficiency in the AVC-based MCTF approach. A PSNR gain higher than 1 dB was obtained for a particular test point. In particular, at very low bit rates (with the exception of the base layer) for all spatial resolutions, the improvement in reconstruction quality was clearly confirmed.

  In some situations, the method of the present invention may be implemented in hardware or software. This implementation can be carried out on a digital storage medium, in particular a disc or CD, having electronically readable control signals that cooperate with a programmable computer system so that the method is carried out. . As such, the present invention generally also resides in a computer program product having program code for performing the method of the present invention stored on a machine-readable carrier when the computer program product is executed on a computer. To do. In other words, the present invention can be realized as a computer program having a program code for executing this method when the computer program is executed on a computer.

  Furthermore, the invention relates to a computer readable medium on which a scalable data stream having a first scaling layer and a second scaling layer with associated control characters is stored for various decoder-side means. Thus, the computer readable medium may be a data carrier or the Internet where a data stream is transmitted from a provider to a recipient.

FIG. 1a is a preferred embodiment of the coder of the present invention. FIG. 1b is a detailed representation of the base picture coder of FIG. 1a. FIG. 1c illustrates the function of the intermediate layer prediction flag. FIG. 1d illustrates the motion data flag. FIG. 1e is a preferred implementation of the extended motion compensator 1014 of FIG. 1a. FIG. 1f is a preferred implementation of the extended motion data determination means 1078 of FIG. FIG. 1g is a schematic representation of three preferred embodiments for calculating extended motion data and, if necessary, processing extended motion data for signaling and residual data transmission. FIG. 2 is a preferred embodiment of the decoder of the present invention. FIG. 3 is a block diagram of a decoder having four levels. FIG. 4 is a block diagram for explaining lifting decomposition of a temporal subband filter bank. FIG. 5a is a functional representation of the lifting scheme shown in FIG. FIG. 5b is a representation of two preferred lifting specifications with unidirectional prediction (hair wavelet) and bidirectional prediction (5/3 transform). FIG. 5c is a preferred embodiment of a prediction and update operator with motion compensation and reference index for the arbitrary selection of two pictures processed by the lifting scheme. FIG. 5d is a representation of an intra mode in which original picture information can be inserted into a high-pass picture for each macroblock. FIG. 6a is a schematic representation for signaling the macroblock mode. FIG. 6b is a schematic representation for upsampling motion data in spatial scalability according to a preferred embodiment of the present invention. FIG. 6c is a schematic representation of the data stream syntax for motion vector differences. FIG. 6d is a schematic representation of residual value syntax extension according to a preferred embodiment of the present invention. FIG. 7 is a schematic diagram for explaining the time shift of a group of 8 pictures, for example. FIG. 8 is a preferred temporal arrangement of low-pass pictures for a group of 16 pictures. FIG. 9 shows H.264 for a macroblock. 2 is a schematic block diagram for explaining a basic coder structure for a coder according to the H.264 / AVC standard. FIG. FIG. 10 is a context array composed of two pixel elements A and B adjacent to the left and top of the current syntax element C, respectively. FIG. 11 is a representation of a partition into a slice of a picture.

Claims (28)

An apparatus for generating an encoded video sequence having a base scaling layer (1002) and an enhanced scaling layer (1004), the apparatus comprising:
A base motion compensator (1006) for calculating base motion data indicating how a macroblock in the current picture has moved with respect to another picture in the group of pictures;
A base motion estimator (1012) for calculating a base sequence of residual pictures using the base motion data;
A base picture coder (1010) configured to calculate the encoded base scaling layer (1002) from the base sequence of residual pictures;
An extended motion compensator (1014) for determining extended motion data;
An extended motion estimator (1016) for calculating an extended sequence of residual pictures;
An intermediate layer predictor (1018) for calculating an extended prediction residual picture using the extended sequence of residual pictures and further using information about the base sequence of residual pictures (1026);
An apparatus comprising: an extended picture coder (1028) for encoding the extended prediction residual picture to obtain the encoded extended scaling layer (1004). The base picture coder (1010) is configured to perform quantization using a base quantization parameter (1034);
The extended picture coder (1028) is configured to perform quantization using an extended quantization parameter (1036), and the extended quantization parameter (1036) is finer than the base quantization parameter (1034). Can perform efficient quantization,
The base picture coder (1010) is configured to reconstruct the base sequence of residual pictures quantized using a first quantization parameter to obtain a reconstructed base sequence; A layer predictor (1026) uses the extended sequence of residual pictures, and further uses the reconstructed base sequence of residual pictures as information about the base sequence of residual pictures, The apparatus of claim 1, configured to calculate a picture. A decimator (1032) for thinning out the resolution of the group of pictures, the decimator (1032) having a base resolution lower than the extended resolution of the group of pictures provided to the extended motion compensator (1014) A decimator configured to provide the base compensator (1006) with a group of pictures having
An interpolator (1022) for spatially interpolating said base sequence of residual pictures or a reconstructed base sequence of residual pictures to obtain an interpolated base sequence of residual pictures, comprising: 3. An interpolator capable of supplying an interpolated base sequence of pictures as information (1026) about the base sequence of residual pictures to the intermediate layer predictor (1018). The device described in 1.   The apparatus of claim 3, further comprising a motion data upsampler (1042) for adapting the base motion data to the extended resolution.   The apparatus of any preceding claim, wherein the base motion compensator (1006) is configured to calculate a two-dimensional motion vector for a macroblock.   The base motion estimator (1012) is configured to subtract a macroblock predicted using the base motion data from a current macroblock to obtain a macroblock of the residual picture of the base sequence of residual pictures. A device according to any of the preceding claims.   The intermediate layer predictor (1018) is adaptive to the block whether the coder gain can be increased by applying a prediction compared to the use of the block from the extended sequence of residual pictures. An apparatus according to any preceding claim, wherein the apparatus is configured to determine   The intermediate layer predictor (1018) is configured to obtain a macroblock of an extended prediction residual picture, the base sequence of the residual picture, the reconstructed base sequence of the residual picture, and the interpolated of the residual picture. A macroblock predicted using a macroblock from the reconstructed and interpolated base sequence of the base sequence or the residual picture is subtracted from the current macroblock of the extended sequence of the residual picture. An apparatus according to any preceding claim.   The base motion compensator (1006) is configured to calculate the base motion data based on the base quantization parameter (1034), a distortion formula term for transmitting the base motion data, and a data rate. The apparatus of claim 2.   The extended motion compensator (1006) is configured to calculate the extended motion data based on the extended quantization parameter (1036), a distortion formula term for transmitting the extended motion data, and a data rate. 10. An apparatus according to claim 2 or claim 9, wherein   The apparatus of any preceding claim, wherein the enhanced motion compensator (1006) is configured to use scaled motion data based on the base motion data or mode control signal.   The base scaling layer (1002) has a lower resolution than the enhanced scaling layer (1004), and the enhanced motion compensator (1014) is configured to determine residual motion data based on a control signal (1048). The apparatus of any preceding claim, further wherein the enhanced picture coder (1028) is configured to encode the residual motion data into the enhanced scaling layer (1004).   The apparatus of any preceding claim, wherein the base motion estimator (1012) and the enhanced motion estimator (1016) are configured to perform motion compensated prediction operations.   The base motion estimator (1012) and the extended motion estimator obtain a motion compensated temporal decomposition into at least one low pass and several high pass pictures, apart from the motion compensated predictive operation. The apparatus according to claim 1, wherein the apparatus is configured to perform the following. A method for generating an encoded video sequence having a base scaling layer (1002) and an enhanced scaling layer (1004), the method comprising:
Calculating (1006) base motion data indicating how a macroblock in the current picture has moved with respect to another picture in the group of pictures;
Calculating a base sequence of residual pictures using the base motion data (1012);
Encoding (1010) information about the base sequence of residual pictures to calculate the encoded base scaling layer (1002) from the base sequence of residual pictures;
Determining extended motion data (1014);
Calculating an extended sequence of residual pictures (1016);
Calculating (1018) an extended prediction residual picture using the extended sequence of residual pictures and further using information about the base sequence of residual pictures (1026);
Encoding (1028) the enhanced prediction residual picture to obtain the encoded enhanced scaling layer (1004). An apparatus for decoding an encoded video sequence having a base scaling layer (1002) and an enhanced scaling layer (1004), the apparatus comprising:
A base picture decoder (1060) for decoding the base scaling layer to obtain a decoded base sequence and base motion data of a residual picture;
A base motion combiner (1064) for obtaining the base scaling layer sequence of pictures using the base motion data and the decoded sequence of residual pictures;
An enhanced picture decoder (1066) for decoding the enhanced scaling layer to obtain an enhanced prediction residual picture;
An intermediate layer combiner for combining the decoded base sequence of residual pictures or the interpolated base sequence of residual pictures and the extended prediction residual picture to obtain an extended sequence of residual pictures 1074)
An extended motion combiner (1076) configured to obtain a sequence of pictures of the extended scaling layer using the extended sequence and extended motion data of residual pictures. The base picture decoder (1060) is configured to perform inverse quantization using a base quantization parameter that is larger than an extended quantization parameter, and the intermediate layer combiner (1074) is configured to decode the residual picture. 17. The apparatus of claim 16, wherein the apparatus is configured to use an interpolated version of the base sequence or residual picture. The pictures of the base scaling layer (1002) may have a lower resolution than the pictures of the enhanced scaling layer (1004), and a residual for interpolating the decoded base sequence of residual pictures to a higher resolution The apparatus according to claim 16 or 17, further comprising a value interpolator (1070). The base scaling layer picture has a lower resolution than the enhanced scaling layer picture, and further converts the base motion data to the enhanced resolution to obtain the enhanced motion data or a prediction of the enhanced motion data. 19. Apparatus according to any of claims 16 to 18, further comprising a motion data upsampler (1086). The encoded video sequence has residual motion data, and a motion data determiner (1078) for combining the residual motion data and the prediction of the extended motion data to determine the extended motion data 20. The device according to claim 19, comprising:   21. The base motion combiner (1064) and the extended motion combiner are configured to perform only a reverse prediction operation or reverse prediction and reverse update operations. The apparatus in any one of.   21. Apparatus according to any of claims 18 to 20, wherein the interpolator (1070) is configured to use interpolation filter operations with a plurality of pixels in the environment of interpolated pixels.   23. The interpolation filter according to claim 22, wherein the interpolation filter is configured to use a synthesized value instead of a pixel of another transform block when a pixel of another transform block is present near the pixel to be interpolated. The device described.   The encoded video sequence has extended motion data for a macroblock, and a motion data control signal (1070) is provided for the extended motion data for which a prediction signal is derived from a scaled motion vector of the base motion data. Associated with the extended motion data indicating whether it is underlying or whether a prediction signal is underlying the extended motion data that is not derived from the base motion data; The control signal (1070) is read and further configured to calculate the extended motion data based on the control signal using the scaled version of the base motion data. The device according to any one of the above. The encoded video sequence for a macroblock has an intermediate layer prediction control signal (1030) in the enhanced scaling layer that indicates whether the macroblock was generated with or without intermediate layer prediction; The apparatus is further configured to activate the intermediate layer combiner (1074) only when the intermediate layer prediction control signal (1030) indicates intermediate layer prediction for a considered macroblock. 25. The apparatus according to any one of items 16 to 24. A method for decoding an encoded video sequence having a base scaling layer (1002) and an enhanced scaling layer (1004), the method comprising:
Decoding (1060) the base scaling layer to obtain a decoded base sequence and base motion data of a residual picture;
Performing base motion combining (1064) to obtain a sequence of pictures of the base scaling layer using the base motion data and the decoded sequence of residual pictures;
Decoding (1066) the enhanced scaling layer to obtain an enhanced prediction residual picture;
Combining the decoded base sequence of residual pictures or the interpolated base sequence of residual pictures with the extended prediction residual picture to obtain an extended sequence of residual pictures; (1074);
Performing extended motion combining (1076) to obtain a sequence of pictures of the enhanced scaling layer using the extended sequence and extended motion data of residual pictures.   27. A computer program for performing the method of claim 15 or claim 26 when the method is executed on a computer.   17. A computer readable medium having an encoded video sequence having a base scaling layer (1002) and an enhanced scaling layer (1004), wherein the encoded video sequence is decoded according to claim 16. A medium configured to generate a decoded first scaling layer and a decoded second scaling layer when decoded by an apparatus for.

Images