KR100700515B1 - Down converting mpeg encoded high definition sequences to lower resolution with reduced memory in decoder loop - Google Patents

Abstract

MPEG encoding interlaced high definition video streams are decoded and converted to low resolution by an MPEG decoder with a reduced amount of reference picture memory. The receive frame and field DCT coding block is first down-converted into a reconstructed pixel field block through conversion of the receive frame DCT coding block into a transform field DCT coding block. Vertical IDCT and horizontal IDCT are then performed in the receive field DCT coding block and the transform field DCT coding block to generate appropriate redundancy and pixel field blocks. The vertical IDCT is an N point horizontal IDCT, and the horizontal IDCT is an M point horizontal IDCT, where N corresponds to the original block size and N> M. Redundant and pixel field blocks are spatially filtered and downsampled vertically. IDCT, filtering, and downsampling operations are effectively combined by a single linear operator. Different horizontal and vertical minimum drift prediction filters are used to form the field prediction to be added to the down sampling field surplus block.

MPEG, low resolution, frame, pixel, prediction filter, redundant, IDCT, prediction, down sampling

Description

DOWN CONVERTING MPEG ENCODED HIGH DEFINITION SEQUENCES TO LOWER RESOLUTION WITH REDUCED MEMORY IN DECODER LOOP}

1 is a schematic block diagram of a known MPEG-2 encoder.

2 is a schematic block diagram of a known MPEG-2 decoder.

3 is a block diagram of a known downconversion decoder for HDTV applications.

4 shows an exemplary pixel data set useful for describing the background of the present invention.

5 is a block diagram of a down conversion decoder according to the present invention;

6 is an embodiment of the IDCT module of FIG.

7-9 illustrate exemplary pixel data sets useful for describing the present invention.

FIG. 10 is a detailed view of a block diagram of the motion compensator of FIG. 5. FIG.

11-12 show additional pixel data sets useful for describing the present invention.

This application claims priority as US Provisional Application No. 60 / 112,795, filed December 18, 1988.

This application claims the rights to a subject matter similar to that of US patent application Ser. No. 09 / 106,367, filed June 29, 1998.

 The present invention relates to a down conversion decoder for down converting and decoding high resolution encoded video with a low resolution receiver.

The international standard ISO / IEC 13818-2, entitled "Video and Related Audio Information: Comprehensive Encoding of Video" and Guidelines for Using the ATSC Digital Television Standard, describes a system known as MPEG-2 for encoding and decoding digital video data. The standard allows video to be encoded at a wide range of resolutions, including high resolution, commonly known as HDTV, according to which digital video data is encoded as a series of codewords in a complex way, for example, This results in a shorter average length of codewords than can be via an 8-bit valued encoding, which is also known as data compression.

In the above system, encoded pictures are composed of pixels. Each 8x8 pixel array is called a block, and a 2x2 array of these 8x8 blocks is called a macroblock. Known prediction techniques (motion estimation in encoder, motion compensation in decoder), two-dimensional discrete cosine transform (DCT) performed on 8x8 pixel blocks, quantization of DCT coefficients and Hoffman and execution Compression is done using / level, but prediction is not used for all pictures, so even though a P picture is encoded with a prediction from a previous picture, and a B picture does not have a prediction from a previous or subsequent picture. Although encoded using the I picture, the I picture is encoded without prediction.

An MPEG-2 encoder is schematically shown in FIG. Data representing a macroblock of pixels is input to the subtractor 12 and the motion calculator 14. In the case of P picture and B picture, motion estimator 14 compares each new macro block to be encoded with the macro block of the reference picture previously stored in the reference picture memory 16. Motion estimator 14 finds the macro block that most closely matches the new macro block in its stored reference picture.

Motion estimator 14 reads this matching macroblock (also known as predictive macroblock) from reference picture memory 16 and sends it to subtractor 12, which subtracts 12 pixel by pixel. The matching macroblock is subtracted from the new macroblock written in the MPEG-2 encoder 10 in the " The output of subtractor 12 is an error, i.e., residual (or residual), representing the difference between the predicted macroblock and the new macroblock to be encoded. This surplus is often very small. This surplus is transformed from the spatial domain by the two-dimensional DCT 18. The DCT coefficients coming from the two-dimensional DCT 18 are then quantized by the quantization block 20 in the process of reducing the number of bits needed to represent each coefficient. Usually, many coefficients are effectively quantized to zero. The quantized DCT coefficients are coded in Hoffman and Run / Level by coder 22, which also reduces the average number of bits per coefficient.

Motion estimator 14 also calculates a motion vector mv representing the horizontal and vertical displacement of the predicted macroblock in the reference picture from the position of the new macroblock in the current picture being encoded. It should be noted that motion vectors may have a half pixel resolution obtained by linear interpolation between adjacent pixels. The data encoded by the coder 22 is combined with the motion vector data from the motion estimator 14 and other information (such as an indication of whether the picture is an I, P or B picture), the combined data being Is sent to a receiver that includes an MPEG-2 decoder 30 (shown in FIG. 2 and described below).

With respect to the P picture, the quantized DCT coefficients from the quantization block 20 are also supplied to an inner decoder loop representing the operating portion of the MPEG-2 decoder 30. Within this inner loop, the excess from quantization block 20 is inverse quantized by inverse quantization block 24 and inverse DCT transformed by inverse discrete cosine transform (IDCT) block 26. The predicted macro block, which is read out from the reference picture memory 16 and supplied to the subtractor 12, is also added back to the output of the IDCT block 26 by the adder 28 in a pixel-by-pixel manner, with the result that It is stored back into the reference picture memory 16 to serve as a macro block of the reference picture for prediction. The block of this inner loop is such that the data in the reference picture memory 16 of the MPEG-2 encoder 10 matches the data in the reference picture memory of the MPEG-2 decoder 30. The B picture is not stored as a reference picture.

In the case of an I picture, no motion calculation occurs, and the negative (-) input to the subtractor 12 is forced to zero. In this case, the quantized DCT coefficients provided by the two-dimensional DCT 18 do not represent redundant values, as in the case of P and B pictures, but represent converted pixel values.

The MPEG-2 decoder 30 shown in FIG. 2 schematically shows an MPEG-2 decoder. The decoding processor executed by MPEG-2 is decoded into Hoffman and run / level by Hoffman and Run / Level Decoder 32 as opposed to the encoding process executed by MPEG-2 Encoder 10.

Motion vectors and other information are separated from the data streams flowing through Hoffman and the run / level detector 32. The motion vector is fed to a motion compensator 34. The quantized DCT at the output of Hoffman and the run / level decoder 32 is fed to the inverse quantization block 36 and then to the IDCT block 38, which in turn sends the inverse quantized DCT coefficients to the spatial domain. Send back to.

For P and B pictures, each motion vector is translated to a memory address by motion compensator 34 to read a particular macroblock (predictive macroblock) from reference picture memory 42 containing a pre-stored reference picture. do. The adder 44 adds this prediction macroblock to the surplus value provided by the IDCT block 38 to form reconstructed pixel data.

In the case of an I picture, there is no prediction, and the prediction provided to the adder 44 becomes zero (0). In the case of I and P pictures, the output of the adder 44 is then fed into the reference picture memory 42 and stored as a reference picture for future prediction.

The MPEG encoder 10 may encode a sequence of progressive or interlaced pictures. In a sequence of interlaced pictures, the picture is encoded as a field picture or frame picture. In the case of a field picture, one picture contains the odd line of the raster and the next picture contains the even line of the raster. All encoder and decoder processing is done in the field. Thus, DCT transformations are performed in 8x8 blocks containing all odd or all even lines. These blocks are referred to as field DCT coding blocks.

On the other hand, in the case of a frame picture, each picture contains lines of odd and even numbers of the raster. The macro block of the frame image is encoded as a frame in terms of the encoding macro block including odd and even lines. However, the DCT performed on four blocks in each macro block of the frame picture may be done in two different ways.

Each of the four DCT transform blocks in the macro block may include radix and even lines (frame DCT coding block), or alternatively two of the four DCT blocks only the radix line of the macro block and the other two blocks. May include only even lines (field DCT coding blocks) of the macro block. ISO / IEL 13818-2, section 6. 1.3 See Figures 6-13 and 6-14. The coding decision of how to encode the picture is made adaptively by the MPEG-2 encoder 10 based on how effective data compression is achieved.

The excess macro block in the field picture is field DCT coded and predicted from the reference field. The excess macro block in the frame picture is frame DCT coded and predicted from the reference frame. A macroblock in a field DCT coded frame picture has two blocks predicted from one reference field and two blocks predicted from the or other reference field.

For sequences of progressive pictures, all pictures are frame pictures with frame DCT coding and frame prediction.

As noted above, MPEG-2 includes encoding and decoding of video in high resolution (HDTV). In order to enable people to use their existing NTSC televisions to watch programs transmitted on HDTV, it is necessary to provide a decoder that decodes high resolution MPEG-2 encoded data into reduced resolution video data for display on existing NTSC televisions. Desirable (reducing the resolution of television signals is often referred to as down decoding). Thus, such a down conversion decoder allows viewers to watch HDTV signals without having to purchase an expensive HDTV display device.

Methods are known for making downconversion decoders that require less circuitry and are therefore less expensive than decoders that output full HDTV resolution. One of these methods is disclosed in US Pat. No. 5,262,854. The down conversion technique disclosed in this patent is described in relation to the down converter 50 shown in FIG. The down converter 50 includes the Hoffman and Run / Level Decoder 52 and the Inverse Quantization Block 54, which are related to the Hoffman and Run / Level Decoder 32 and the Inverse Quantization Block 36 of FIG. 2. It operates as described above. However, down converter 50 uses down sampler 56 instead of the 8x8 IDCT block 38 shown in FIG. 2, which down sampler 56 uses 48 high-ranking DCT coefficients of the 8x8 block. Discard and perform the remaining 4x4 IDCT of DCT coefficients. This process is commonly referred to as DCT domain down sampling. The result of this down sampling is a redundant sample of filtered and down sampled 4x4 blocks (for P or B pictures) or a pixel for an I picture.

For the surplus samples, the prediction is added by the adder 58 to the surplus samples from the down sampler 56 so as to produce 4 × 4 blocks of pixels with reduced decoded resolution. This block is stored in the reference picture memory 60 for subsequent prediction. Thus, the prediction is made from a reduced resolution reference, where the prediction made in the decoder loop in the encoder is made in the full resolution reference picture. This difference means that the prediction derived from the reduced resolution criterion is somewhat different from the corresponding prediction made by the encoder, resulting in an error in the surplus + prediction sum provided by the adder 50 (this error is then predicted afterwards). Prediction errors increase when predictions are made under predictions until the reference is refreshed by the next I picture.

Motion compensator 62 attempts to reduce this prediction error by using a full resolution motion vector even if the reference picture is at a lower resolution. First, the portion of the reference picture including the prediction block is read out from the reference picture memory 60. This part is selected based on all the bits of the motion vector except the least significant bit. This prediction block is interpolated back to full resolution by the 2x2 prediction upsampling filter 64. Using a full resolution motion vector (which may include 1/2 pixel resolution), the predicted full resolution macroblock is extracted from the upsampling portion based on every bit of the motion vector. The down sampler 66 then performs 2 × 2 down sampling on the extracted full resolution macro block to match the resolution of the 4 × 4 IDCT output of the down sampler 56. In this way, the prediction from the reference picture memory 60 can be upsampled to match the full resolution surplus pixel structure to make use of the full resolution motion vector. The full resolution reference image is then down sampled prior to addition by the adder 58 to match the down sampled excess resolution from the down sampler 56.

Various good predictive upsampling / downsampling methods are known to minimize prediction errors caused by upsampling a downsampled reference picture with 4x4 IDCT.

These methods typically contain 5 to 8 taps and a tap value that changes depending on the motion vector value for the predictive macro block relative to the closest macro block boundaries in the reference picture and the position of the current pixel being interpolated within the predictive block. The two-dimensional filter which has is used. Such a filter upsamples the reduced resolution reference to full resolution and then downsamples in a single operation, as well as including half-pixel interpolation (when required due to an odd motion vector). (The article "Minimal Error Drift in Frequency Scalability for Motion Compensated DCT coding" in Mokry and Anastassiou, published in IEE Transactions on Circuits and Systems for Video Technology, August 1994, and IEEE Workshop on Visual Signal Processing in Melbourne, Australia, September 1993. See, "Drift Minimization in Frequency Scaleable Coders Using Block Based Filtering" published by Johnson and Princen in and Communication.

A more general derivation of the minimum drift prediction filter using the Moore-Penrose inverse of blocks based on down-sampling filters is described in Vetro, Sun, Da Graca, and Poon's article in IEEE Transactions on consumer Electronics in August 1998. -Layer Scalable DTV Decoding ".

The following example shows the predictive upsampling / downsampling filter described in the papers of Mokry and Johnson. This example is a one-dimensional example, but is easily extended to two dimensions. Assume that pixel y ₁ and pixel y ₂ as shown in FIG. 4 represent two adjacent blocks in a downsampled reference picture and the desired prediction block is straddled between the two blocks. Pixel y1 is upsampled to pixel p1 using a four tap filter with a different set of tap values for eight calculated pixels p1. Likewise, pixel y1 is upsampled to pixel p1 using the same four-tap filter (if the motion vector requires 1/2 pixel interpolation, this interpolation is performed between pixel values based on pixels p1 and p2). Is performed using linear interpolation). From these 16 pixels p1 and pixel p2, an up-sampling prediction consisting of eight pixels q can be read using the complete motion vector. Pixel q is then filtered and down sampled to pixel q 'by an 8 tap filter having a different set of tap values for each of the four pixels q'. Johnson's paper is the optimal value for these filters when the reference picture is downsampled by 4-point IDCT. The Johnson and Mocry papers also show that upsampling, linear interpolation, and downsampling filters have a single 8 with tap values that depend on the motion vector value associated with the closest macroblock boundary of the reference picture and depend on the specific pixel q ＇computed. It shows that it can be combined with a tap filter. Thus, this single 8-tap filter allows four pixels q 'to be calculated directly from eight pixels y1 and y2.

For down sampling methods other than 4-point IDCT, Vetro's paper describes how to determine the optimal tap values for an up sampling filter. This upsampling is also combined with linear interpolation and downsampling operations to form a single predictive filter.

Down decoder 50, which is generally sufficient for progressive pictures with frame DCT coding blocks, does not solve the problems caused when attempting to down convert a sequence of interlaced pictures into a mixed frame and field DCT coding block. These problems arise with respect to vertical down sampling and vertical predictive filtering.

Assume that horizontal down sampling is performed in the DCT domain using 4-point horizontal IDCT. Vertical down sampling may also use 4 point IDCT or some other method. In the case of a field picture, a vertical down sampling operation is performed in the incoming field coding block. In the case of a frame picture, a vertical down sampling operation is performed on the mixed field and frame coding block. Thus, the reference picture can be down sampled on a field-based, frame-based, and mixed basis thereof. As mentioned above, low drift prediction filtering is derived from the down sampling filter. If different reference pictures are downsampled differently, they need different matching prediction filters.

There is a larger problem in the case of a reference picture comprising a mixed field and a frame down sampling block. Certain required predictions may overlap block types. This problem can be solved by converting all incoming blocks into frames or fields prior to downsampling. As a result of this transformation, a consistent vertical structure for the reference pictures is formed so that the same predictive filter is always available.

It was proposed to convert all incoming pictures to frames prior to vertical down sampling (see the article "Frequency Domain Down Conversion of HDTV Using Adaptive Motion Compensation" published by Vetro, Sun, Bao and Poon in ICIP '97). As a result of the conversion to the frame before vertical down sampling, a better vertical resolution than in the case of field based down sampling will be obtained. However, frame based downsampling requires additional memory at the decoder, since the first field must be stored upon receipt to allow the second field to arrive to form a frame block. In addition, severe blocking defects may occur in motion sequences (see the article "Frequency Domain Down Conversion of HDTV Using an Optical Motion Compensation Scheme" published by Vetro and Sun in the August 1998 Journal of Imaging Science and Technology). .                         

One solution to these problems and the proposed alternative in the latter paper is to convert all incoming pictures prior to vertical down sampling. Therefore, the present invention described herein always utilizes field based processing such that incoming blocks that are frame coded are first converted into fields prior to vertical down sampling.

In addition, four-point ICDT downsampling is known to cause visible defects. In the case of progressive images, the degree of visibility thereof can generally be tolerated in horizontal and vertical processing. However, for interlaced pictures using field based vertical down sampling, these defects are more visible. Thus, the present invention introduces a different technique than 4-point IDCT for its field based vertical down sampling (still uses 4 point IDCT for horizontal down sampling).

According to one aspect of the present invention, each of the received frame and field DCT coding block includes N × N values. The method comprises the steps of: a) converting the received frame DCT coding blocks into transformed field DCCT coding blocks; b) performing horizontal M point IDCT, vertical N point IDCT, vertical spatial filtering and down sampling on the received field DCT coding blocks and the transformed field DCT coding blocks to produce appropriate redundancy and pixel field blocks; And wherein at least the vertical space filtering and down sampling comprises at least N points and N> M; And c) adding prediction reference pixels as appropriate to the redundant field blocks to form reconstructed pixel field blocks.

According to another aspect of the present invention, there is provided a method of decoding a received first DCT coefficient block into a reconstructed field pixel block, which method comprises: a) a vertical operator and a horizontal operator to generate intermediate redundant or pixel values; Applying to a first DCT coefficient block, wherein the vertical operator is simultaneously applied to the coefficients of the first DCT coefficient block and second and third DCT coefficient blocks, wherein the second DCT coefficient block is applied to the first DCT Is above a coefficient block, wherein the third DCT coefficient block is below the first DCT coefficient block, and the horizontal operator is applied to the second and third DCT coefficient blocks asynchronously with the first DCT coefficient block; And b) appropriately adding prediction reference pixels to the intermediate surplus value to form reconstructed pixels.

According to another aspect of the invention, a method is provided for decoding a received first DCT coefficient macro block having a frame DCT coding block into reconstructed field pixel blocks, which method comprises: a) generating intermediate surplus or pixel values; To apply a vertical operator and a horizontal operator to a first DCT coefficient macro block, wherein the vertical operator is simultaneously applied to coefficients of the first DCT coefficient macro block and the second and third DCT coefficient macro blocks, A second DCT coefficient macro block is above the first DCT coefficient macro block, the third DCT coefficient macro block is below the first DCT coefficient macro block, and the horizontal operator is asynchronous with the first DCT coefficient macro block. Is applied to the second and third DCT coefficient macro blocks; And b) appropriately adding prediction reference pixels to the intermediate surplus value to form reconstructed pixels.                         

According to another aspect of the invention, there is provided an apparatus arranged to reconstruct pixels from a target DCT coefficient macro block, the apparatus comprising a vertical operator, a horizontal operator and an adder. The vertical operator is of sufficient size to be applied simultaneously to the target DCT coefficient macro block and the adjacent DCT coefficient macro block. The horizontal operator is arranged to horizontally filter the target DCT coefficient macro block to produce intermediate pixel values in conjunction with the vertical filter. To form the reconstructed pixels, an adder is arranged to add the prediction reference pixel values to the intermediate pixel values.

These and other features and advantages of the present invention will become more apparent from the following detailed description of the invention with reference to the drawings.

I. Down Conversion

A down conversion decoder 100 according to an embodiment of the present invention is shown in FIG. For vertical processing, the down transform decoder 100 performs DCT domain frame-field transform, field-based vertical 8 point DCT domain frame-field transform, field-based vertical 8 point IDCT, vertical space interblock filtering and down sampling and complementary vertical minimum Perform drift prediction filtering. In the case of horizontal processing, the down transform decoder 100 performs horizontal 4-point IDCT and complementary horizontal minimum drift prediction filtering for filtering and down sampling.

The filtering and down sampling module 102 of the down transform decoding 100 receives its input from the inverse quantizer of FIG. 3. The vertical processing performed by the filtering and down sampling module 102 is described in detail. Vertical filtering is an 8-tap low-pass symmetric FIR filter, a cubic filter derived when B = C = 1/3 in Mitchell and Netravali's paper "Reconstruction Filters in Computer Graphics," published in Siggraph, 1988. It is composed. Therefore, the tap values of this filter are given by

This 8 point vertical filter is effectively used to perform successive interblock field based vertical filtering operations, filter [f] _{8 operating} on the received redundant and I field pixel blocks.

The vertical filtering and down sampling performed by the filtering and down sampling module 102 is given by the following equation and can be seen as a vertical spatial domain operation in the entire field done in one column (at 540 pixels) in time.

Where [x] is a column of pixels from the field of pixels to be filtered and downsampled, [x '] is a column of resulting vertically filtered and downsampled pixels, and [d] is a downsampling given by the following cyclic matrix Is an operator.

(According to the nomenclature used here, uppercase letters are used to indicate DCT domains, and lowercase letters are used to indicate spatial domains). Each row of [d] contains all zeros except for the _eight elements of the filter [f] ₈ . In addition, the eight elements [f] ₈ of the filter in the row are shifted to the right by two places relative to the previous row.

At the decoder, it is not particularly practical for the operator [d] _270x540 to operate on every column of the entire field. Therefore, in accordance with the present invention and in the manner described below, filter [f] ₈ operates vertically in redundant and I blocks when they arrive. This interblock vertical filtering and down sampling can be combined with an interblock vertical 8 point IDCT in an efficient manner.

IA1. Field image (all macroblock fields are DCT coded): vertical

In order to vertically filter a particular 8 point block column, the DCT coefficients for that block column and the block columns above and below it must be received and stored.

Suppose an 8-point coefficient block column is vertically filtered and downsampled to produce 4 filtered and downsampled pixels [x _b '] _4x1 . Then, let [Xa] _8x1 and [Xc] _8x1 be the block columns above and below [X _b ].

The matrix [FT] _8x8 is an eight point forward DCT transformation matrix used by the two-dimensional DCT 18 of FIG. 1 to convert redundant and pixel values into DCT coefficients. Therefore, a row of [FT] _8x8 consists of a known 8 point DCT based vector. The transpose matrix of [FT] _{8 ×} 8 provides an 8 point IDCT transformation matrix. So [IT] _8x8 = [FT] ^T ,

Becomes

Matrix [IT ³ _{mid] 4x24} is the upper five rows, and the lower five rows to be removed from the [IT ^3], is defined as the middle 14 rows of the [IT ^3]. Matrix [IT ³ _mid ] _14x24 is applied to [X _b ] _8x1 and block column [X _a ] _8x1 according to the following equation.

Here, [X] _{14 × 1} is 14 full resolution pixels needed as input for the down sampling filter to calculate 4 down sampling pixels by block b (ie, block [X _b ] _{4 × 1} ). . If [d] _{4 × 14} is

If the downsampling filter given by is applied, this downsampling filter [d] is applied to 14 pixels [x] _14x1 , and the desired downsampling [x ' _b ] _4x1 (that is, [d] [x] _14x1 = [x' _{b). 4x1} ) is calculated.

IDCT and down sampling operations can be combined in a single step per block column according to the following equation.

By _applying this IDCT / down sampling filter [Q _dit ] _4x24 to 24 pixels of three block columns [X _a ], [X _b ] and [X _c ], the desired down sampling pixel [X _b '] is obtained according to the following equation. _4x1 is calculated.

IA2. Field Burn: 2-D Processing

The above description can be easily extended to a two-dimensional 8x8 field DCT coding block with vertical processing in each column using [Q _dit ] and horizontal processing in each row using 4-point horizontal IDCA as described above. Thus, X _b can be easily extended to an 8x8 DCT coding block. Each 8x8 DCT coefficient block X _b is redefined so that the upper and 8x8 DCT coefficient block below.

As a first step in extending the down sampling filter of the present invention to a two-dimensional 8x8 field DCT coding block, the high rank horizontal coefficients of each of blocks X _a , X _b , and X _c are discarded, whereby each is 8x8. An array (i.e. the left half of each block is retained) and [Z _a ] _8x4 becomes the left half of [X _a ] _8x8 , and [Z _b ] _8x4 and [Z _c ] _8x4 are defined similarly. These matrices are stacked and defined according to the following equation.

Subsequently, vertical filtering and down sampling of the pixel [Z] _24x4 can be performed according to the following equation.

Where [Q _dit ] _4x24 is the IDCT / down sampling filter given by equation (7). Horizontal IDCT domain filtering and down sampling can then be applied to the result [G _b ] _{4 × 4} of equation (10) according to the following equation.

Here, [T4] is a four point DCT based vector matrix including a known four point based vector.

IB1. Frame picture: Vertical processing: Assume that all macro blocks are frame DCT coded.

In the case of a picture where all macro blocks are frame DCT coded, it is desirable to perform frame-field based vertical 8 point IDCT, field based vertical interblock filtering and down sampling in a single efficient step.

The macro block column may be defined to be any 16 point column (ie two vertical block columns) of a 16 × 16 macro block. In order to authorize an interblock vertical operator, DCT coefficients for the macro column and the macro blocks above and below it must be received and stored. Therefore, [X _b] 1 _6xl the "in _4xl and for lower fields [xbot _b [xtop _b]" for the top field; 16-point frame coding to be filtered and down-sampled to produce an eight-field-based pixel defined by _4xl Assume that there is a macroblock column (ie two 8-point vertical blocks). It is also _assumed that [X _a ] _16xl and [X _c ] 1 _6xl are the frame DCT coding macro blocks above and below [X _b ]. The frame DCT coding macro block columns [X _a ], [X _b ] and [X _c ] may be combined according to the following equation. An unshuffling operator capable of prescaling a 16 pixel frame ordering column to reorder even and odd pixel lines in the upper and lower fields may be defined according to the following equation.

An unshuffling operator capable of pre-magnifying a 16 pixel frame ordering column to reorder even and odd pixel lines in the upper and lower fields can be defined according to the following equation.

The unshuffling operator [US] may be applied to an ordered macroblock column of pixels that is the inverse DCT of [X _b ] according to the following equation.

Where [xtop _b ] and [xbot _b ] are the pixels [x _b ] reordered into the top and bottom field pixels, respectively. If [FT8] is an 8-point DCT forward transform matrix with rows that are known 8-point DCT-based vectors, then the inverse of [FT8] is given by [FT8] ^T as follows.

Where [IT8 ₂ ] is the IDCT operator for two vertically stacked 8 point DCT columns. The operators [US] and [IT8 ₂ ] can then be combined into a single operator [USIT] according to the following equation.

The following equation is computed from Formula (14) and Formula (16).

The upper eight rows of [USIT] may be defined as [USIT _top ] _8x16 and the lower eight rows of [USIT] may be defined as [USIT _botp ] _8x16 . These newly defined operators are applied to [X _b ] according to the following equation.

The operator [USIT _top3 ] _24x48 is defined according to the following equation.

The operator [USIT _top3 ] _24x48 is applied to the coefficients of blocks X _{a '} , X _b' , and X _{c according to the} following equation.

However, only the middle 14 pixels are required as input to the down sampling filter. (I.e. the top 5 and bottom 5 pixels can be discarded from the right side of equation (21). Therefore, operator [MID _top ] _4x48 can be defined. The operator [MID _top ] _14x18 is only in the middle of [USIT _top3 ]. It contains 14 rows Intermediate _unshuffling operator [MID _top ] _4x48 can be applied to the coefficients of blocks X _{a ',} X _b', X _{c according to the} following equation.

Where [xtop] _14x1 is a top field and a down-sampling of the pixel [xtop _b '] a ring unloading shuffling is needed as an input to the upper field down-sampling filter, to calculate the filter _4xl 14 pixels. The middle lower _unshuffling operator [MID _bot ] _14x48 is derived in the same way. This intermediate lower _unshuffling operator [MID _bot ] _14x48 is applied to the coefficients of blocks X _a , X _b , X _c according to the following equation.

Where [xbot] _14x1 is lower field down-sampled and filtered pixel [xbot _b '] is a bottom field down-sampling the input 14 pixels ring unloading shuffling is needed as to the filter in order to calculate the _4xl. Downsampling filter [d ] _4xl4 can be combined with the IDCA / _unshuffling operator [MID] as IDCT, _unshuffling and downsampling filters [QL _top ] _4x48 and [QL _bot ] _{4x48 according} to the following _equation .

_Unshuffling , IDCT, and Down Sampling Filters [QL _top ] _4x48 and [QL _bot ] _4x48 provide block X _{a '} to perform 8-point vertical IDCT, frame-field conversion, vertical _interblock filtering, and downsampling in a single step. , X _{b '} and X _c can be applied to the coefficients.

_Unshuffling and Down Sampling Filters [QL _top ] _4x48 and [QL _bot ] _4x48 can be combined as a single _unshuffling and down sampling filter according to the following _equation .

Thus, the _unshuffling and downsampling filter [QL] _8x48 is _a block X _{a '} . May be applied to the coefficients of X _{b '} , and X _c .

IB2. Frame picture: Vertical processing: Assume that some macro blocks are frame DCT coded and some macro blocks are field DCT coded.

As mentioned above, the frame picture may comprise a mixture of frame and field DCT coding macroblocks. Thus, any macro block [X _a ], [X _b ] or [X _c ] is frame coded or field coded. For example, if this [X _a ] is field coded and [X _b ] and [X _c ] are frame coded, the field coded macroblock column [X _a ] is received according to the following equation.

Therefore, this macroblock column [X _a ] does not need to be divided into field columns as macro block columns [X _b ] and [X _c ] should be. Therefore, no unshuffling operator is required, and the IDCT operator is authorized according to the following equation.

It should also be noted that [IT8 _2top ] _8x16 [X _a ] _16xl = [xtop _a ] _8xl and [IT8 _2bot ] _8x16 [X _a ] _16xl = [xbot _a ] _{8xl '} .

The combined unshuffling and IDCT operator is defined by the following equation (similar to equation (20)).

Here, if the vertical macro block [X _n ] is frame coded, [OP _topn ] = [USIT], and if the vertical macro block [X _n ] is field coded, [OP] = [IT8 _2top ]. There are 2 ³ = 8 possible configurations for this flexible operator [USIT]. In the case of the field coding vertical macro block column [X _a ] and the frame coding vertical macro block column [X _b ] and [X _c ] of the above example, the operator [USOT] is given by the following equation.

As before, operator [MID _top ] _4x48 contains the middle 14 rows of [USIT _top3 ] _24x28 . In the same way, the flexible operators [USIT] and [MID _bot ] _14x48 can be derived. The [MID] operator may be combined with the down sampling filter [d] by the methods of (24) and (25) to form the flexible operators [QL _top ] and [QLbot] for any combination of frame and field DCT coding macroblocks. Can be.

Thus, for any combination of frame or field DCT coding macroblocks, a combining operator including frame and interblock filtering and downsampling for 8 point IDCT and field transform can be devised in the following equation.

The upper and lower [QL] operators can be combined according to the following equation.

Thus, the _unshuffling and down sampling filter [QL] _8x48 can be applied to the coefficients of blocks X _{a '} , X _b' , X _c in one single step per macroblock column according to the following equation.

IB3. Frame image: Two-dimensional processing

The above discussion is only about vertical processing of frame coded or field coded frame pictures, but as described above, the vertical processing is _performed in each column using [QL] _8x48 in Eq. (37), and the 4 point IDCT of each 8x8 block is used. By horizontally processing each row, it can be easily extended to two-dimensional 16x16 macroblocks.

Therefore, if X _{a '} , X _b' and X _c are redefined to be three vertically stacked 16x16 macroblocks, they are as follows.

The left and right halves of the macro block [X _a ] _16x16 may be redefined according to the following equation.

Macro blocks [X _bL ] _16x8 , [X _bR ] _16x8 , [X _cL ] _16x8 and [X _cR ] _16x8 are similarly defined.

As described above, as the first step in deploying the operator to perform vertical processing in each column and horizontal processing in each row of each 8x8 block with 4 point IDCT using [QL] _8x48 from equation (37), The high order horizontal coefficients of each 8x8 block [X] are discarded, and each 8x8 block [X] becomes an 8x4 block [Z]. Thus, [Z _a1 ] _8x4 is defined as the left half of [X _a1 ] _8x8 , [Z _a2 ] _8x4 is defined as the left half of [X _a2 ] _8x8 , and [Z _a3 ] _8x4 is the left of [X _a3 ] _8x8 Defined as half, [Z _a4 ] _8x4 is defined as the left half of [X _a4 ] _8x8 . These 8x4 blocks are combined according to the following equation.

[Z _bL ] _16x4, [Z _bR ] _16x4 , [Z _cL ] _16x4 and [Z _cR ] _16x4 can be defined similarly. Then, field based vertical filtering and down sampling of the left and right halves of [X _b ] are given by the following equation (using the flexible operator [QL] from equation (32)).

Then, horizontal DCT domain filtering and down sampling are performed according to the following equation.

Where [T4] is a 4-point DCT based vector matrix. Thus, two two-dimensional down-sampled and filtered macro blocks are given by the following equation.

IC. -IDCT module: block diagram (Figure 5, module 102)

The filtering and downsampling module 102 of FIG. 5 is shown in more detail in FIG. 6, and with the QL for the derived operator [Q _dit ] and frame picture for the field picture to perform vertical processing, and In addition, it uses a four-point IDCT operator [T4] for horizontal processing. Thus, the filtering and down sampling module 102 includes only field DCT coding macro blocks and field pictures (including all field DCT coding blocks) or a mixture of field DCT coding macro blocks and frame DCT coding macro blocks. All frame images can be processed. The output of the filtering and down sampling module 102 is always field based filtering and down sampling. All blocks are the same horizontal processing (4 point IDCT), but in vertical processing, the [QL] operator is used when an effective conversion from field coding to field coding is required, followed by filtering and downsampling each field separately. Used. Note that the matrix product does not hold the exchange law (ie A * B is generally not equal to B * A). Therefore, each matrix multiplier has a pre and post input.

IC1. Field burns

Coefficient divider 120 in FIG. 6 discards four high order horizontal coefficients from each fhdn of every 8x8 DCT coefficient block. The memory 122 stores three rows of I or redundant coefficient macro blocks. (The rows of macro blocks consist of rectangular pixels of one macroblock as high as the number of macroblocks needed to go from the left to the right side of the picture.) In the case of untransmitted redundant macroblocks, all coefficients are assumed to be zero. . Three vertically stacked blocks [Z] _24x4 at one time (see equation (9)) are read from the memory 122 and the vertical matrix multiplier 124 by the router 126 according to the DCT coding type (field or flame). Headed to. Vertical matrix multiplier 124 performs 8 point vertical IDCT using vertical internal block filtering and down sampling in accordance with equation (10). Accordingly, vertical matrix multiplier 124 generates [G _b ] _4x4 by premultiplying three vertically stacked blocks [Z] _24x4 by [Q _dit ] _4x24 . Selector 128 directs vertically processed block G to horizontal matrix multiplier 130. Horizontal matrix multiplier 130 performs horizontal four point IDCT on [G _b ] according to equation (11) to produce [x _b '].

IC2. -Frame image

The coefficient divider 120 discards four high order horizontal coefficients from each row of every 8x8 coefficient block as before. The memory 122 stores three rows of I or redundant coefficient macro blocks. For redundant macroblocks not transmitted, all coefficients are assumed to be zero. The three vertically stacked macroblocks consist of [Z _L ] _48x4 so that at any time [Z _R ] _48x4 (see equations (45) and (46)) is read from the memory 122 and the router 126 To the vertical matrix multiplier 132 according to the DCT coding format (field or frame). Router 126 also routes the DCT coding type (field or frame) for each macro block to QL module 134. Depending on the coding type of the three vertically stacked macroblocks containing [[Z _L ] and [Z _R ], the QL module 134 previously mentioned the "before" input for the vertical matrix multiplier 132. Create a flexible [QL] operator as shown.

Vertical matrix multiplier 132 is an 8-point vertical IDCT and field-frame transform in a frame coding macroblock, an interlock at [Z _L ] and then at [Z _R ] as in equations (45) and (46). Perform filtering. The output from the vertical matrix multiplier 132 is first

And then

이 된다.

Becomes

These output matrices are fed to the storage and delay module 136, which provides the following 4x4 blocks to the horizontal matrix multiplier 130 through the selector 128 one at a time.

[Gtop _bL ], [Gbot _bL ], [Gtop _bR ], [Gbot _bR ]

The horizontal matrix multiplier 130 performs horizontal 4-point IDCT on the row of each 4x4 input block to generate the next corresponding 4x4 output block according to equations (47) and (48).

[xtop _bL '], [xbot _bL '], [xtop _bR '], [Gbot _bR ']

II. Upsampling and Downsampling Prediction

IIA. Derivation of the vertical upsampling filter matrix [u]

The vertical predictive filtering implemented by the motion compensator 104 of FIG. 5 will now be described. The upsampling filter [u] applied by the motion compensator 104 is determined from the known Moore-Penrose inverse of the downsampling operator [d] from equations (2) and (3) according to the following equation.

The up sampling filter [u] is applied to the column of the reduced resolution reference picture [x '] _b270xl from the memory 106 according to the following equation.

Where [x '] _b270xl is the final upsampling column of the resulting pixel. The structure of [u] is based on the following 21 tap filters (with 13 non-zero taps).

[g ₁ (-10) .. g ₁ (10)] = [0 0 0 0 0 0 0 0 .0011 .0372 1.2542-.4051. 1546 -.0576 .0215 -.0081 .0030 -.0011 .0004 -.0002 .0001] and

[g ₂ (-10) .. g ₂ (10)] = [0001 -.0002 .0004 -.0011 .0030 -.0081 .0215 -.0576 .1546 -.4051 1.2542 .0372 .0011 0 0 0 0 0 0 0 0]

The filter [g ₂ ] has the same tap value as the filter [g ₁ ] except in the reverse order and when g ₁ (0) = g ₂ (0) = 1.2542. The up sampling filter [u] is a circular matrix composed of one step shifted versions of [g ₁ ] and [g ₂ ] in alternating rows. Therefore, the up sampling filter [u] is given by the following equation.

Sufficient prediction can be achieved using only the non-zero taps with the eight largest (or smaller numbers) of g ₁ and g ₂ . The smaller the non-zero taps, the fewer reference data to be read from the memory 106 for each macro block prediction. For example, sufficient prediction can be obtained using a filter given by the following equation.

The vertical upsampling operation using the filters g ₁ and g ₂ of equations (53) and (54) can be considered as follows. In order to calculate the upsampling pixel above a particular reference pixel in the downsampling reference field, the median value of the filter g ₁ (ie, the value at g ₁ (0)) is adjusted to the reference pixel, and the non-zero tap and its The sum of the products between the aligned pixels is performed. To calculate the upsampling pixel below this particular reference pixel in the downsampling reference field [x '], the median of filter g ₂ (ie, the value at g ₂ (0)) is adjusted to the reference pixel and is non-zero. The sum of the products between the tap and its adjusted pixels is performed.

In the down transform decoder 100, when the upsampling reservation is formed with the motion compensator 104, not all of the columns need to be read from the reference memory 106 to be filtered with [u]. Instead, only enough pixels need to be read to satisfy the combined requirements of the predictive filters formed from the up sampler filters [g ₁ ] and [g ₂ ] and the down sampler [f] as described below.

IIB. Combined prediction upsampling and downsampling

IIB1. Field prediction for field pictures

As shown in FIG. 7, the first column represents the portion of pixels [x] _50xl from the field in the full resolution reference picture 16 as it exists in the encoder 10 of FIG. 1. The second column represents the corresponding portion of the column of pixels [x '] _25xl from that field in the down sampling criterion 106 of the decoder loop of FIG. The vertical full resolution motion vector vmv for the desired field prediction is pointed at the pixel indicated in the first column. This motion vector is in 1/2 pixel units, so if mv is an odd number it is pointed between the pixels at full resolution.

Consider a modulo value of vmvtype defined by                     

Here,% means modulo operation. The value of vmvtype will be 0 or 1, and FIG. 7 illustrates both cases. When vmvtype = 0, the vertical vector vmv is pointed at or just below a pair of pixels in the full resolution reference field (ie, pixels with a rectangular box around them in the first column of FIG. 7). Pixels in the down-sampled reference field (ie, pixels with a square box around them in the second column of FIG. 7) lie between the pixels of this pixel pair. When vmvtype = 1, the vertical motion vector is pointed at or just below the bottom pixel of this pixel pair. Again, the pixel in the down sampling reference field lies between the pixels of this pixel pair. The low resolution motion vector used to access the prediction from the down sampling reference field is given by the following equation.

This low resolution motion vector lvmv is pointed to a pixel in the down sampling reference field (ie, a pixel with a square box around it in the second column of FIG. 7). The input to the predictive filter must include that pixel in a down sampling reference field with eight pixels above and sixteen pixels below it. These 25 pixels shown in the second column of FIG. 7 are the exact amount that should be read as input to the vertical prediction filtering process to produce the predicted vertical field macroblocks.                     

The next column of FIG. 7 is [x ^] _24xl which represents the upsampling field prediction generated with the filters g ₁ and g ₂ arranged as shown in equation (52). In the second column of Fig. 7, the circular pixel shows an upper limit and a lower limit, between which the g ₁ and g ₂ filter center taps traverse the up sampling convolution filtering operation. This upsampling filtering operation is given by the following equation.

Where [x '] _25xl represents a 25 pixel column from a reference picture, [x ^] _24xl represents an upsampled pixel, and the upsampling filter [u] is based on equations (53) and (54) , Given by

After upsampling, subsequent predictive downsampling operations are needed to produce the final desired downsampling field prediction macroblock [x ^] _8xl column. This downsampling described later depends on whether vmvtype (given by equation (56)) and vmv are good or odd.

vmvtype = 0

For the case of even-value vertical motion, the desired down sampling field prediction is given by the convolution of [x ^] _24xl to [f] ₈ as follows.

Here, [x ^] _24xl is a column of upsampling reference pixels to be downsampled, [x ^ '] _8xl is a resultant downsampling reference pixel, and the predicted down sample filter [d ₀ ] is given by the following equation.

And where [f] 8 is given by equation (1). Vertical prediction up-sample and down-sample filtering can be combined into a single operation given by the following equation.

Where [x '] _25xl is the reference pixel column from memory 106, [x ^'] _8xl is the pixel from predictive upsampling and downsampling, and the upsampling and downsampling filter [PL _0-ev ] Is given by

If vmv is a radix value, half linear interpolation, given by

Where [x ^] _25xl is the pixel resulting from the predictive upsampling, [x _i ^] _24xl is the predicted upsampled pixel resulting from the linear interpolation, and the linear interpolation filter [LI] is given by

Subsequently, down sampling is performed according to the following equation.

Vertical predictive upsampling, linear interpolation and predictive downsampling can be combined into a single operation as given by                     

Here, the combined vertical predictive upsampling, linear interpolation and predictive downsampling filter [PL _0-odd ] _8x25 are given by the following equation.

vmvtype = 1

For the case of the even-valued vertical motion vector vmv, the desired _downfield prediction is given by the convolution of [f] ₈ at [x ^] _24xl as follows.

Here, the predictive down sampling filter [d ₁ ] is given polynomial.

Vertical prediction upsample and downsample filtering can be combined into a single operation according to the following equation.

Here, the combined vertical prediction up sample and down sample filter [PL _1-ev ] are given by the following equation.

For the case where the odd value is the vertical motion vector vmv, the half pixel linear interpolation given by equation (63) must be performed after the upsampling. Then down sampling is performed according to equation (67). Vertical upsampling, linear interpolation and downsampling can be combined in a single operation given by

Here, the combined vertical upsampling, linear interpolation and downsampling filter [PL _1-odd ] is given by the following equation.

IIB2. Field prediction for frame pictures

In the case of field prediction for a frame picture, two independent field predictions for the frame macro block are required. As shown in FIG. 8, the first column represents the portion of the pixel [x] _42xl column from one field in the full resolution reference image 16 as present in the encoder 10 of FIG. The second column represents the corresponding portion of the pixel [x '] _21xl column from that field in the down sampling criterion 106 in the decoder loop of FIG. The full resolution vertical motion vector vmv for the desired field prediction is pointed to the indicated pixel in the first column. This motion vector is placed in units of 1/2 pixel. Thus, if the full resolution vertical motion vector vmv is an odd value it is pointed between the pixels at full resolution.

Consider again the modulo value of vmv given by equation (55). As indicated, the value of vmvtype will be 0 or 1. 8 illustrates two cases. When vmvtype = 0, the vertical motion vector vmv is pointed at or just below the top pixel of the pair of pixels in the full resolution reference field (i.e. the pixel with a rectangular box around them in the first column of FIG. 8). . Pixels in the down-sampled reference field (ie, pixels with a square box around them in the second column of FIG. 8) lie between the pixels of this pixel pair. When vmvtype = 1, the vertical motion vector vmv is pointed at or just below the bottom pixel of this pixel pair. Again, the pixels in the down sampled reference field lie between the pixels of this pixel pair. The low resolution motion vector lvmv used to access the prediction from the down sampled reference field is given by equation (56). This low resolution motion vector lvmv is pointed at a downsampled reference field (second column) with a box around it. The input to the prediction filter should include that pixel, eight pixels above it, and twelve pixels below it. These 21 pixels are the exact number of pixels that must be read as input to the vertical prediction filtering process to produce the predicted vertical field block.

The third column in FIG. 8 is [x ^] _16xl , which represents the upsampling field prediction produced by the filters g ₁ and g ₂ given by equations (53) and (54). Sampled pixels in the second column [x ＇] _21xl exhibit an upper limit and a lower limit such that the center taps of the filters g ₁ and g ₂ are traversed during the up sampling convolution operation.

The upsampling operation is given by

Here, [x '] _21xl is 21 pixels from the reference picture stored in the memory 106, [x ^] _16xl is the resulting upsampled pixel, and the upsampling filter [u] is given by the following equation.

Subsequently, subsequent prediction down sampling operations are required to produce the final desired down sampled field prediction [x ^ '] _4xl . This downsampling, which will be described below, depends on whether vmvtype and vmv are even or odd values.

vmvtype = 0.

For the case where the vertical motion vector vmv is an even value, the desired down sampled prediction is given by a convolution of [x ^] _16xl to [f] _{8 according} to the following equation.

Where [x ^] _16xl is the upsampling pixel to be downsampled, [x ^] _4xl is the resulting downsampling pixel, and [d ₀ ] is the downsampling filter given by the following equation.

Vertical prediction up-sample and down-sample filtering can be combined into a single operation given by

Here, the combined up sampling and down sampling filter [PL _o-ev ] is given by the following equation.

For the case where the vertical motion vector vmv is an odd value, after the upsampling, 1/2 pixel linear interpolation, which is given by the following equation, should be performed.

Where [x] _16xl is the up-sampled pixel to be linearly interpolated, [x ^ _i ] _16xl is the resulting linearly interpolated pixel, and the linear interpolation filter is given by the following equation.

Subsequently, down sampling is performed according to the following equation.

Where [x ^ _i ] _16xl is the upsampling and linearly interpolated pixels to be downsampled, [x ^ '] _4xl is the resulting downsampling pixel, and [d _o ] is the downsampling given by equation (76) Filter.

This vertical upsampling, linear interpolation and downsampling combine into a single operation given by

Here, the combined vertical upsampling, linear interpolation and downsampling filter [PL _o-odd ] is given by

vmvtype = 1.

For the case where the vertical motion vector vmv is an even value, the desired down sampling field prediction is given as a convolution of [x ^] _16xl to [f] _{8 according} to the following equation.

Here, the downsampling filter [d ₁ ] is given by the following equation.

Vertical prediction upsample and downsample filtering can be combined in a single operation given by

Here, the combined upsampling and downsampling filter [PL _1-ev ] is given by the following equation.

For the case where the vertical motion vector vmw is an odd value, after the upsampling, 1/2 pixel linear interpolation, which is given by the following equation, should be performed.

Here, the linear interpolation filter is given by equation (80). Subsequently, down sampling may be performed according to the following equation.

Vertical upsampling, linear interpolation and downsampling can be combined in a single operation given by

Here, the combined vertical up sampling, linear interpolation and down sampling filter [PL _1-odd ] is given by the following equation.

IIB3. Frame Prediction for Frame Images

For the case of frame prediction, two field predictions must be derived by the filtering and down sampling module 102 to match the surplus values converted from the frame DCT coding macroblock to the field DCT coding macroblock.

As shown in FIG. 9, the first column represents the vertical column portion of pixels from the full resolution reference frame 16 in the encoder 10 of FIG. 1. The second column represents the corresponding portion of the pixel [xtb '] _42xl column from that frame in the down sampling criteria from memory 106 of the decoder loop. The full resolution motion vector vonv for the desired frame prediction is pointed at the symbol indicated in the first column. This motion vector is pointed at a half pixel. This motion vector is melted in units of 1/2 pixel. If vmw is an odd value, it is pointed between the pixels at full resolution.

The input to the predictive filter of the motion compensator 104 is a group of 42 vertical pixels [xtb '] in the second column. In order to accurately access the group of 42 vertical pixels, the vertical component of the full resolution motion vector is transformed according to the following equation.                     

This low resolution vertical motion vector lvmv will be pointed to the top field pixel (xtop ') in the down sampling reference frame. Input to the predictive filter would have to include 16 pixels above it and 25 pixels below it.

The vertical motion vector vmv can be classified into one of four types based on the value vmvtype, each of which has a different spatial relationship with the down sampling reference frame as shown in FIG. These classifications are given by

Here, vmvtype may have a value of 0, 1, 2 or 3 according to equation (9). From the definitions of [g ₁ ] ₁₃ and [g ₂ ] ₁₃ in shims 53 and 54, respectively, two 25 tap filters can be defined according to the following equation.

Then, the up sampling filter [u] is given by the following equation.                     

Independent upsampling for each field of [xtb '] _42xl from the down sampling reference frame of FIG. 9 is performed according to the following equation.

Therefore, the upsampling prediction consists of [x ^ top] _16xl and [x ^ bot] _{16xl '} , which become columns 3 and 4, respectively, in FIG.

Subsequently, a subsequent predictive downsampling operation is required (in each upsampling field) to generate the final desired field prediction pair. Downsampling, described below, depends on vmvtype and whether vmv is an even or odd value. .

vmvtype = 0 and vmv is an even value.

The down sample filter for each field prediction is given by                     

Subsequently, the down sampling filter [d _o ] _8x32 is defined according to the following formula.

By downsampling each upsampling field independently, the desired field prediction is generated as given by the following equation.

The combined up-sample down-sample filter is given by

This combined up sample-down sample filter is also used to generate the desired field prediction, as given by

vmvtype = 0 and vmv is an odd value.

As described above, independent upsampling field prediction is given by the following equation.

Individual upsampling fields should be supple and linearly interpolated. The suppling / interpolation operator is given by

This suppling and linear interpolation performed in the up sampling field is given by the following equation.

By independently downsampling each upsampling, supplementing, and linearly interpolated field, the desired prediction field given by

Here, the down sampling filter [d _o ] _8x32 is given by equation (?).

The upsampling filter [u], the suppling and linear interpolation filter [SLI], and the downsampling filter [d _o ] can be combined with the following equation.

And the desired field prediction is given by

vmvtype = 1

The down sampler for each field prediction is given by

And can be combined into a single operator as given by                     

Using this, the combined down sampling filter [d ₁ ], the combined filter [PL _1-ev ] _8x42 and [PL _1-odd ] _8x42 (shown in equations (101) and (108)) [PL _o-ev ] And [PL _o-odd ] can be derived in a similar manner, and can be used according to the following equation to generate the desired field prediction for the case where vmv is an excellent value

And if vmv is a radix value it can be used according to the following equation.

vmvtype = 2

The downsampling filter for each field prediction is given by

And it can be combined according to the following equation.                     

By using the down sampling filter of equation (116) in the same way as the derivation for vmvtype = 0, the combined filters [PL _2-ev ] _8x42 and [PL _2-odd ] _8x42 are derived and vmv is an even value Regarding the following expression

Can be used to generate the desired field prediction, and for vmv is an odd value,

Can be used to generate the desired field prediction.

vmvtype = 3

The down sampler for each field prediction is given by

These two down sampling filters can be combined according to the following equation.

By using the down sampling filter of equation (121) in the same way as the derivation for vmvtype = 0, 1, 2, the combined filters [PL _3-ev ] _8x42 and [PL _3-odd ] _8x42 can be derived, If vmv is an even value,

Can be used to generate the desired field prediction, and for vmv is an odd value,

Can be used to generate the desired field prediction.

ICC. Overview of derived vertical prediction filter operators

ICC 1.-Field prediction for field pictures

The next operator from equations (62), (66), (70) and (72) is used for field prediction of the field image and is an 8x25 matrix.

IIC2. Field prediction for frame pictures

The following operators from equations (78), (87) and (91) are used for field prediction of the frame picture and are a 4x21 matrix with two predictions required per macroblock.

IIC3. -Frame prediction for frame pictures

The following operators from equations (101), (108) and similar equations are used for frame prediction of the frame field picture, which is an 8x42 matrix.

IID. Motion Compensation Block Diagram (FIG. 5, Module 104)

The motion compensator 104 is shown in more detail in FIG. 10. Motion vector translator 150 translates the received full resolution motion vector into a low resolution motion vector as described above. This low resolution motion vector is then converted to a reference picture memory address for supply to memory 106. Vertically, this translation and transformation depends on the prediction type (frame or field) as described in the previous discussion. Reference pixels read from memory 106 by address from motion vector translator 150 are supplied to horizontal prediction filter 152. Horizontal prediction filtering occurs in horizontal prediction filter 152 and is the same as described above with respect to minimum drift prediction for 4-point IDCT.

Supplied to the horizontal prediction filter 154. Vertical prediction filtering (upsampling and downsampling) is performed by the vertical prediction filter 154 according to the prediction type and vmv value (odd or even) as described above. Thus, depending on the prediction type and vmv value (odd or even), the correct vertical filter operator [PL] is read from memory and read from each column of pixels when they are received from horizontal prediction filter 152. The horizontally and vertically filtered predictions are fed to the adder 108 (FIG. 5) by the vertical prediction filter 154 and added to the incoming redundant macroblocks from the filtering and down sampling module 102.

The spatial effect of the vertical down sampling method performed by the filtering and down sampling module 102 of FIG. 5 should be considered. 11 shows the original full resolution pixels of the interlaced picture (pixel A for the top field and pixel B for the bottom field, as shown in the first column of FIG. 11) and the pixels of the down-sampled interlaced picture. 11 shows the effective spatial relationship between (pixel "a" for the upper field and pixel "b" for the lower field) as shown in the second column of FIG. Note that the "a" pixels are halved between all other pairs of A pixels and the "b" pixels are halved between all other B pixels. Also note that in the original full resolution image, the B pixels are halved between the A pixels. However, in the resulting down sampled image, it can be seen that the "b" pixels are not halved between the "a" pixels.

Therefore, the interpolator 110 of FIG. 5 includes a vertical filter to effectively adjust the vertical position of the "a" and "b" fields so that the "b" pixels are between the "a" pixels as shown in FIG. In the middle. An example vertical filter consists of a simple linear interpolator that shifts up the "a" pixels to a 1/8 sample position and shifts down the "b" field to a 1/8 sample position. More complex filters than these linear interpolators can also be used.

Also, by changing the filtering and down sampling module 102 of FIG. 5 to achieve the 1/8 sample shift for each of the fields, the need for the interpolator of FIG. 5 can be eliminated. In this case, two different filters may be needed as a replacement for f ₈ in equation (1), one for A or the upper field, and one for B or the lower field. The downsampling filter for the A field filter, given by the following equation, is derived in a manner similar to that of equation (1), but may include 1/8 sample shift.

Likewise, the downsampling filter for the B field given by the following equation includes 1/8 sample shift in the opposite direction.

Each matching prediction filter must be derived for each of the fields in the exact manner as described above.

IIE. Drift reduction for the case of 1/2 pixel resolution motion vectors

As mentioned above, the motion vectors for MPEG-2 encoded video have a 1/2 pixel resolution. This half-pixel resolution is used when forming a prediction from a full resolution criterion or an upsampled reduced resolution criterion, as in the case of the down conversion decoder 100 of FIG. 1/2 interpolation is required. The MPEG-2 standard specifies that half-pixel interpolation for the odd-valued horizontal motion vector or even-valued vertical motion vector should be calculated according to the following equation.

Here, x and y are integers (assumed to be 8 bits) indicating the l value of the full resolution reference pixel.

This equation shows how to handle the least significant bit (LSB) of the computed interpolation pixel. If x + y = S and S is good, then S / 2 = I.5, where I is an integer and Eq. (126) will effectively round up to I + 1. This increase from 1.5 to I + 1 is equivalent to the increase in 1/2 LSB. In the mean, if half of time S is excellent and half of time S is odd, equation (126) requires an average increase of 1/4 LSB in the calculated interpolation prediction pixel. A decoder that linearly combines the calculation of S / 2 with other filtering operations and then rounds the result (like the single-step matrix operator of equation (66) executed by vertical prediction filter 154 in FIG. 10). Will effectively hold the fractional part of S (S value is I.5, 50% of the time instead of the desired value I + 1) until the final round off. Therefore, such a decoder will show a DC prediction drift towards the darker picture until the next I frame.

Therefore, in order to more closely meet the requirements of equation 126 for operation as the requirements of equation 65 when the motion vector is an even radix value, the prediction output from vertical prediction filter 154 of FIG. It must be changed by adding .25 (equivalent to 1/4 LSB) to its output and then rounding the prediction result to an 8-bit integer before being provided to the adder 108 of FIG.

IIE2. 1/2 pixel interpolation in both directions

When both horizontal and vertical motion vectors are odd values, the standard specifies four-way interpolation given by

Equation (127) represents a method of processing the LSB of the calculated interpolation pixel. If a + b + c + d = S and S is evenly distributed by 4 (case 1), the operation is exactly equal to S / 4. Alternatively, there are three other considerations (% indicates modulo operation). In the case of case 2, (S / 4)% 4 = 1 and therefore S / 4 = I.25. Equation (127) reduces S / 4 by 0.25 (-1/4 LSB). In case 3, (s / 4)% 4 = 2, so S / 4 is I.5. Equation (127) increases S / 4 by 0.5 (+1/2 LSB). In Case 4, (S / 4)% 4 = 3, so S / 4 is I.75. Equation (127) increases S / 4 by 0.25 (+1/4 LSB).

Assuming all four cases are equally possible, equation (127) in the average requires an increase of (0-0.25 + 0.5 + 0.25) in the calculated interpolation prediction pixel. Therefore, for the case of motion vectors of horizontal and vertical radix values, the prediction output of FIG. 10 adds .125 (equivalent to 1/8 LSB) to the vertical prediction filter 154 and then adds the prediction result to FIG. 5. Must be changed by rounding to an integer of 8 bits before being provided to the adder 108 of.

III. Expansion to other down sampling rates

Note that in the foregoing section, both horizontal and vertical down sampling should be performed with a factor of two. However, the method is not so limited, it can be extended to other down sampling ratios in an easy way. For example, as a result of horizontal downsampling through the use of three point IDCT, horizontal downsampling is performed with an 8/3 factor, which is known in the art. The novel vertical down sampling method described herein can be extended to down sampling, for example with an 8/5 factor. A vertical filter f, similar to the filter of equation (1), may be used (this filter probably has somewhat less attenuation at higher frequencies). The down sampling process is decomposed by up to five filters into up samples and then down sampled to eight. The minimum drift prediction filter may be derived from the down sampling filter f as described above.

Some variations of the invention have been described above. Those skilled in the art will be able to make other modifications. For example, according to the above description, 14 rows are used to perform vertical IDCT during down conversion. However, any number of rows can be used instead.                     

Accordingly, the detailed description of the invention has been made by way of example only, and is intended to present to those skilled in the art the best mode of carrying out the invention. The invention is susceptible to many different modifications without departing from the spirit of the invention and has the exclusive right to all modifications within the scope of the appended claims.

Claims (28)

A method of down converting receive frame and field DCT coding blocks into a reconstructed pixel field block, each of the receive frame and field DCT coding blocks comprises an N × M value, a) converting the received frame DCT coding blocks into transform field DCT coding blocks; b) performing horizontal M point IDCT, vertical N point IDCT, vertical spatial filtering and down sampling on the receive field DCT coding blocks and the transform field DCT coding blocks to produce redundant and pixel field blocks; The vertical spatial filtering and downsampling comprises at least N or more points, where N is N> M; c) adding prediction reference pixels to the redundant field blocks to form reconstructed pixel field blocks. The method of claim 1, The down sampling of step b) is based on a lowpass symmetric FIR filter. The method of claim 2, And said step a) comprises applying an unshuffling operator to said receive frame DCT coding block to convert said receive frame DCT coding blocks into a corresponding transform field DCT coding block. The method of claim 3, And said unshuffling operator comprises an element arranged to transform said received frame DCT coding block and field DCT coding block into a transform field DCT coding block. The method of claim 3, And the unshuffling operator comprises an element arranged to convert the received frame DCT code blocks and the mixer of field DCT code blocks into transformed field DCT code blocks. The method of claim 1, Step b) includes applying an operator [Q _dit ] to the receive field DCT coding block, where [Q _dit ] is given by

Where [d] is typically a down sampling filter having the following structure,

Where [f] is a low pass symmetric FIR filter, where [IT] is typically a vertical IDCT operator with

Here, [FT] is a down conversion method comprising an N point DCT based vector. The method of claim 6, And a post-processing step of applying the following operator to the result of the operation as described in claim 6.

The method of claim 1, Steps a) and b) include applying an operator [Q _L ] to a receive macro block containing a frame DCT coding block, where [Q _L ] is given by

Where [US] generally has the structure

Where [IT] is typically a vertical IDCT operator having the structure

Here, [FT] includes an N point DCT based vector, [d] is a down sampling filter having the following structure in general,

Where [f] is a lowpass symmetric FIR filter. The method of claim 1, Steps a) and b) include applying an operator [Q _L ] to a receiving macro block comprising a mixed frame and field DCT coding block, where operator [Q _L ] is given by

Here, [OP] is a matrix containing only [US] [IT] for a picture with only a frame DCT coding block, where [OP] is [IT] for a picture with both a frame and a field DCT coding block. And [US] [IT], wherein [US] generally has the structure

Where [IT] is typically a vertical IDCT filter with the structure

Here, [FT] includes an N point DCT based vector, [d] is a down sampling filter having the following structure in general,

Where [f] is a low pass symmetric FIR filter. The method of claim 1, The frame and field DCT coding block has an associated motion vector, and step c) c1) selecting a prediction reference pixel using the motion vector; And c2) appropriately adding the selected prediction reference pixel to the redundant field block to form a reconstructed field block. The method of claim 1, The frame and field DCT coding block has associated full resolution vertical and horizontal motion vectors, and step c c ₁ ) selecting a prediction reference pixel based on the full resolution motion vector; c ₂ ) upsampling the prediction reference pixel; c ₃ ) down sampling the upsampled prediction reference pixel; And c ₄ ) adding the up-sampled and down-sampled prediction reference pixels to a redundant field block to form a reconstructed field block. The method of claim 11, For field prediction on a field picture, steps c2) and c3) include applying a next operator to a prediction reference pixel,

Where [d ₀ ] is typically a down sampling filter having the structure

[U] is an upsampling filter having the structure

Wherein [g ₁ ] and [g ₂ ] are derived from the equation

Where [d] is typically a down sampling filter having the following structure,

Where [f] = f (-n + 1) ... f (n) = low-pass symmetric FIR filter. The method of claim 11, Regarding field prediction for the field image, steps c ₂ ) and c ₃ ) include applying the next operator to the prediction reference pixel,

Where [d ₀ ] is typically a down sampling filter having the structure

Where [u] is typically an upsampling filter having the structure

Here, [g1] and [g2] are calculated | required by the following formula,

Where [d] is typically the next sampling filter having the structure

Where [f] = f (−n + 1)... F (n) = low pass symmetric FIR filter, where [LI] is a linear interpolator which generally has the following structure.

The method of claim 11, Regarding field prediction for the frame picture, steps c2) and c3 include applying the next operator to the predictive reference pixel, [d ₀ ] [u] Where [d ₀ ] is typically a down sampling filter having the structure

Where [u] is typically an upsampling filter having the structure

Where [d] is a down sampling filter having the general structure

Where [f] = f (−n + ₁ )... f (n) = downpass metric FIR filter. The method of claim 11, Regarding frame prediction for a frame picture, the steps c2) and c3) include applying a next operator to a prediction reference pixel,

Where [d ₀ ] is typically a down sampling filter with

Where do ₁ and do ₂ are given by

Where [u] is typically an upsampling filter having the structure

Wherein [gx ₁ ] and [gx ₂ ] are derived from the equation

Where [d] is a downsampling filter generally having the following structure,

Where [f] = f (−n + 1)... f (n) = downpass metric FIR filter. A method of decoding a received first DCT coefficient block into a reconstructed field pixel block, the method comprising: a) applying a vertical operator and a horizontal operator to a first DCT coefficient block to generate intermediate surplus or pixel values, wherein the vertical operator comprises the first DCT coefficient block and the second and third DCT coefficients; Applied simultaneously to the coefficients of the block, the second DCT coefficient block is above the first DCT coefficient block, the third DCT coefficient block is below the first DCT coefficient block, and the horizontal operator is the first DCT Applied to the second and third DCT coefficient blocks and asynchronously with a coefficient block; And b) appropriately adding prediction reference pixels to the intermediate surplus value to form reconstructed pixels. The method of claim 16, And said step a) further comprises applying a vertical down sampling filter operator to produce an intermediate pixel value. The method of claim 17, The vertical operator and the vertical down sampling filter operator are combined as a single operator. The method of claim 18, The first, second and third coded DCT coefficient blocks are field DCT coding blocks, wherein step a) includes applying an operator [Q _dit ] to the receive field DCT coding block, where the operator [Q _dit Is given by [Qdit] = [d] [IT] Where [d] is a downsampling filter generally having the following structure,

Where [f] is a low pass symmetric FIR filter and [IT] is typically a vertical IDCT filter with

[FT] is a horizontal operator including an N point DCT based vector and [T] being a horizontal operator including an M point DCT based vector. A method of decoding a received first DCT coefficient macro block having a frame DCT coding block into reconstructed field pixel blocks, the method comprising: a) applying a vertical operator and a horizontal operator to a first DCT coefficient macro block to generate intermediate surplus or pixel values, wherein the vertical operator is configured to perform the first DCT coefficient macro block and the second and third DCT coefficients; Applied simultaneously to the coefficients of the macro block, the second DCT coefficient macro block is above the first DCT coefficient macro block, the third DCT coefficient macro block is below the first DCT coefficient macro block, and the horizontal operator Is applied to the second and third DCT coefficient macro blocks asynchronously with the first DCT coefficient macro block; And b) appropriately adding predictive reference pixels to the intermediate surplus value to form reconstructed pixels. The method of claim 20, Step a) further includes applying an unshuffling operator to transform the field DCT coding block of the frame DCT coding block of the first DCT coefficient macro block and the frame DCT coding block of the second and third DCT coefficient macro blocks. Down conversion method comprising a. The method of claim 21, And said step a) further comprises applying a vertical down sampling filter to generate intermediate surplus or pixel values. The method of claim 21, The unshuffling operator has a plurality of elements, and almost all of the elements of the unshuffling operator correspond to the frame coding block of the first DCT coefficient macro block and the frame DCT coding block of the second and third DCT coefficient macro blocks. Down conversion method which is arranged to convert to a DCT coding block. The method of claim 20, The first, second and third DCT coefficient macro blocks are comprised of a mixed block of frame DCT coding blocks and field coding blocks, wherein step a) further comprises applying an unshuffling operator, where: And the ring operator has a plurality of elements, wherein almost all elements of the unshuffling operator are arranged to transform a mixed block of the frame DCT coding block and the field DCT coding block into a corresponding field DCT coding block. The method of claim 20, The first, second and third macroblocks comprise a frame DCT coding block, wherein step a) comprises applying an operator [Q _L ] to the first, second and third macroblocks, wherein The operator [Q _L ] is given by [Q _L ] = [US] [IT] [d] Where [US] generally has the following structure,

Here, [IT] is generally a vertical operator having the structure

Where [FT] is an N point DCT based vector, [d] is a down sampled filter with

Where [f] is a low pass symmetric FIR filter. The method of claim 20, Said step a) comprises applying an operator [Q _L ] to the first, second and third macroblocks, [Q _L ] = [OP] [d] Here, [OP] is a matrix including only [US] [IT] for a picture having only a frame DCT coding block, where [OP] is a [IT] and for a picture having both a frame and a field DCT coding block. [US] A matrix containing [IT], where [US] generally has the following structure:

Where [IT] is typically a vertical IDCT filter with the structure

Here, [FT] includes an N point DCT based vector, [d] is a down sampling filter having the following structure in general,

Where [f] is a low pass symmetric FIR filter. An apparatus arranged to reconstruct pixels from a target DCT coefficient macro block, the apparatus comprising: A vertical operator having a size sufficient to be simultaneously applied to the target DCT coefficient macro block and the adjacent DCT coefficient macro block; A horizontal operator arranged to horizontally filter the target DCT coefficient macro block to produce intermediate pixel values in association with a vertical IDCT filter; And And an adder arranged to add predictive reference pixel values to intermediate pixel values to form reconstructed pixels. The method of claim 27, And further comprising a vertical down sampling filter arranged to produce down sampled intermediate pixel values.

Images