KR100563015B1 - Upsampling Filter and Half-Pixel Generator for HDTV Downconversion System - Google Patents

Abstract

A video down conversion system conforming to the Advanced Television System Standard (ATSC) includes a decoder that uses a down conversion processor to decode the main profile, high level (MP @ HL) picture and generate a standard definition video signal. The system stores the subsampled image to reduce memory requirements and uses an upsampling filter to generate reference image data from the stored subsampled image. The reference image data spatially corresponds to the image data generated by the video decoder before it is subsampled. The upsampling filter uses other filter coefficients based on the sub-sampling phase of the stored image and the half-pixel indicator of the motion vector used to locate the downsampled reference image data in the stored subsampled image.

Description

Up-Sampling Filters and Half-Pixel Generators for HDTV Down Conversion Systems

This patent application claims the benefit of US Provisional Application No. 60 / 040,517, filed March 12,1997.

The entire disclosure of US Provisional Application No. 60 / 040,517 is specifically incorporated herein by reference.

The present invention relates to a decoder for receiving, decoding, and converting frequency domain encoded signals, such as MPEG-2 encoded video signals, to standard output video signals, and more specifically to decoding the encoded high resolution video signal. It relates to upsampling and a half-pixel generator of a decoder that converts to a low resolution output video signal.

In the United States, the standard, the Advanced Television System Committee standard (ATSC), defines the digital encoding of high definition television (HDTV) signals. Some of these standards are essentially identical to the MPEG-2 standard proposed by the International Organization for Standardization (ISO) 's Moving Picture Experts Group (MPEG). The standard is described in the International Standard (IS) publication entitled "Information Technology-General Coding of Audio Related to Moving Pictures, Recommendation H.626", ISO / IEC 13818-2, IS, 11/94, which is available from ISO. And the teachings for the MPEG-2 digital video coding standard are incorporated herein by reference.

The MPEG-2 standard is actually several other standards. Several different profiles are defined in MPEG-2, each corresponding to a different level of complexity of the encoded image. For each profile, different levels are defined, each level corresponding to a different image resolution. One of the MPEG-2 standards known as Main Profile, Main Level is intended to code video signals that conform to existing television standards (ie NTSC and PAL). Another standard, known as Main Profile, High Level, is intended for coding high-quality television images. Images encoded according to the main profile, high level standard, may have 1,152 active lines per image frame and 1,920 pixels per line.

On the other hand, the main profile, main level standard defines a maximum picture size of 720 pixels per line and 567 lines per frame. At a frame rate of 30 frames per second, signals encoded according to the standard have a data rate of 720 * 567 * 30 or 12,247,200 pixels per second. In contrast, images encoded according to the main profile, high level standard, have a maximum data rate of 1,152 * 1,920 * 30 or 66,355,200 pixels per second. This data rate is greater than five times the data rate of video data encoded according to the main profile, main level standard. The standard for HDTV encoding in the United States is a subset of the standard and has a maximum frame rate of 1,080 lines per frame, 1,920 pixels per line, and 30 frames per second for the frame size. The maximum data rate for this standard is much greater than the maximum data rate for the main profile, main level standard.

The MPEG-2 standard defines a complex syntax that includes a mixture of data and control information. Some of this control information is used to enable signals with several different formats to be covered by this standard. These formats define images with different numbers of pixels (pixels) per line, different numbers of lines per frame or field, different numbers of frames or fields per second. Also, the basic syntax of the MPEG-2 main profile is five layers: sequence layer, group of pictures layer, picture layer, slice layer, macroblock. Defines a compressed MPEG-2 bit stream representing a sequence of images in a macroblock layer. Each of these layers is introduced with control information. Finally, other control information, also known as side information (e.g., frame type, macroblock pattern, image motion vectors, coefficient zig-zag patterns, dequantization information) may be coded bits. Intersperse through the stream.

Format converting the encoded high resolution main profile high level pictures into lower resolution main profile high level pictures, main profile main level pictures, or other lower resolution picture formats is a) using a single decoder to convert existing multiple videos. Providing use in formats, b) providing an interface between main profile high level signals and personal computer monitors or existing consumer television receivers, and c) reducing the cost of implementing HDTV. . For example, conversion can be done with lower picture resolutions that support expensive high quality monitors used with main profile high level encoded pictures, for example main profile main level encoded pictures such as NTSC or 525 progressive monitors. Allows replacement with cheap existing monitors with In one aspect, down conversion converts a high quality input picture to a lower resolution picture for display on a lower resolution monitor.

In order to receive digital images efficiently, the decoder must process video signal information quickly. To be optimally effective, decoding systems must be relatively inexpensive and have sufficient power to decode these digital signals in real time. Thus, a decoder that supports conversion to multiple low resolution formats should minimize processor memory.

The MPEG-2 main profile standard defines a sequence of pictures at five levels: sequence level, group level of pictures, picture level, slice level, macroblock level. Each of these levels may be considered a record in the data stream, and later-listed levels may occur as nested sub-levels of the levels listed first. The records for each level include a header section that contains the data used to decode its sub-records.

Each macroblock of an encoded HDTV signal contains six blocks, each block representing 64 respective coefficient values of a discrete cosine transform (DCT) representation of 64 pixels (pixels) in the HDTV image. Contains data.

In the encoding process, pixel data may require motion compensated differential coding prior to discrete cosine transform, and the blocks of transformed coefficients are also encoded by applying run-length and variable length encoding techniques. . A decoder that reconstructs the picture sequence from the data stream reverses the encoding process. Such decoders use entropy decoders (eg, variable length decoders), inverse discrete cosine transform processors, motion compensation processors, interpolation filters.

1 is a high level block diagram of a prior art video decoding system for processing MPEG-2 encoded pictures. The general method used to decode MPEG-2 encoded pictures without subsequent processing, down conversion or format conversion is described by the MPEG-2 standard. The video decoding system includes an entropy decoder (ED) 110, which may include a variable length decoder (VLD) 210 and a run length decoder 212. The system also includes an inverse quantizer 214 and an inverse discrete cosine transform (IDCT) processor 218. Controller 207 controls the various components of the decoding system in response to control information retrieved from the input bit stream by ED 110. For processing prediction images, the system may include a memory 199 having a reference frame memory 222, a summing network 230, a motion vector processor 221, and a half-pixel generator 228. It further includes a motion compensation processor 206a.

The ED 110 receives the encoded video image signal and matches the macroblocks of quantized frequency-domain (DCT) coefficient values and the previous decoded image corresponding to the macroblock of the predicted image currently being decoded. Reverse the encoding process to generate control information that includes a motion vector describing the relative displacement of the macroblock. The inverse quantizer 214 receives the quantized DCT transform coefficients and reconstructs the quantized DCT coefficients for a particular macroblock. The quantization matrix used for the particular block is received from the ED 110.

The IDCT processor 218 may reconstruct the reconstructed DCT coefficients for each block of 8x8 matrix values representing the luminance or chrominance components of the macroblock and the components of the predicted differential or differential luminance of the macroblock. For each block of 8 by 8 matrix values).

If the current macroblock is not predictively encoded, the output matrix values provided by IDCT processor 218 are the pixel values of the corresponding macroblock of the current video image. If a macroblock is interframe encoded, the corresponding macroblock of the previous video picture frame is stored in memory 199 for use by motion compensation processor 206. The motion compensation processor 206 receives the previously decoded macroblock from the memory 199 responsive to the motion vector and then adds the previous macroblock to the current IDCT macroblock (currently predictively encoded frame) of the network 230. Corresponding to the residual component) to generate a corresponding macroblock of pixels for the current video image, which is stored in the reference frame memory 222 of the memory 199.

1 is a high level block diagram of a prior art video decoder.

2 is a high level block diagram of a down conversion system used by an exemplary embodiment of the present invention.

3A is a macroblock diagram illustrating input and decimated output pixels for a 4: 2: 0 video signal using 3: 1 decimation.

3B is a pixel block diagram illustrating input and decimated output pixels for a 4: 2: 0 video signal using 2: 1 decimation.

4A is a pixel chart showing predicted pixels corresponding to subpixel locations for 3: 1 and 2: 1 exemplary embodiments of the present invention.

4B is a flow chart illustrating an upsampling process performed for each row of an input macroblock for an exemplary embodiment of the invention.

FIG. 5 is a pixel chart showing multiplication pairs for first and second output pixel values of an exemplary embodiment of a block mirror filter. FIG.

FIG. 6 is a block diagram illustrating an example implementation of a filter for down-conversion for a two-dimensional system that processes horizontal and vertical components implemented with cascaded one-dimensional IDCT. FIG.

The present invention is implemented in an upsampling filter for a digital video signal down conversion system. The down conversion system decodes the digitally encoded video signal representing the video image and decimates the decoded signal to produce a subsampled video signal. The subsampled signal is stored for use in decoding subsequently received encoded video signals that are encoded with differential pixel values in relation to the previously decoded video. When a subsequent image signal representing differential pixel values is received and decoded, segments of the stored image are viewed to produce decoded image pixel values that can be combined with the decoded differential pixel values to produce a decoded image signal. The filter according to the invention is searched and upsampled. The upsampling filter according to the invention comprises an upsampling filter having programmable coefficient values and a plurality of filter coefficient sets, each coefficient set corresponding to a separate subsampling phase. The filter also includes circuitry for receiving the motion vector from the encoded video signal, processing the received motion vector to locate the desired segment in the stored video, and determining the subsampling phase of the desired segment. In response to this determination, the circuit retrieves the desired segment from the location identified in the stored image data, programs the filter with the appropriate set of filter coefficients, and provides the retrieved image data to match the decoded differential image data. Filter the segments.

According to another aspect of the invention, the motion vectors of the encoded video signal define the position of the desired segment at a resolution of one half of the pixel position, the plurality of coefficient sets comprising a number 2N coefficient sets, wherein N is a decimation factor applied to the decoded image to produce a subsampled image.

According to another aspect of the invention, the motion vectors of the encoded video signal define the position of the desired segment at a resolution of one half of the pixel position, the plurality of coefficient sets comprising a number N coefficient sets, wherein N is a decimation factor applied to the decoded image to produce the subsampled image, and the filter is a desired filter filtered to produce output pixel values shifted by a half pixel position from the pixel values of the desired segment being filtered. It further includes a linear interpolator that averages adjacent pixels in the segment.

According to another aspect of the invention, the filter is programmable to decimate a decoded video image by one of a plurality of decimation factors, and comprises a plurality of groups of coefficient sets, each group having a plurality of decimations. Corresponds to one of the factors.

These and other features and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings.

Exemplary embodiments of the invention decode conventional HDTV signals encoded according to the MPEG-2 standard and in particular the Main Profile High Level (MP @ HL) and Main Profile Main Level (MP @ ML) MPEG-2 standards. Provide decoded signals as video signals having a lower resolution than received HDTV signals having a selected one of a number of formats.

Down-conversion decoder

2 is a high level block diagram of a down conversion system that includes one exemplary embodiment of the present invention using such a DCT filtering operation, which may be used by exemplary embodiments of the present invention. As shown in FIG. 2, the down conversion system includes a variable length decoder (VLD) 210, a run-length (R / L) decoder 212, an inverse quantizer 214, an inverse discrete cosine transform (IDCT) processor 218. ). The down conversion system also includes a down conversion filter 216 and a down sampling processor 232 for filtering the encoded pictures. The following describes an exemplary embodiment for MP @ HL encoded input, while the present invention may be practiced with any similarly encoded high resolution video bit stream.

The upsampling filter and half-pixel generator used by the down-conversion system according to the present invention includes an upsampling processor 226 and a half-pixel generator 228 and uses a reference frame memory 222. The down conversion system also includes a motion compensation processor 206 that includes a motion vector (MV) translator 220 and a motion block generator 224.

The system of FIG. 2 also includes a display conversion block 280 having a vertical programmable filter (VPF) 282 and a horizontal programmable filter (HZPF: 284). Display conversion block 280 converts the downsampled images into images for display on a particular display device having a lower resolution than the original image.

The down conversion filter 216 performs low-pass filtering of high resolution (eg, main profile high level DCT) coefficients in the frequency domain. The down sampling processor 232 is further configured by decimation of the filtered main profile high level picture to produce a set of pixel values that can be displayed on a monitor with a lower resolution than required to display the MP @ HL picture. Remove the spatial pixels. Exemplary reference frame memory 222 stores spatial pixel values corresponding to at least one previously decoded reference frame having a resolution corresponding to the down-sampled picture. For interframe encoding, the MV translator 220 scales the motion vectors for each block of the received picture that corresponds to a decrease in resolution, and the high resolution motion block generator 224 stores the reference frame memory 222. Receive the low resolution motion blocks provided by, upsample these motion blocks, and perform the half-pixel interpolation required to provide the motion blocks with pixel positions corresponding to the decoded and filtered differential pixel blocks.

Note that in the down conversion system of FIG. 2, downsampled images, rather than high quality images, are stored, resulting in a significant reduction in the memory required to store the reference images.

The operation of the exemplary down-conversion system of the present invention for intra-frame encoding is now described. The MP @ HL bit-stream is received and decoded by the VLD 210. In addition to the header information used by the HDTV system, the VLD 210 draws DCT coefficients for each block and macroblock and provides motion vector information. The DCT coefficients are run length decoded at R / L decoder 212 and inverse quantized by inverse quantizer 214.

Since the received video image represented by the DCT coefficients is a high resolution image, an exemplary embodiment of the present invention uses low pass filtering of the DCT coefficients of each block before decimation of the high resolution video image. Inverse quantizer 214 prior to providing the DCT coefficients to IDCT processor 218, DCT to DCT filter 216 to perform low pass filtering in the frequency domain by weighting the DCT coefficients to predetermined filter coefficient values. Provide coefficients. In one exemplary embodiment of the present invention, this filtering operation is performed on a block basis.

IDCT processor 218 provides spatial pixel sample values by performing an inverse discrete cosine transform of the filtered DCT coefficients. The down sampling processor 232 reduces the picture sample size by removing spatial pixel sample values according to a given decimation ratio, so storing a lower resolution picture stores a higher resolution MP @ HL picture. Use smaller frame memory compared to what is required to do.

The operation of an exemplary embodiment of the down-conversion system of the present invention for prediction frames of the encoding standard is now described. In this example, the currently received picture DCT coefficients represent the DCT coefficients of the residual components of the predicted picture macroblocks. In the described exemplary embodiment, the horizontal components of the motion vectors for the predicted frame are scaled because the low resolution reference pictures of the previous frames stored in memory do not have the same number of pixels as the high resolution predicted frame MP @ HL. .

Referring to FIG. 2, the motion vectors of the MP @ HL bit stream provided by the VLD 210 are provided to the MV translator 220. Each motion vector is scaled by the MV translator 220 to reference the appropriate prediction block of the reference frame of the previous picture stored in the reference frame memory 222. The size of the retrieved block (number of pixel values) is smaller than the block provided by IDCT processor 218; Thus, the retrieved block is upsampled to form a predictive block having the same number of pixels as the residual block provided by IDCT processor 218 before the blocks are combined by summing network 230.

The predictive block is upsampled by an upsampling processor 226 in response to a control signal from the MV translator 220 to generate a block corresponding to the original high resolution pixel block, and up-graded in the half-pixel generator 228. If indicated by the motion vector for the sampled prediction block, half-pixel values are generated to ensure proper spatial alignment of the prediction block. The upsampled and aligned prediction block is added to the currently filtered block in summing network 230, which is, for example, a reduced resolution residual component from the prediction block. All processing is performed based on macroblock units. After the motion compensation process is completed for the current high-resolution macroblock, the reconstructed macroblock is decimated according to the down sampling processor 232. This process does not reduce the resolution of the image but simply removes redundant pixels from the low resolution filtered image.

Down Sampling for Low Resolution Formats

Down sampling is accomplished by the down sampling processor 232 of FIG. 2 to reduce the number of pixels in the downconverted image. 3A shows input and decimated output pixels for a 4: 2: 0 signal format for 3: 1 decimation. 3B shows the input and decimated output pixels for 4: 2: 0 chrominance type 2: 1 decimation. Table 1 shows legend identification for the luminance and chrominance pixels of FIGS. 3A and 3B. Pixel positions before and after downconversion in FIGS. 3A and 3B are interlaced (3: 1 decimation) and progressive (2: 1 decimation) cases, respectively.

Table 1

For down sampling of an interlaced image, which can be converted from a 1920 * 1080 pixel image to a 640 * 1080 pixel horizontal compressed image, two of every three pixels are decimated on the horizontal axis. For the example 3: 1 decimation, there are three different macroblock types after the down conversion process. In Fig. 3A, the original macroblocks are defined as MB0, MB1, MB2. Luminance pixels down sampled in MB0 start at the first pixel of the original macroblock, while down sampled pixels start at the third and second pixels in MB1, MB2. Also, the number of down-sampled pixels in each macroblock is not the same. In MB0, there are six down-sampled pixels horizontally, while in MB1, MB2 there are five pixels. These three MB types are being repeated, so modulo 3 arithmetic must be done. Table 2 summarizes the downsampling pixels and the number of offsets for each input macroblock MB0, MB1, MB2.

Table 2

For downsampling of progressive format pictures, the luminance signal is subsampled horizontally for every second sample. For the chrominance signal, the down-sampled pixel has a spatial position that is one half below the pixel position of the original image.

Macroblock Prediction for Downconversion

For the exemplary down conversion process, since the reference frames of previous images are down sized in the horizontal direction, the received motion vectors pointing to these frames can also be translated according to the conversion ratio. The following describes motion translation for a luminance block in the horizontal direction. Those skilled in the art will be able to extend the following discussion to motion translation in the vertical direction, if desired. Define x and y as the current macroblock address of the original picture frame, Dx as the horizontal decimation factor, mvx as the half pixel horizontal motion vector of the original picture frame, and motion in the original picture frame If the address of the upper left pixel of the block is defined as XH of the half pixel unit, the following equation [1] is given.

[Equation 1]

XH = 2x + mv _x

Pixels with addresses corresponding to the motion block initiation in the down-sampled image and defined by x * and y * may be determined using equation [2].

[Equation 2]

The division of equation [2] is an integer division that truncates.

The exemplary filter 216 and down sampling processor 232 only reduce the horizontal components of the image, so that the vertical components of the motion vector are not affected. For chrominance data, the motion vector is one half of the luminance motion vector of the original image. Thus, the definition of translating the chrominance motion vector may also use two equations [1], [2].

Motion prediction is done by a two step process. First, pixel accuracy motion estimation of the original image frame may be achieved by upsampling of the down-sampled image frame in the upsampling processor 226 of FIGS. 2A and 2B, and then the half pixel generator 228 is the nearest neighbor. Half pixel interpolation is performed by averaging pixel values.

The reference image data is added to the output data provided by the IDCT processor 218. Since the output values of summing network 230 correspond to an image with multiple pixels that match the high resolution format, these values can be downsampled for display on a lower resolution display. Downsampling at downsampling processor 232 is substantially equivalent to subsampling of the image frame, but adjustments can be made based on the conversion ratio. For example, in the case of 3: 1 downsampling, the number of horizontally downsampled pixels is 6 or 5 for each input macroblock, and the first downsampled pixels are always the first pixel of the input macroblock. It is not.

After obtaining the correct motion prediction block from the down-sampled image, upsampling is used to obtain the corresponding prediction block in the high resolution image. Thus, subpixel accuracy in motion block prediction is desirable in down sampled pictures. For example, using 3: 1 decimation, it is desirable to have 1/3 (or 1/6) subpixel accuracy of motion prediction in downconverted pictures. In addition to the down-sampled motion block, a subpixel is determined that is the first pixel obtained by the motion vector. Subsequent subpixel positions are then determined using the modulo calculation described below. The subpixel positions are defined by x _s as given in equation [3].

[Equation 3]

Where "%" represents division by module.

For example, the range of x _s is 0,1,2 for 3: 1 upsampling and 0,1 for 2: 1 upsampling. FIG. 4A shows 17 prediction pixels corresponding to subpixel locations for 3: 1 and 2: 1 examples, and Table 3 shows the legend of FIG. 4A.

Table 3

As mentioned above, the upsampling filters can be upsampling polyphase filters, and Table 4 characterizes these upsampling polyphase interpolation filters.

Table 4

The following two tables, Tables 5 and 6, show the polyphase filter coefficients for the exemplary 3: 1 and 2: 1 upsampling polyphase filters.

Table 5 3: 1 Upsampling Filter

Table 6 2: 1 Upsampling Filter

In fixed-point display, the numbers in parentheses in Tables 5 and 6 are 9-bit complement representations of two bits, with corresponding double-precision numbers on the left. According to the subpixel position of the motion prediction block of the downsampled reference picture frame, one corresponding phase of the polyphase interpolation filter is used. Also, for the exemplary embodiment, additional pixels to the left and right are used to interpolate 17 horizontal pixels of the original image frame. For example, in the case of 3: 1 decimation, up to six horizontal downsampled pixels are generated for each input macroblock. However, when upsampling, nine horizontal pixels are used to generate corresponding motion prediction block values because the upsampling filter requires more left and right pixels outside the boundary for the filter to work. Since the exemplary embodiment uses half pixel motion estimation, 17 pixels are required to obtain 16 half pixels, which are average values of two nearest pixel samples. The half pixel interpolator performs an interpolation operation that provides half-pixel resolution to a block of pixels. Table 7A illustrates an example mapping between subpixel positions and polyphase filter elements and shows the number of left pixels required in addition to the pixels of the upsampled block for the upsampling process.

Table 7A

4B summarizes the upsampling process performed for each row of the input macroblock. First, in step 310, the motion vector for the block of the input image frame being processed is received. In step 312, the motion vector is translated to correspond to a downsampled reference frame of memory. In step 314, the scaled motion vector is used to calculate the coordinates of the reference image block stored in memory 130. In step 316, the subpixel point for the block is determined, and then initial polyphase filter values for upsampling are determined in step 318. The identified pixels for the reference block of the stored downsampled reference frame are then retrieved from memory 130 at step 320.

Prior to the first pass in filtering step 324, the registers of the filter may be initialized in step 322, which step 322 includes loading registers with initial 3 or 5 pixel values as an exemplary embodiment. . Then, after filtering step 324, the process at step 326 determines whether all pixels have been processed, an exemplary embodiment is 17 pixels. If all the pixels have been processed, the upsampled block is complete. In an exemplary embodiment, the 17x9 pixel block is returned for higher or lower motion blocks. If all the pixels have not been processed, the phase is updated at step 328 and checked if the phase is zero. If phase is zero, the registers are updated for the next set of pixel values. Updating the phase in step 328 updates the phase value to 0,1,2 for the filter loop period for the exemplary 3: 1 upsampling, and zero for the filter loop period for the 2: 1 upsampling. Update to 1 If the left-most pixel is outside the boundary of the video image, the first pixel value of the video image can be repeated.

For an exemplary embodiment, the upsampling filtering operation may be implemented in accordance with the following guidelines. First, several factors can be used. 1) The half-pixel motion prediction operation averages two full pixels, and the corresponding filter coefficients are also averaged to provide half-pixel filter coefficients. 2) A fixed number of filter coefficients, such as 5, which may be equivalent to the number of filter taps, may be used regardless of the particular down conversion. 3) Five parallel input ports can be provided in the upsampling block for each forward, backward, lower, and upper block, and five input pixels LWR (0) -LWR for each clock transition for each reference block. (4) is combined with the corresponding filter coefficients to provide one output pixel. 4) The sum of the filter coefficients h (0) -h (4) combined with each pixel LWR (0) -LWR (4) provides the output pixel of the sampling block.

Since the ordering is the opposite of the normal arrangement of filter coefficients, it may be desirable for the filter coefficients to be reversed and some coefficients to zero. Table 7B shows example coefficients for a 3: 1 upsampling filter, and Table 7C shows example coefficients for a 2: 1 upsampling filter.

Table 7B

Table 7C

In tables 7B and 7C, x ^* is the downsampled pixel position defined in equations [1] and [2] and the subpixel position x _s is redefined from equation [3] to equation [3 '].

[Equation 3 ']

x _s = (XH)% (2Dx)

For the chrominance values of the exemplary implementation, XH is scaled to 2 and equations [1], [2], [3 '] apply. In one embodiment, phase and half pixel information (coded in two and one bits respectively) is used by motion compensation processor 220 and half-pixel generator 228 of FIG. 2B. For example, the reference block pixels are provided first with U pixels, then with V pixels, and finally with Y pixels. U and V pixels are clocked for 40 cycles, and Y pixels are clocked for 144 cycles. The reference blocks may be provided for 3: 1 decimation by providing the first five pixels, repeating twice, shifting the data one by one, and repeating until one row is over. The same method can be used for 2: 1 decimation except that it is repeated once rather than twice. The input pixels are repeated because decimation follows half-pixel generation with addition and residual values of the output from motion compensation. Thus, for 3: 1 decimation, two of the three pixels are deleted, and the dummy pixels for these pixel values are not important.

DCT Domain Filtering Employing Weighting of DCT Coefficients

An exemplary embodiment of the present invention is used with the DCT filter 216 of FIG. 2 which processes DCT coefficients in the frequency domain and replaces the low-pass filter operation in the spatial domain. As contemplated by the MPEG or JPEG standards, there are several advantages to DCT domain filtering instead of spatial domain filtering for DCT coded pictures. Most notably, the DCT domain filter is computationally more efficient and requires less hardware than the spatial domain filter applied to the spatial pixel sample values. For example, a spatial filter with N taps may use N additional multiplications and additions for each spatial pixel sample value. This is compared to only one additional multiplication in the DCT domain filter.

The simplest DCT domain filter of the prior art is the truncation of high frequency DCT coefficients. However, the removal of high frequency DCT coefficients does not result in a flat filter and has disadvantages such as "ringing" near the edges of the decoded picture. The DCT region low-pass filter of the exemplary embodiment of the present invention is obtained from a block mirror filter in the spatial domain. Filter coefficient values for the block mirror filter are optimized, for example, by numerical analysis in the spatial domain, and these values are then converted into coefficients of the DCT domain filter.

Although the exemplary embodiment only shows DCT region filtering in the horizontal direction, DCT region filtering can be done by combining the horizontal or vertical direction or the horizontal and vertical filters.

Derivation of the DCT Domain Filter Coefficients

One exemplary DCT filter for use with the present invention is derived from two restraints. First, the filter processes the image data on a block-by-block basis for each block of the image without using information from previous blocks of the image. Second, the filter is a block of block boundaries that occur when the filter processes boundary pixel values. Decreases visibility.

According to the first constraint, in DCT-based compression of an MPEG picture sequence, for example, N × N DCT coefficients give rise to N × N spatial pixel values. Thus, an exemplary embodiment of the present invention implements a DCT region filter that merely processes the current block of the received picture.

According to a second constraint, if the filter is simply applied to a block of spatial frequency coefficients, there is a variation in the filtering operation at the block boundary caused by an insufficient number of spatial pixel values beyond the boundary to fill the rest of the filter. That is, because the N-tap filter only has values for N / 2 taps, and the remaining values cross the block boundary, the coefficient values at the edge of the block cannot be properly filtered. There are several ways to supply missing pixel values. 1) repeating a predetermined constant pixel value across a boundary 2) repeating a pixel value equal to a boundary pixel value 3) reflecting pixel values of the block to simulate previous and subsequent blocks of pixel values adjacent to the processed block. A method of mirroring pixel values that is repeated without prior information about the contents of the preceding or subsequent blocks is considered to be a good method. Thus, one embodiment of the present invention uses this mirroring method for the filter and is referred to as a "block mirror filter".

The following describes an example embodiment of implementing a horizontal block mirror filter that low-pass filters the eight input spatial pixel sample values of a block. If the size of the input block is an 8x8 block matrix of pixel sample values, horizontal filtering may be done by applying a block mirror filter to each row of 8 pixel sample values. It will be apparent to those skilled in the art that the filtering process may be implemented by applying the filter coefficients to the block matrix in the column direction or that multidimensional filtering may be accomplished by filtering the rows and then filtering the columns of the block matrix.

5 shows an example between filter taps for an example mirror filter for 8 input pixels using the 15 tap spatial filter represented by input pixel values x ₀ to x ₇ (group X ₀ ) and tap values h ₀ to h ₁₄ . Shows the enemy response. The input pixels are reflected on the left side of group X0 shown as group X1 and on the right side of group X0 shown as group X2. The output pixel value of the filter is the sum of 15 multiplications of the filter tap coefficient values with corresponding pixel sample values. 5 illustrates multiplication pairs for first and second output pixel values.

The following shows that the block mirror filter in the spatial domain is equivalent to the DCT domain filter. Mirror filtering relates to circular convolution with 2N points (N = 8).

Define the vector x 'as shown in equation [4].

[Equation 4]

x '(n) = x (n) + x (2N-1-n), 0 = n <= 2N-1

In the case of N = 8

x '= (x0, x1, x2, x3, x4, x5, x6, x7, x7, x6, x5, x4, x3, x2, x1, x0)

If we rearrange filter tap values h ₀ to h ₁₄ and define the rearranged values by h ',

h '= (h7, h8, h9, h10, h11, h12, h13, h14,0, h0, h1, h2, h3, h4, h5, h6)

Thus, the mirror filtered output y (n) is a cyclic convolution of x '(n) and h' (n) given by equation [5].

[Equation 5]

y (n) = x '(n) ⓧh' (n)

This is equivalent to equation [6].

[Equation 6]

Where x '[n-k] is a cyclic modulo of x' (n),

x '[n] = x' (n), n> = 0

x '[n] = x' (n + 2N), n <0.

The cyclic convolution in the spatial domain shown in equation [5] corresponds to a scalar product in the Discrete Fourier Transform (DFT) domain. If Y (k) is defined as the DFT of y (n), equation [5] becomes equation [7] in the DFT domain.

[Equation 7]

Y (k) = X '(k) H' (k)

Where X '(k) and H' (k) are the DFT of x '(n) and h' (n), respectively.

Equations [4] to [7] are valid for filters with fewer taps than 2N. The filter is also limited to a symmetric filter with odd taps, with H '(k) being a real number. Thus, the DFT of X '(k), i. Can be. Since the N-point DCT of x (n) is obtained by the 2N-point DFT of x '(n), which is a joint sequence consisting of x (n) and its mirror x (2N-1-n), X' ( The values of k) are very closely related to the DCT coefficients of the original N-point x (n).

The following describes the derivation of the DFT coefficients of the spatial filter, H '(k), by assuming a symmetric filter with odd taps, 2N-1, where h (n) = h (2N-2-n) And h '(n) = h' (2N-n) and h '(N) = 0 equally. H '(k) is defined in equation [8].

[Equation 8]

Where W _2N ^kn = exp {-2πkn / (2N)} and H '(k) = H' (2N-k).

The inventors determined that the 2N-point DFT of x '(n), X' (k), can be represented by its DCT coefficients as shown in equation [9].

[Equation 9]

On the other hand, the DCT coefficient of x (n), C (k), is given by equation [10].

[Equation 10]

Is

And in other cases C (k) = 0.

The DFT coefficients of x '(n), the values of X' (k) can be represented by the DCT coefficients of x '(n), C (k) by the matrix of equation [11].

[Equation 11]

The original spatial pixel sample values, x (n), can also be obtained by IDCT (inverse discrete cosine transform) shown by equation [12].

[Equation 12]

Here, α (k) = 1/2 with respect to k = 0 and 1 otherwise.

The values of y (n) for 0 <= n <= N-1 are obtained by IDFT of X '(k) H' (k) given in [13].

[Equation 13]

The values y (n) of equation [13] are the spatial values of IDCT of C (k) H '(k). Thus, spatial filtering can be replaced by performing a DCT weighting of the input frequency-domain coefficients representing the image block with H '(k) and IDCT of the weighted values to reconstruct the filtered pixel values in the spatial domain.

One embodiment of exemplary block mirror filtering of the present invention is obtained by the following steps. 1) One-dimensional low-pass symmetric filter is selected with odd number of taps smaller than 2N taps. 2) Filter coefficients are increased to 2N values by adding zero. 3) The filter coefficients are rearranged such that the original intermediate coefficients go to the zeroth position by the left cyclic shift. 4) DFT coefficients of the rearranged filter coefficients are determined. 5) DCT coefficients are multiplied by the real DFT coefficients of the filter. 6) An Inverse Discrete Cosine Transform (IDCT) of the filtered DCT coefficients is performed to provide a block of low-pass-filtered pixels ready for decimation.

The cutoff frequency of the lowpass filter is determined by the decimation ratio. For one exemplary embodiment, the cutoff frequency is π / 3 for 3: 1 decimation and π / 2 for 2: 1 decimation, where π corresponds to one half of the sampling frequency.

The DCT domain filter of MPEG and JPEG decoders allows the memory requirements to be reduced because the IDCT processing of the inverse quantizer and blocks is already present in prior art decoders, and only an additional scalar product of the DCT coefficients by the DCT domain filter Required. Thus, no separate DCT domain filter block multiplication is physically required in a particular implementation, and another embodiment of the present invention simply combines IDCT processing coefficients with DCT domain filter coefficients and applies the combined coefficients to an IDCT operation. .

For the exemplary down conversion system of the present invention, horizontal filtering and decimation of DCT coefficients have been considered, and the following relates to two exemplary implementations.

1.Convert 1920H vs 1080V interlaced to 640 vs 1080 interlaced

(Horizontal 3: 1 decimation)

2. Convert 1280H to 720V progressive to 640 to 720 progressive

(Horizontal 2: 1 decimation)

Table 8 shows the DCT block mirror filter (weighted) coefficients, and the numbers in parentheses in Table 8 are 10 bit two's complement representations. " ^* " In Table 8 represents the over-boundary value for the 10-bit two's complement indication, because the value is greater than one. However, as is known to those skilled in the art, the product of the column coefficients of a block and the values marked with * can be easily implemented by adding the coefficient value to the coefficient multiplied by the remaining value of the filter value (rest).

Table 8

These horizontal DCT filter coefficients weight each column of an 8x8 block of DCT coefficients of the encoded video image. For example, the DCT coefficients in column 0 are weighted by H [0], the DCT coefficients in column 1 are weighted by H [1], and so on.

The above description shows a horizontal filter implementation using one-dimensional DCTs. As is known in the digital signal processing technique, such processing can be extended to two-dimensional systems. Equation [12] illustrates the IDCT for the one-dimensional case, so equation [12 '] gives the more general two-dimensional IDCT.

[Equation 12 ']

Where C (u), C (v) is u, v = 0 And otherwise 1.

Where f (x, y) is the spatial domain representation, x and y are the spatial coordinates of the sample domain, and u, v are the coordinates of the transform domain. Since the coefficients C (u), C (v) are known as the values of the cosine term, only transform domain coefficients need to be provided for the processing algorithms.

In a two-dimensional system, the input sequence is now represented by a matrix of values, each representing its coordinates in the transform domain, the matrix having periodic sequences in a column sequence with period M and a row sequence with period N as It can be seen that N and M are integers. The two-dimensional DCT may be implemented with a first one-dimensional DCT performed on columns of the input sequence and a second one-dimensional DCT performed on rows of the DCT processed input sequence. In addition, as known in the art, two-dimensional IDCT may be implemented as a single process.

6 shows an example implementation of a filter for down-conversion for a two-dimensional system that processes horizontal and vertical components implemented with serially connected one-dimensional IDCTs. As shown in FIG. 6, the DCT filter mask 216 and IDCT 218 of FIG. 2 are implemented for a vertical processor 510 and vertical components that include a vertical DCT filter 530 and a vertical IDCT 540. It may be implemented by a horizontal processor 520 that includes the same horizontal DCT filter and horizontal IDCT. Since filtering and IDCT processes are linear, the order in which they are implemented may be rearranged (e.g., horizontal and vertical DCT filtering first, horizontal and vertical IDCT second, or vice versa. ) First, horizontal processor (510) second)

In the particular implementation shown in FIG. 6, a vertical processor 510 is followed by a block transpose operator (550), which switches the rows and columns of the block of vertically processed values provided by the vertical processor. This operation may be used to increase the efficiency of the calculation by preparing the block for processing by the horizontal processor 520.

An encoded video block, e.g., an 8x8 block of matrix values, is received by the vertical DCT filter 530, which weights each row entry of the block by DCT filter values corresponding to the desired vertical decimation. Next, vertical IDCT 540 performs inverse DCT on the vertical components of the block. As mentioned above, since both processes simply perform matrix multiplication and addition, the DCT LPF coefficients may be combined with the vertical DCT coefficients for matrix multiplication and addition operations. Vertical processor 510 then supplies the vertically processed blocks to pre-operator 550, which provides the horizontally processed processor with a transposed block of vertically processed values. If the IDCT operation is only performed by row or column, then pre-operator 550 is not necessary. The horizontal processor 520 performs weighting by the DCT filter values of each column entry of the block corresponding to the desired horizontal filtering, and performs inverse DCT on the horizontal components of the block.

As described with reference to equation [12 '], only the coefficients in the transform domain are provided to the processing algorithms, and only operations that allow mathematical operations on these coefficients are linear. As is already apparent from equation [12 '], the operations on IDCT form a sum of products. Thus, a hardware implementation stores known coefficients in memory, such as a ROM (not shown), coefficients selected from a matrix of input transform coordinates, as well as a group of multiplication and addition circuits (not shown) that receive these coefficients from the ROM. It needs to be. In more advanced systems, the ROM-accumulator method can be used if the order of the mathematical operations is modified according to distributed arithmetic that translates from the sum implementation of the products to the bit-serial implementation. Such techniques are described, for example, in Stanley A. White, Application of Distributed Arithmetic to Digital Signal Processing, A Tutorial Review, IEEE ASSP Magazine, July, 1989, which calculates the total gate count in the implementation of the sum of the products. Symmetry in the field.

In alternative embodiments of the invention, the DCT filter operation may be combined with an inverse DCT (IDCT) operation. For such an embodiment, since the filtering and inverse transform operations are linear, the filter coefficients may be combined with the coefficients of the IDCT forming the modified IDCT. As is known in the art, modified IDCT and thus combined IDCT and DCT downconversion filtering can be performed through a hardware implementation similar to that of a simple IDCT operation.

While illustrative embodiments of the invention have been shown and described herein, it will be appreciated that such embodiments are provided by way of example only. Various modifications, variations, and substitutions will occur to those skilled in the art without departing from the spirit of the invention. Accordingly, the appended claims are intended to cover all such changes as fall within the scope of the invention.

Claims (9)

A reference picture for decoding a digitally encoded video signal representing a video picture and for decoding a subsequently received encoded picture signal that is encoded as differential picture element values for the previously decoded picture. Upsampling filter circuit for a digital video signal down conversion system that decimates the decoded signal to produce a subsampled video signal stored for use as data. circuitry, wherein the upsampling filter circuit comprises: A programmable filter having a plurality of programmable coefficient values, A coefficient memory for holding a plurality of filter coefficient sets, each coefficient set corresponding to a separate subsampling phase; Receiving a motion vector from the encoded video signal, processing the received motion vector to locate a desired segment in the stored video, and searching for the desired segment from the stored subsampled video signal With a motion vector translator, Control circuitry for determining the subsampling phase of the desired segment from the positional information provided by the motion vector translator and for programming the programmable filter into the set of coefficients corresponding to the determined subsampling phase. , The programmable filter filters the retrieved desired segment to provide reference image data corresponding to the decoded differential image data of an image pixel location.  The method of claim 1, The motion vectors of the encoded video signal define the position of the desired segment at a resolution of one half the pixel position, The plurality of coefficient sets comprises a number 2N coefficient sets, where N is a decimation factor applied to the decoded image to produce the subsampled image, and each coefficient set is associated with a subsampling phase. Corresponds to a distinct combination of half pixel positions, The control circuitry determines the subsampling phase of the desired segment from the position information provided by the motion vector translator, determines the half pixel position from the motion vector, and maps the programmable filter to the determined sampling phase. An upsampling filter circuit for programming with said set of coefficients corresponding to half pixel positions. The method of claim 1, The motion vectors of the encoded video signal define the position of the desired segment at a resolution of one half the pixel position, The filter is a linear interpolation that selectively averages adjacent pixels in the filtered desired segment to produce output pixel values shifted by a half pixel position from the pixel values of the filtered desired segment in response to the motion vector. The upsampling filter circuit further comprising a linear interpolator. The method of claim 1, The video signal down conversion system is configured to decimate the decoded video signal by one of a plurality of decimation factors, The filter is programmable to decimate the decoded video image by one of a plurality of decimation factors, The coefficient memory maintains a plurality of groups of coefficient sets, each group corresponding to one of the plurality of decimation factors, The control circuit is responsive to the decimation factor and the positional information provided by the motion vector translator to program the programmable filter into the set of coefficients corresponding to the decimation factor and the determined sampling phase. Circuit. The method of claim 4, wherein The decimated video signal is decimated only in the horizontal direction, and the programmable filter is a one-dimensional finite impulse response (FIR) filter. A reference picture for decoding a digitally encoded video signal representing a video picture and for decoding a subsequently received encoded picture signal that is encoded as differential picture element values for the previously decoded picture. An upsampling filter circuitry for a digital video signal downconversion system that decimates the decoded signal to produce a subsampled video signal stored for use as data, wherein the upsampling filter circuitry. Filter circuit, A programmable filter having a plurality of programmable coefficient values, Receives a motion vector from the encoded video signal that defines a position of a desired segment at a resolution of half the pixel position, and the received motion to locate the desired segment in the stored image. A motion vector translator for processing a vector and retrieving the desired segment from the stored subsampled video signal; A coefficient memory for holding a plurality of filter coefficient sets, each coefficient set corresponding to a distinct combination of subsampling phase and half pixel position; Control circuitry for determining the subsampling phase of the desired segment from the positional information provided by the motion vector translator and for programming the programmable filter into the set of coefficients corresponding to the determined subsampling phase. , The programmable filter filters the retrieved desired segment to provide reference image data corresponding to the decoded differential image data of an image pixel location. The method of claim 5, The video signal down conversion system is configured to decimate the decoded video signal by one of a plurality of decimation factors, The filter is programmable to upsample the decoded video image by one of a plurality of upsampling factors, each upsampling factor corresponding to one of the decimation factors, The coefficient memory maintains a plurality of groups of coefficient sets, each group corresponding to one of the plurality of decimation factors, The control circuit is responsive to the position information and the decimation factor provided by the motion vector translator to program the programmable filter to the set of coefficients corresponding to the decimation factor and the determined sampling phase. Circuit. A subsampled digital image stored for use in a digital video signal down conversion system that decodes a digitally encoded video signal representing a video image and decimates the decoded signal to produce a stored subsampled image. A method of upsampling data representing a method, the subsampled image being encoded subsequently received as encoded as differential pixel (pixel) values for a previously decoded image before the decoded image is subsampled. Stored for use as reference video data in decoding the video signal, the upsampling method comprising: Receiving a motion vector from the encoded video signal; Processing the received motion vector to locate a desired segment in the stored image; Retrieving the desired segment from the stored subsampled video signal; Determining a subsampling phase of the desired segment from the location of the desired segment; Selecting a set of coefficients used to filter the retrieved segment from a plurality of coefficient sets in response to the determined subsampling phase of the desired segment; Filtering the retrieved desired segment using the selected set of coefficients to produce reference image data corresponding to the decoded differential image data of an image pixel position. The method of claim 8, The motion vector defines the position of the desired image segment at half pixel resolution, The selecting step selects a pair of coefficient sets representing the subsampling phase shifted by a half pixel position with the subsampling phase in response to the subsampling phase, The method, Identifying a coefficient set of the pair of coefficient sets that should be used by the filtering step to generate the reference image data in response to the motion vector.