KR20070111467A - Scratch pad for storing intermediate loop filter data - Google Patents

Abstract

A video processing apparatus and methodology are implemented as a combination of a processor and a video decoding hardware block to decode video data by providing the video decoding block with an in-loop filter and a scratch pad memory, so that the in-loop filter may efficiently perform piecewise processing of overlap smoothing and in-loop deblocking in a macroblock-based fashion which is a much more efficient algorithm than th frame-based method.

Description

Scratch PAD for storing intermediate loop filter data {SCRATCH PAD FOR STORING INTERMEDIATE LOOP FILTER DATA}

The present invention relates to video processing techniques. In one aspect, the present invention is directed to decompression of digital video information.

Since video information requires a large amount of storage space, video information is generally compressed. Thus, for example, in order to display compressed video information stored in a CD-ROM or DVD, the decompressed video information can be provided only when the compressed video information is decompressed. The decompressed video information is then provided to the display device as a bit stream. The decompressed bit stream of video information is typically stored as a bit map in memory locations corresponding to pixel locations on the display. The video information needed to provide a single screen of information on the display is referred to as a frame. In order to provide motion video by displaying a series of frames, the goal of many video systems is to quickly and efficiently decode the compressed video information.

Standardizing various aspects of recording media, devices, and data processing such as, for example, video compression, is highly desirable for the continued development of these technologies and their applications.

Many compression (decompression) standards have been developed or are under development, which compress and decompress video information, such as the Motion Picture Expert Group (MPEG) standards for video encoding and decoding (e.g. MPEG- 1, MPEG-2, MPEG-3, MPEG-4, MPEG-7, MPEG-21, or the Window Media Video compression standard (for example, WMV9). Each of the MPEG and WMV standards is referred to herein as a whole, as if disclosed herein.

In general, video compression techniques operate to compress video information by reducing both spatial and temporal redundancy appearing in video frames, intraframe compression and interframe compression. ). Intraframe compression techniques use only the information embedded in a frame to compress the frame, which is called an I-frame. Interframe compression techniques compress frames associated with preceding and / or subsequent frames, which are called predicted frames, P-frames, or B frames. Intraframe and interframe compression techniques typically use spatial or block-based encoding, whereby one video frame is divided into blocks for encoding (also a block conversion process). It may also be called). For example, one I-frame is divided into 8x8 blocks. The blocks may be coded using a discrete cosine transformation (DCT) coding scheme that encodes a coefficient to the amplitude of a particular cosine-based function, or some other transforms (eg, integers). Coded using Transform. The transformed coefficients are then quantized, producing a run or subsequence of coefficients with nonzero amplitude levels and coefficients with an amplitude level of zero. The quantized coefficients are then run level encoded (or run length encoded) to compress long runs of zero coefficients.

These results are then entropy coded in a variable length coder (VLC) using a statistical coding technique that assigns codewords to values to be encoded, or other entropy coding techniques [eg, context]. Entropy coded using Context-based Adaptive Binary Arithmetic Coding (CABCA), Context Adaptive Variable Length Coding (CAVLC) and the like. Values with high frequency of occurrence are assigned to short codewords and values with low frequency are assigned to long codewords. On average, more frequent short codewords prevail, resulting in a shorter code string than the original data. Thus, spatial or block-based encoding techniques compress digital information related to one frame. To compress digital information related to a series of frames, video compression techniques use P-frames and / or B-frames, which take advantage of the fact that there is a temporal correlation between successive frames. To do this. Although different implementations may use different block configurations, interframe compression techniques will identify differences between different frames and then use DCT, quantization, run length, and entropy encoding techniques to account for this difference information. We will encode spatially. For example, a P-frame is divided into 16 × 16 macroblocks (eg, four 8 × 8 luminance blocks and two 8 × 8 chrominance blocks) and the macro block Are compressed. Regardless of whether interframe or intraframe compression techniques are used, the use of spatial or block-based encoding techniques to encode video data means that the compressed video data has been variable length encoded, as described above. Compressed differently using block-based compression techniques.

At the receiver side or the playback device side, the above compression steps are reversed to decode the video data that has been processed into block transforms. 1 shows a typical system 30 for decompressing video information, which comprises an input stream decoder 35, a motion decoder 38, a summer 39, a frame. And a buffer 40 and a display 41. Referring to the input stream decoding unit 35, the stream of compressed video information is received by the input buffer 31, the VLC decoder 32 performs variable length decoding, and the dequantizer 33 performs zigzag and quantization. Performs upside down, inverts DCT conversion in inverse DCT unit 34 and provides blocks of statically decompressed video information to summer 39. In the motion decoding unit 38, a copy of the previous picture data (stored in the previous picture storage buffer 36) and motion information from the VLC decoder 32 are added to the motion compensation unit 37. ) Receives and provides motion-compensated pixels to summer 39. Summer 39 receives statically decompressed video information and motion compensated pixels, and provides decompressed pixels to frame buffer 40, which then displays the information 41 To provide.

In a typical video encoder and decoder design, blocking artifacts (significant discontinuities between blocks) result from block-based transform, motion compensation, quantization, and / or other lossy processing steps. Can be introduced into. Prior attempts to reduce blocking artifacts have used overlap smoothing or deblocking filtering (prior to in-loop or processing), which processes frames by smoothing the boundaries between blocks. For example, in the case of the WMV9 standard, it has been described above that overlap smoothing and in loop deblocking are processed on the whole picture to reduce blocking artifacts. In the case of WMV9 decoding, overlap smoothing is performed only on 8x8 block boundaries, so smoothing starts in the vertical direction for the entire frame, and then smoothing is performed in the horizontal direction for the entire frame. Next, in-loop deblocking, when activated, is performed in the following order. (Iii) horizontal boundaries of all 8x8 blocks in the frame, starting from the topmost line, are filtered; (Ii) horizontal boundaries of all 8x4 subblocks in the frame are filtered, starting from the topmost line; (Iii) the vertical boundaries of all 8x8 blocks are filtered, starting from the leftmost line; (Iii) The vertical boundaries of all 4x8 subblocks, starting from the leftmost line, are filtered out. Conventional methods used two passes for the entire frame, where the first pass is to perform overlap smoothing and the second step is for in-loop deblocking. There may be other requirements that apply when deciding whether to perform individual steps (eg, related to parameter PQUANT and block types), but the purpose of these processes is 16 × 16 macroblocks, 8 × 8 blocks. , Or edges of 4x4 subblocks, thus eliminating artifacts (such as blockiness) by two-dimensional transform and quantization.

For processor-based approaches that handle video decompression, adding smoothing or deblocking functionality is a very computational filtering process. This kind of processing can be performed in software when there is a very large memory buffer that can hold a frame (e.g., a VGA size of 640x480 pixels corresponds to 307 kBytes). On the other hand, for hardware-based approaches for decoding, smoothing and deblocking were not performed at the same time, and deblocking was performed for the entire frame, which requires very large local memory, It imposes significant bus bandwidth demands and sacrifices memory access time. As a result, there is a considerable need to reduce processing requirements associated with decompression methods and in particular to improve decompression operations, including overlap smoothing and / or deblocking filter operations.

After reviewing the remainder of the present application with reference to the accompanying drawings and the following detailed description, further disadvantages and limitations of the prior system will become apparent to those skilled in the art.

By using a combination of software and hardware in video decompression, a flexible decompression system is provided that can quickly and effectively handle a variety of different video compression schemes. The flexible decompression system includes a processor that performs front end decompression steps and a video accelerator that performs back end decompression steps. In order to reduce memory bandwidth requirements in video accelerators that perform overlap smoothing and in-loop deblocking filter operations on video frame data, the in-loop filter is connected to a scratch pad memory or a storage device, which is a macroblock. Facilitate piecewise processing of overlap smoothing and in-loop deblocking in a -based manner. Using a scratch pad that performs the fractional processing of the filtering operation in a macro block-based manner is more efficient than the frame-based method. Since the size of the scratch pad memory is related to the width of the frame, the amount of on-chip memory is reduced. For example, the size of the scratch pad memory may be large enough to hold only one row of partially filtered blocks from one frame of video data.

In accordance with one or more embodiments of the present invention, a video processing system, apparatus, and method are provided whereby a processor and video decode circuit decode video data processed into block transforms into a plurality of macro blocks. Regarding the decode operation, the in-loop filter and scratch pad memory provided on the at least one integrated circuit may be further configured to include one or more completed blocks and one or more partially filtered by smoothing and deblocking selected pixel data in the first macro block. Used to perform the fractional processing of generating blocks, the at least one partially filtered blocks contain control data and pixel data (stored in scratch pad memory). As a result, any block adjacent to the previously processed macro block may be completely filtered for overlap smoothing and deblocking and then output as a completed block during the first filtering operation. On the other hand, a block adjacent to a subsequently processed macro block may be partially filtered for overlap smoothing and deblocking and then stored as a partially filtered block in the scratch pad memory. The scratch pad memory is used by an in-loop filter to provide a partially filtered block that is used to smooth and deblock pixel data in a first macro block, wherein the fetched partially filtered block is Created while processing a macro block. Continuously smoothing and deblocking in a pipelined manner over multiple macroblocks by successively processing each row of macroblocks in a video frame to overlap and deblock one macroblock at a time. An in-loop filter and scratch pad memory may be used. Objects, advantages and other novel features of the present invention will become apparent to those skilled in the art upon reading the following detailed description in conjunction with the appended claims and drawings.

1 is a diagram illustrating a system for decompressing video information.

2 illustrates an exemplary video decompression system in accordance with the present invention.

FIG. 3 is a simplified illustration of an in-loop filtering process using scratch pad memory to effectively handle overlap smoothing and in-loop deblocking in hardware according to an embodiment selected in the present invention.

4 illustrates an example technique for reducing blockiness of a decoded frame using smoothing and deblocking filters in a video encoder or decoder.

5A-5K illustratively illustrate how fractional processing may be used to implement a smoothing and deblocking procedure for luma blocks.

6A-6F illustrate how fractional processing may be used to implement a smoothing and deblocking procedure for chroma blocks.

Although exemplary embodiments of the invention are described below, it will be understood that the invention may be practiced without the specific details and for purposes of clarity, not all features of an actual implementation have been described herein. In developing any of these practical implementations, it is understood that many implementation specific issues must be determined in order to achieve the specific objectives of the developer (e.g. meeting system and business limitations). Should be. Furthermore, such development efforts may be complex and time consuming, but it should nevertheless be understood that such development efforts may be routine to those skilled in the art to be assisted by the disclosure herein. For example, although selected aspects are shown in block diagram form, they are not so detailed, in order to avoid obscuring the invention. These representations and descriptions are used by those skilled in the art to convey or explain the substance of their work to others skilled in the art. The invention will now be described with reference to the accompanying drawings.

2, an exemplary video decompression system 100 in accordance with the present invention is shown. As shown, the video decompression system 100 may be, for example, a desktop or laptop computer, a wireless or mobile device, a personal digital assistant (PDA), a mobile phone or cellular phone, and any other video playback device, including video imaging features. It may be implemented in any video playback device, such as.

As shown in FIG. 2, video decompression system 100 includes a bus 95 connected to one or more processors or processing units 50 and video (or media) acceleration hardware units 101. Implemented as a unit or an application processing unit. The video decompression system 100 also includes a main memory system that includes a large amount of DDR SDRAMs 62 and 64 accessed through the DDR controller 60. Additionally or alternatively, one or more memories (eg, IDE 72, flash memory unit 74, ROM 76, etc.) are accessed via static memory controller 70. All or one of the DDR SDRAMs or other memories may be integrated into the video decompression system 100 or may be implemented external to the video decompression system 100. Of course, other peripheral devices and display devices 82, 84, 86, 92 may be accessed through respective controllers 80, 90. For clarity and ease of understanding, not all components that make up the video decompression system 100 have been described in detail. Such details are well known to those skilled in the art and may vary depending on the particular computer vendor and type of microprocessor. Furthermore, video decompression system 100 may include further buses, devices, and / or subsystems, depending on the desired implementation. For example, the video decompression system 100 may include a cache, modem, parallel or serial interface, SCSI interface, network interface card, and the like. In the illustrated embodiment, the CPU 50 executes software stored in the flash memory 74 and / or the SDRAMs 62 and 64.

In the video decompression system 100 shown in FIG. 2, the CPU 50 performs an initial variable length decoding function as shown in the VLD block 52, while the media acceleration hardware unit 101 is decoded. For data, inverse quantization 104, inverse transform 106, motion compensation 108, in-loop filtering 110, color space conversion 112, scaling 114 And filtering 116. The resulting decoded data may be temporarily stored in output buffer 118 and / or frame buffer (not shown) before being displayed in display 92. By dividing the decode processing function between the processor 50 and the media accelerator hardware 101, the front end decoding steps (e.g., variable length decoding) can be performed using various different compression schemes (e.g. MPEG-1, MPEG-2, MPEG-3, MPEG-4, MPEG-7, MPEG-21, WMV9, etc.) may be implemented in software. The decode data generated by the front end decoding steps is provided to the media acceleration hardware 101, which further decodes the decode data to obtain pixel values on a macroblock basis until the frame is complete ( macroblock by macroblock basis) Provided to the output buffer or frame buffer.

In operation, video decompression system 100 receives a compressed video signal from a video signal source, such as, for example, a CD ROM, DVD, or other storage device. The compressed video signal is provided to the processor 50 as a stream of compressed video information, which processor provides a variable length coded portion of the compressed signal to provide a variable length decoded data (VLD data) signal. Run the command to decode. Once software assist is applied to perform variable length coding, the VLD data (which includes headers, matrix weights, motion vectors, transformed residue coefficients, and Differential motion vectors are transmitted to the media acceleration hardware unit 101, either directly or as described in more detail in US Appl. No. 11/042365 (Title: Lightweight Compression Of Input Data). Delivered using data compression techniques. In the media accelerator hardware unit 101, once VLD data is received, the data is provided to an inverse zig-zag and quantizer circuit 104, wherein the inverse zig-zag and quantizer circuit is a zig-zag decoded signal. Decode the VLD data signal to provide. Inverse zig-zag and quantization allows the compressed video signal to be compressed in a zig-zag run length code scheme, while the zig-zag decoded signal is an inverse DCT circuit (IDCT) 106 as sequential blocks of information. To compensate). As a result, this zigzag decoded signal provides the blocks, which have the order required to perform raster scanning on the display 92. This zigzag decoded signal is then provided to an inverse transform circuit 106 (e.g., an IDCT or integer transform), which inversely blocks on a zigzag decoded video signal. Inverse discrete cosine transform is performed to provide statically decompressed pixel values or decompressed error terms. The statically decompressed pixel values are processed on a block-by-block basis via motion compensation unit 108, where motion compensation unit 108 provides intraframe, predicted and bidirectional motion compensation, It includes support for two and four motion vectors (16x16, 16x8 and 8x8 blocks). The in-loop filter 110 performs overlap smoothing and / or deblocking to reduce or eliminate blocking artifacts according to the WMV9 compression standard, so that the in-loop filter 110 stores the partially terminated macro block filter data ( 111), which will be described later. Color space converter 112 converts one or more input data formats (eg, YCbCr 4: 2: 0) into one or more output formats (eg, RGB), and the result is a filter Filtered and / or scaled at 116.

As disclosed herein, the smoothing and deblocking in-loop filter 110 removes boundary discontinuities between adjacent blocks by partially filtering or processing each row of macro blocks during the first pass. The processing for the partially processed blocks is completed while processing the next row of macro blocks. Due to this technique, the small scratch pad memory 111 can be effectively used to store partially processed blocks in the scratch pad memory, which is a large memory that stores the entire frame image for filtering in the deblocking process according to the prior art. This is quite different from using.

If processing of each block for overlap smoothing and deblocking is completed on a row by row basis, the completed blocks are sent from the filter 110 to the FIFO buffer (before being sent to the Color Space Converter (CSC) 112). It may be output to (not shown).

3 is an exemplary diagram illustrating a macroblock based in-loop filtering process using a scratch pad memory to effectively handle overlap smoothing and deblocking in accordance with a selected embodiment of the present invention. In the filtering process, each pass of the in-loop filter for the row of macro blocks is completed blocks (fully filtered to smooth and deblock the blocks) and partially completed blocks. (Stored in scratch pad memory for subsequent use in smoothing and deblocking the next row of macro blocks). As shown, every individual macro block (e.g., macro block 4 or "mb4" comprising four luma blocks, mb4y0, mb4y1, mb4y2, and mb4y3) may be subjected to the following in-loop filtering process: It goes through the processing sequence.

(Iii) Completely smooth and deblock the 8x8 blocks (e.g., mb4y0, mb4y1) adjacent to the previous macro block (e.g., macroblock 1), and then complete the next macroblock (e.g., Partially complete smoothing and deblocking for 8x8 blocks (eg, mb4y2, mb4y3) adjacent to macroblock 7;

(Ii) output the completed 8x8 blocks (eg mb4y0, mb4y1) and store the partially completed 8x8 blocks (eg mb4y2, mb4y3) in the scratch pad memory;

(Iii) when the next macro block (e.g., macro block 7) is processed, fetch the partially completed 8x8 blocks (e.g., mb4y2, mb4y3) from scratch pad memory, and fetch Complete the processing for the 8 × 8 blocks (eg, mb4y2, mb4y3); And

(Iii) Output the completed 8x8 blocks (eg, mb4y2, mb4y3) together with the completed 8x8 blocks (eg, mb7y0, mb7y1) of the next macro block.

Although detailed implementations may vary depending on the application, FIG. 3 illustrates that image frame 150 processed by in-loop filter 110 may include macro blocks (eg, mb0, mb1, mb2, An example implementation consisting of mb3, mb4, mb5, mb6, mb7, mb8 and the like, and arranged in multiple rows (e.g., first row 151 consisting of mb0, mb1 and mb2). An example is shown. As shown in 3A of FIG. 3, the in-loop filter 110 has already performed a first pass on the first row 151 of macro blocks. As a result of performing the first pass on the first row 151, the upper blocks mb0y0, mb0y1, mb1y0, mb1y1, mb2y0, mb2y1 are fully processed (indicated by cross-hatching) for overlap smoothing and deblocking, On the other hand, the lower blocks mb0y2, mb0y3, mb1y2, mb1y3, mb2y2, mb2y3 are only partially processed for overlap smoothing and deblocking. In order to complete the partially processed blocks of the first row 151 during the next pass of macro block processing, the partially processed blocks of the first row 151 are stored in the scratch pad memory 111 (dot pattern Indicated by).

As further shown in 3A of FIG. 3, the in-loop filter 110 begins processing the second row 152 of macro blocks, which is partially processed in the first row 151 by this process. Blocks will complete. As a result, the mb0y2 and mb3y0 blocks are fully processed for smoothing and deblocking, and the mb3y2 blocks are only partially processed (and stored in the scratch pad). For blocks to be processed by filter 110 in 3A of FIG. 3 (filtered block 154 is indicated by diagonal hatching), the mb0y3, mb3y1 and mb3y3 blocks are partially processed for smoothing and deblocking [ And held in filter 110], the mb1y2 block is a partially completed block fetched from the scratch pad, and the remaining blocks mb4y0 and mb4y2 are obtained from the current macro block (e.g., macro block 4). . As the in-loop filter 110 processes the filtered blocks 154, smoothing and deblocking for one or more partially processed blocks (eg, mb0y3, mb3y1) are completed, while remaining The blocks mb1y2, mb4y0, mb4y2 and mb3y3 are only partially completed.

After processing the filtered blocks 154, the in-loop filter 110 shifts to new data. This is illustrated in frame 155 shown in FIG. 3B, which outputs any completed blocks (eg, mb0y3, mb3yl) and outputs one or more partially completed blocks (eg, mb3y3). ) Into the scratch pad memory, shift the partially completed blocks (e.g. mbly2, mb4y0, mb4y2) remaining in the filter by one block position, and partially block from the previous row of macro blocks (e.g. For example, by fetching mb1y3 and loading new blocks (eg, mb4yl, mb4y3) from the current macro block, filter 110 obtains filtered blocks 156. As the in-loop filter 110 processes the filtered blocks 156, smoothing and deblocking for one or more partially processed blocks (eg, mbly2, mb4y0) are completed, and the remaining blocks are completed. (Mbly3, mb4yl, mb4y3 and mb4y2) are only partially completed.

After processing the filtered blocks 156, the in-loop filter 110 again shifts to new data, which is illustrated in frame 157 shown in 3C of FIG. In particular, it outputs any completed blocks (eg mbly2, mb4y0), stores one or more partially completed blocks (eg mb4y2) in scratch pad memory, and partially completed blocks remaining in the filter. (E.g., mbly3, mb4yl, mb4y3) shift by one block position, fetch a partially completed block (e.g. mb2y2) from the previous row of macro blocks, and new blocks from the current macro block ( For example, by loading mb5y0, mb5y2, filter 110 obtains filtered blocks 158. As the in-loop filter 110 processes the filtered blocks 158, smoothing and deblocking for one or more partially processed blocks (e.g., mbly3, mb4y1) are completed, and the remaining blocks (Mb2y2, mb5y0, mb5y2 and mb4y3) are only partially completed. At this point, smoothing and deblocking for the upper blocks mb4y0 and mb4y1 of macro block 4 are completed, but the lower blocks mb4y2 and mb4y3 are only partially completed. By storing the partially completed lower blocks in the scratch pad memory, the filter operation can be completed when the filter 110 processes the next row of macro blocks.

Further details of another embodiment of the present invention are illustrated in FIG. 4, which uses a smoothing and deblocking filter in a video encoder or decoder to reduce the blockiness of the decoded frame 200. Figure is a diagram. As will be appreciated, the depicted techniques may be used to process luma or chroma blocks and although there may be specific corner cases at the periphery of each frame, those skilled in the art Will adjust or apply the invention as desired. However, for the sake of simplicity, the presently disclosed content focuses on the in-loop filtering step performed on the inner macro blocks of each frame.

Referring to FIG. 4, once the video encoder / decoder generates at least one first macro block for a frame (201), the in-loop filter places the top row of macro blocks one macro block at a time. Processing to filter the boundaries of each block with its adjacent blocks. As will be appreciated, since no smoothing or deblocking is performed on the periphery of the frame, partially completed blocks used in the filtering process do not exist outside of the frame. However, as the first row of macro blocks are filtered, the scratch pad memory is filled with partially completed blocks. Beginning at the first macro block (201), the encoder / decoder loads the required blocks and retrieves any adjacent blocks partially completed from scratch pad memory (except the first row of macro blocks). If the macro block consists of four luma blocks (y0, yl, y2, y3) and two chroma blocks (Cb, Cr), the blocks enter the encoder / decoder in the following order: y0, yl , y2, y3, Cb, Cr.

Next, the video encoder / decoder filters 210 certain boundaries of blocks loaded with the filter, along with adjacent blocks or sub-blocks. In selected embodiments, a separate processing technique may be used to partially process each block in the filter. For example, after decoding an 8x8 block in either luminance or chrominance, all or some of the left and / or right (vertical) edges are subjected to smoothing filter processing (211). . Additionally or alternatively, all or part of the top and / or bottom (horizontal) edges of the block are subjected to smoothing filter processing (212). In addition to overlap smoothing, deblocking filter processing may be applied to all or part of selected horizontal boundaries of 8x8 blocks (213) and / or to all or part of selected horizontal boundaries of 8x4 sub-blocks. (214). Additionally or alternatively, the deblocking filter process may be applied to all or some of the selected vertical boundaries of 8x8 blocks (215) and / or to all or some of the selected vertical boundaries of 8x4 sub-blocks. 216.

Once the blocks in the filter are processed separately, the results are stored or shifted in the filter for further processing. In particular, for the filter to process new data, any completed blocks in the filter are output from the filter (217). In addition, any partially completed blocks that will not be processed with new blocks are stored in scratch pad memory for further processing with subsequent use and the next row of macro blocks, which is the last of the macro blocks. Until the row is processed (until a positive result at decision step 218), the step of storing in scratch pad memory may be omitted.

Since extra space has been created in the filter by storing the selected blocks (217, 219), the filter can now process new data. In particular, if there are additional blocks in the frame (positive result at decision 220), the partially filtered blocks remaining in the filter are shifted to the left (222). For rows below the top row (negative result at decision step 224), the space available in the filter is filled by retrieving the next block that is partially completed from scratch pad memory (226), and any The remaining space is filled with new blocks (228). Once the filter is loaded with new data, the block filtering process 210 is repeated for a new set of filter blocks. By repeating this series of operations, each macro block in the frame is successively filtered, retrieving from the scratch pad memory the partially completed blocks created during processing for the previous row of macro blocks, and removing the partially filtered blocks. It stores in scratch pad memory for subsequent use during processing for the next row of macro blocks. On the other hand, if there are no blocks left to be filtered (negative result at decision step 220), the smoothing and deblocking process for the current frame ends. At this point the next frame is retrieved (230), and the filtering process is repeated starting at the first macro block of the new frame.

Now, a description will be given with reference to FIGS. 5A to 5K. 5A-5K illustrate an exemplary embodiment of the present invention, wherein the WMV smoothing and deblocking procedure for luma blocks of macro block 4 (or mb4) may be implemented using a separate processing technique. It is a figure which shows whether there exists. The starting point of the filtering process is shown in FIG. 5A, where filter 320 is already partially partially for overlap smoothing (e.g., indicated by elliptical 322) and deblocking (e.g., indicated by straight line 323). It is already loaded into the processed blocks mb0y3, mb3y1 and mb3y3.

The filter 320 is then filled with additional blocks as shown in FIG. 5B. In particular, the partially completed block (eg mb1y2) is retrieved from the scratch pad memory and loaded into the filter. Also selected blocks (eg mb4y0, mb4y2) from the current macro block (eg mb4) are loaded into the encoder / decoder. Although the macro block loading sequence (eg mb4y0, mb4yl, mb4y2 and mb4y3) requires at least one of the blocks to be loaded, it does not require shifting to filter 320 at this point in time. This is as shown at 321.

Once the filter blocks are loaded, filter 320 performs fractional overlap smoothing as shown in FIG. 5C. In particular, vertical overlap smoothing (V) is performed on the selected inner vertical edges 301, 302. Next, horizontal overlap smoothing H is performed for the selected inner horizontal edges (e.g., 303, 304, 305 and 306).

After the filter blocks are partially smoothed, the filter 320 performs fractional deblocking as shown in FIG. 5D. First, horizontal in-loop deblocking (HD) is performed on selected 8x8 block boundaries (e.g., 307, 308, 309, 310), and then selected sub-block boundaries (e.g., 311). Horizontal in-loop deblocking (HDH) is performed for 312, 313, and 314. Next, the filter performs vertical in-loop deblocking (VD) for the selected 8x8 block boundaries (eg, 315, 316), and then selects the selected sub-block boundaries (eg, 317). Vertical in-loop deblocking (VDH).

In each of the smoothing and deblocking steps described above, the order in which the boundary pieces are filtered is not a problem, since there is no dependency between the boundary pieces. Furthermore, in addition to the specific ordering processing of divisions shown in Figures 5C and 5D, other ordering and / or filtering steps may also be implemented in accordance with the present invention. For example, boundary pieces may be filtered in a different order, one or more smoothing or deblocking modes may be applied, and the filtering operation may be performed on three or more pixels on both sides of the boundary. This depends on the filtering scheme, for example the MPEG-4 standard uses two deblocking modes, in which a short filter is placed on both sides of the block edge. In the second mode, a longer filter is applied to two pixels on both sides of the block edge In other embodiments, filter definitions, number of different filters And / or adaptive filtering conditions may be used to meet certain requirements.

Once the smoothing and deblocking filter operation is completed, the processed filter blocks are stored and shifted as shown in FIG. 5E. In particular, any completed blocks (eg, mb0y3, mb3yl) may then be output, since these blocks have been completed. In addition, one or more partially completed blocks (e.g., mb3y3) may be used for scratch pad memory for subsequent use in processing downwardly adjacent macro blocks (see macroblock 6 in relation to mb3y3 in FIG. 3). May be moved to. Any partially completed blocks remaining in the filter (eg, mbly2, mb4y0, mb4y2) may then be shifted into the filter to make room for new data. The results of the output, store, and shift steps are shown in FIG. 5F.

Filter 320 may now be filled with new data blocks as shown in FIG. 5G. In particular, the partially completed block (eg mb1y3) is retrieved from the scratch pad memory and loaded into the filter. In addition, the remaining blocks (eg mb4yl, mb4y3) are loaded into the filter 320 from the current macro block (eg mb4).

Once the filter blocks are loaded, filter 320 performs fractional overlap smoothing as shown in FIG. 5H. In particular, vertical overlap smoothing (V1, V2) is performed on the selected inner vertical edges. Next, horizontal overlap smoothing (H1, H2, H3, H4) is performed on the selected inner horizontal edges.

After the filter blocks have been partially smoothed, filter 320 performs fractional deblocking as shown in FIG. 5I. First, horizontal in-loop deblocking (HD1, HD2, HD3, HD4) is performed on selected 8x8 block boundaries, and horizontal in-loop deblocking (HDH1, HDH2, HDH3, HDH4) at selected subblock boundaries. ) Is subsequently performed. Next, the filter performs vertical in-loop deblocking (VD1, VD2) for the selected 8x8 block boundaries, and then vertical in-loop deblocking (VDH1, VDH2) for the selected sub-block boundaries. This is done.

Once the smoothing and deblocking filter operation is complete, the processed filter blocks are stored and shifted as shown in FIG. 5J. In particular, any completed blocks (eg mb1y2, mb4y0) may then be output, since these blocks have been completed. In addition, one or more partially completed blocks (eg, mb4y2) may be moved to the scratch pad memory for subsequent use when processing a downwardly adjacent macro block. Any partially completed blocks remaining in the filter (eg, mbly3, mb4y1, mb4y3) may then be shifted into the filter to make room for new data. The results of the output, store, and shift steps are shown in FIG. 5K, which corresponds exactly to the initial filter state shown in FIG. 5A. As a result, the series of steps shown in FIGS. 5A-5K may be repeated to continue filtering the filter blocks that include the next macro block (e.g., macroblock 5 or mb5).

Referring now to FIGS. 6A-6F, an exemplary embodiment of the present invention is shown, which illustrates how the WMV9 smoothing and deblocking procedure for Cb or Cr blocks of a macroblock may be implemented using a discrete processing technique. Is shown. Since the Cb block and Cr blocks are similar, the embodiment is described with respect to the current Cb macro block identified by the index label Cb (x, y). The starting point of the filtering process is shown in FIG. 6A, where blocks that have already been partially processed for overlap smoothing (eg, indicated by elliptical 422) and deblocking (eg, indicated by straight line 423) [eg For example, filter 420 is already loaded with Cb (x-1, y-1) and Cb (x-1, y)].

Thereafter, the filter 420 is filled with additional blocks as shown in FIG. 6B. In particular, a partially completed block (eg Cb (x, y-1)) is retrieved from the scratch pad memory and loaded into the filter. Also, a block (eg, Cb (x, y)) from the current macro block is loaded into the filter 420. Once the filter blocks are loaded, the filter 420 performs fractional overlap smoothing, as shown in FIG. 6C. In particular, vertical overlap smoothing (V) is performed on the selected inner vertical edges. Next, horizontal overlap smoothing H1, H2 is performed on the selected inner horizontal edges.

After the filter blocks are partially smoothed, the filter 420 performs fractional deblocking as shown in FIG. 6D. First, horizontal in-loop deblocking (HD1, HD2) is performed on selected 8x8 block boundaries, and horizontal in-loop deblocking (HDH1, HDH2) is subsequently performed at selected subblock boundaries. Next, the filter performs vertical in-loop deblocking (VD) on the selected 8x8 block boundaries, and then vertical in-loop deblocking (VDH) on the selected sub-block boundaries.

Once the smoothing and deblocking filter operation is completed, the processed filter blocks are stored and shifted as shown in FIG. 6E. In particular, any completed blocks (eg, Cb (x-1, y-1)) may be output because these blocks are complete. In addition, one or more partially completed blocks (e.g., Cb (x-1, y1)) may be moved to the scratch pad memory for subsequent use when processing a downwardly adjacent macro block. Any partially completed blocks remaining in the filter (eg, Cb (x, y-1), Cb (x, y)) may then be shifted into the filter to make room for new data. . The results of the output, store, and shift steps are shown in FIG. 6F, which corresponds exactly to the initial filter state shown in FIG. 6A. As a result, the series of steps shown in FIGS. 6A-6F may be repeated to filter the next macro block.

As described earlier, by providing a small scratch pad memory to the hardware decoder unit, the in-loop filter is configured to partially remove from the luma and chroma blocks of the current macro block (indicated by MB (x, y)). Completed filtering results may be temporarily stored in the scratch pad memory. The stored filtering results may then be used to process blocks of the lower row adjacent to the macro block. In particular, when the filter processes the macro block immediately below, that is, MB (x, y + 1), the stored data for MB (x, y) is fetched from scratch pad memory and MB (x, y + Used to process 1).

Although the partially completed filtering results stored in the scratch pad memory should include at least 8 × 8 pixel data, in one selected embodiment, the scratch pad memory also includes control data to determine if boundary filtering is required for the block. Save it. For example, the control data includes a group of headers for six blocks of the current macro block of each block, and includes a 1mv or 4mv selector, block address, position of the block in the frame, mbmode, frame size, coefficient (0 or Nonzero) and motion vectors (two for forward in x and y directions, two for forward in x and y directions). The data may be packed in a manner that allows for efficient use of the burst size.

Because of the small size of scratch pad memory, the memory may be located on the same chip as a video accelerator, and for typical frame sizes, the scratch pad memory may be located on another chip, such as, for example, DDR memory or other external memory. have. However, by placing the scratch pad memory in the video accelerator hardware unit 10, improved memory access performance is obtained. By minimizing the size of the scratch pad memory, the manufacturing cost of the media accelerator hardware unit may be reduced compared to including a large capacity memory buffer to store data blocks for the entire frame. For example, the scratch pad memory used to store partially completed filtering results including control data and pixel data may be calculated as follows:

Size of scratch pad = (576 bytes) x (number of macro blocks horizontally in the frame)

As described above, when the size of the frame is large in the vertical direction, the size of the scratch pad is relatively small. In other words, the size of the scratch pad depends on the frame size in the horizontal direction. As long as filtering can begin on the first macro block before the entire frame is decoded, the fractional processing techniques disclosed herein can be usefully used to improve the speed of the filtering operation, even with respect to the large memory holding the entire frame. You will see that it may. However, using scratch pad memory has advantages in cost and speed compared to using large memory to store the entire frame, and using large memory to store the entire frame is expensive and access time. This has the disadvantage of adding overhead. In addition, the use of scratch pad memory is very well suited for pipelined processing in filtering algorithms.

The specific embodiments described above are merely exemplary and should not be taken as limiting the invention, and the invention may be modified or practiced through different but equivalent methods, which methods benefit from the teachings herein. It will be apparent to those skilled in the art to be obtained. The foregoing detailed description, therefore, is not intended to limit the invention to the precise forms given, but on the contrary, such alternatives may be included within the spirit and scope of the invention as defined by the appended claims. Examples, modifications, and equivalents are intended to cover those skilled in the art, and those skilled in the art will appreciate that various changes, substitutions, and alternatives can be made without departing from the spirit and scope of the invention in its broadest form. It should be understood that it can be made.

Claims (10)

1. A method of decoding video data processed by block transformation into a plurality of macro blocks, the method comprising: Smoothing and deblocking selected pixel data of the first macroblock to produce at least one first partially filtered block and at least one first completed block; And Storing the first partially filtered block in scratch pad memory for use in smoothing and deblocking selected pixel data of a second macro block. Method for decoding the video data made, including. The method of claim 1, Fetching a second partially filtered block from the scratch pad memory for use in the smoothing and deblocking steps, And said second partially filtered block was previously generated during processing of a prior macro block. The method of claim 1, Said smoothing and deblocking steps are performed by an in-loop filter. The method of claim 3, wherein The in-loop filter sequentially processes each row of macro blocks of a video frame to smooth and deblock one macro block at a time. The method of claim 3, wherein the in-loop filter, A method of decoding video data, characterized in that the smoothing and deblocking for a plurality of macro blocks are performed sequentially in a pipelined manner. The method of claim 1, Smoothing and deblocking selected pixel data of the first row of macro blocks to produce partially filtered blocks; And Storing the partially filtered blocks in scratch pad memory The method of decoding video data, further comprising. The method of claim 6, When smoothing and deblocking selected pixel data of a second row of macro blocks, retrieving a first partially filtered block from the scratch pad memory; And Smoothing and deblocking selected pixel data of the first partially filtered block to complete smoothing and deblocking on the first partially filtered block, thereby generating a completed block The method of decoding video data, further comprising. The method of claim 1, wherein the smoothing and deblocking step comprises: And at least one first smoothing and deblocking of at least one first block of each macro block, Wherein the first block is partially processed in a first filter operation, the first block is stored in the scratch pad memory, and the first block is subsequently fully processed in a second filter operation. How to decode. A video processing system for decoding video information from a compressed video data stream, comprising: A processor that partially decodes the compressed video data stream to produce partially decoded video data; And Video decode circuitry to decode the partially decoded video data to produce video frames It is made, including And the video decode circuit comprises a scratch pad memory and an in-loop filter that continuously perform separate processing of overlap smoothing and in-loop deblocking for a plurality of macro blocks of the video frames. . The method of claim 9, The scratch pad memory stores partially filtered blocks from the first row of macro blocks, wherein each partially filtered block is fetched by the in-loop filter during overlap smoothing and deblocking for the second row of macro blocks. And device which may be.

Images