JP2003348598A - Method and apparatus for memory efficient compressed domain video processing and for fast inverse motion compensation using factorization and integer approximation - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and apparatus for memory efficient compressed domain video processing and for fast inverse motion compensation using factorization and integer approximation. <P>SOLUTION: A method for reducing memory requirements needed to decode a bit stream comprises: receiving a video bit stream; decoding the frame of the bit stream into a discrete cosine transform (DCT) domain representation; identifying non-zero coefficients of the DCT domain representation; assembling a hybrid data structure; and inserting the nonzero coefficients of the DCT domain representation into the hybrid data structure. A method for performing inverse motion compensation is provided. The method initiates with receiving a video bit stream, and then, a transform matrix type is identified. The transform matrix type is either a half pixel matrix or a full pixel matrix. If the transform matrix type is a half pixel matrix, then the method includes applying a factorization technique to decode the bit stream corresponding to the half pixel matrix. If the transform matrix type is a full pixel matrix, then the method includes applying an integer approximation technique to decode the bit stream corresponding to the full pixel matrix. <P>COPYRIGHT: (C)2004,JPO

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates generally to digital video technology, and more particularly to a method and apparatus for implementing an efficient memory compression method, and for a compressed domain video decoder. And an apparatus for realizing an efficient reverse motion compensation method of the present invention.

[0002]

BACKGROUND OF THE INVENTION Video access at mobile terminals such as cellular telephones and personal digital assistants encounters a number of difficult challenges due to the limitations of the nature of mobile systems. For example, low power handheld devices are limited by bandwidth, power, memory, and cost requirements. Video data received by such a handheld device is decoded by a video decoder. Video decoders associated with such terminals perform motion compensation in the spatial, or decompressed, domain. Video compression standards such as H.263, H261, and MPEG1 / 2/4 use a motion compensated discrete cosine transform (DCT) scheme to encode video at low bit rates. The low bit rate adopted here is approximately 6 per second.
A bit rate of less than 4 kilobits. The DCT scheme uses motion estimation (ME) and motion compensation (MC) to remove temporal redundancy, while using DCT to remove other spatial redundancy.

FIG. 1 is a schematic diagram of a video decoder for decoding video data and performing motion compensation in the spatial domain. Bit stream 102 is received by decoder 100. The decoder 100 includes a variable decoder (VLD) stage 104, a run-length decoder (RLD) stage 106, an anti-quantization (DQ) stage 108, and an inverse discrete cosine transform (IDCT) stage 11.
0, a motion compensation (MC) stage 112, and a memory (MEM) 114, also called a frame buffer. First four stages (VLD104, RLD106, DQ108, IDCT
110) decodes the compressed bit stream and returns it to the pixel area. Intra-coded (intrac
oded) block, the first four stages, ie, 104, 106, 10 to reconstruct the block in the current frame
8, 110 outputs are used directly. For an intercoded block, the output represents the prediction error, and the output is added to the prediction made from the previous frame to reconstruct the block in the current frame. Therefore, the current frame is reconstructed in block units. Finally, the current frame is sent to the decoder, the output of the display 116, and is also held in the frame buffer (MEM) 114.

The MEM 114 holds an already decoded picture required for the motion compensation 112. M
The size of the EM 114 must be scaled according to the incoming image format. For example, H.263
Has five standardized image formats:
(1) 1/4 or less (sub-quarter) common intermediate format (sub-QCIF), (2) 1/4 common intermediate format (QCI
F), (3) Common intermediate format (CIF), (4) 4CI
F, and (5) 16CIF. Each format defines the aspect ratio as well as the width and height of the image. As is widely known, an image is coded as one luminance component and two chrominance components (Y, Cr, Cb). The components are sampled in a 4: 2: 0 configuration, with each component having an 8-bit resolution per pixel. For example, the video decoder of FIG.
About 200 kilobytes of memory must be allocated for MEM 114 while decoding the bitstream. Further, if multiple bit streams are decoded at once, as is essential in video conferencing systems, the memory demands will be too great.

[0005] The MEM 114 is the single largest memory use source in the video decoder 100. One approach is to reduce the resolution of the color components of the incoming bitstream to reduce memory usage. For example, if the color display density on a mobile device is 65,5
If only 36 colors can be displayed, color components (Y, Cr, Cb)
Can be reduced from 24 bits to 16 bits per pixel. Although this technique can potentially reduce memory usage by 30%, it is a display-dependent solution that must be accommodated circuitically in the video decoder. Also, this technique has no flexibility because it cannot be easily scaled with varying peak signal-to-noise ratio (PSNR) requirements.

[0006] Performing operations on data in the spatial domain requires a larger memory capacity than the compressed domain processing. In the spatial domain, it is easy to calculate motion compensation and apply motion compensation to images of continuous frames. However, when operating in the compressed domain, the motion compensation is based on the fact that the error value is no longer a spatial value, that is, the error value when operating in the compressed domain is not a pixel value, so the motion vector is Not as clear as showing a frame. Moreover, there is no method that has the ability to process compressed domain data efficiently. Prior art approaches mainly transcode and scale the compressed domain,
Focuses on sharpening each application. In addition, compression-compensated de-compensation applications tend to have poor peak signal-to-noise ratio (PSNR) performance while accepting response times in terms of the amount of frames that can be displayed per second. Difficultly slow.

[0007]

[Patent Document 1] US Reissued Patent Invention No. 6,240,210

[Patent Document 2] US Patent No. 6,157,740

[0008]

SUMMARY OF THE INVENTION Accordingly, there is provided a method and apparatus for minimizing the amount of memory required to decode low bit rate video data, while providing a fast and efficient compression domain video recorder. There is a need to solve the problems of the prior art in order to provide a method and apparatus that enables reverse motion compensation.

[0009]

SUMMARY OF THE INVENTION Broadly speaking, the present invention addresses these needs by providing a video decoder configured to minimize memory requirements by employing a hybrid data structure. It satisfies at least one aspect. Note that this aspect of the invention can be implemented in various ways, such as in a method, system, computer readable medium, or device. Several examples of this aspect of the invention are described below.

In one embodiment, a method is provided for reducing the amount of memory required to decode a bitstream. The method starts with receiving a video bitstream. Next, the frame of the bitstream is transformed (eg, discrete cosine transform (DCT))
Decoded to region representation. Next, the non-zero coefficients of the transformed domain representation are identified. Next, the hybrid data structure is assembled. The hybrid data structure includes a fixed size array and a variable size overflow vector. Next, the non-zero coefficients of the transform domain representation are inserted into the hybrid data structure.

[0011] In another embodiment, a method is provided for decoding video data. The method begins by receiving a frame of video data in a compressed bitstream. Next, the blocks in that frame are
Decoded into a transformed (eg, DCT) domain representation in the compressed domain. Next, a hybrid data structure is defined. Next, data associated with the transformed domain representation is held in a hybrid data structure. Next, inverse motion compensation is performed on the data associated with the transformed domain representation in the compressed domain. After performing reverse motion compensation on the data, the data is decompressed for display.

In yet another embodiment, a computer readable medium having program instructions for reordering low rate bit stream data to be maintained in a hybrid data structure is provided. The computer readable medium contains program instructions for identifying a non-zero transform (eg, DCT) coefficient associated with an encoded block of a data frame. Contains program instructions for arranging the non-zero transform coefficients in a fixed size array. Program instructions are provided for determining whether the quantity of non-zero transform coefficients exceeds the capacity of the fixed size array. Includes program instructions for holding non-zero transform coefficients exceeding the capacity of the fixed-size array in a variable-size overflow vector, and program instructions for translating non-zero transform coefficients from the compressed domain to the spatial domain. .

[0013] In yet another embodiment, a circuit is provided. This circuit has a video decoder integrated circuit chip. The video decoder integrated circuit chip includes circuitry for receiving a bit stream of data associated with a frame of video data. Video decoder converts the bit stream of data (eg, DCT)
A circuit configuration for decoding is included in the region representation. Circuit arrangement for arranging non-zero transform coefficients of a transform domain representation in a hybrid data structure in a memory associated with a video decoder. Circuitry for decompressing the non-zero transform coefficients of the transform domain representation for display is also provided.

In another embodiment, an apparatus is provided that is configured to display an image. The device includes a central processing unit (CPU), a random access memory (RAM), and a display screen configured to display an image. A decoder circuit arrangement configured to convert the video bitstream to a transform (eg, DCT) domain representation. The decoder circuit has the ability to arrange the non-zero transform coefficients of the transform domain representation in a hybrid data structure in a memory associated with the decoder circuit. The decoder circuit has a circuit configuration for selectively applying the hybrid factorization / integer approximation technique at the time of inverse motion compensation. It also has a bus in communication with the CPU, RAM, display screen, and decoder circuit.

Broadly speaking, the present invention provides a video decoder which has a reduced video footprint while providing acceptable video quality while at the same time providing the ability to perform inverse motion compensation in the compressed domain. Fulfilling at least another aspect of these needs.
It should be noted that this aspect of the present invention can be implemented in various ways, such as a method, a system, a computer-readable medium, or an apparatus. Some examples of this aspect of the invention are described below.

In one embodiment, a method is provided for performing inverse memory compensation. The method starts with receiving a video bitstream. Next, the transformation matrix type is identified. This transformation matrix type is either a half pixel matrix or a full pixel matrix. This method
If the transformation matrix type is a half-pixel matrix, this includes applying a factorization technique to decode a bitstream corresponding to the half-pixel matrix. If the transformation matrix type is a full pixel matrix, this involves applying an integer approximation technique to decode the bitstream corresponding to the full pixel matrix.

In another embodiment, a method is provided for decoding video data. The method begins by receiving a frame of video data in a compressed bitstream. Next, the blocks of the frame are decoded in the compressed domain to a transform (eg, discrete cosine transform (DCT)) domain representation. Next, data associated with the transformed domain representation is held in a hybrid data structure. Next, inverse motion compensation is performed on the data associated with the transformed domain representation in the compressed domain. Performing inverse motion compensation involves determining the type of transform matrix associated with a portion of the frame of video data and applying hybrid factorization and integer approximation techniques to improve inverse motion compensation.

In yet another embodiment, a computer readable medium having program instructions for performing inverse motion compensation in a compressed domain is provided. The computer readable medium contains program instructions for identifying a transformation matrix. Contains program instructions for determining whether the transformation matrix is a half-pixel matrix or a complete pixel matrix.
The program instructions for applying the factorization technique for decoding the block of the bitstream corresponding to the half-pixel matrix and the program instructions for applying the integer approximation technique for decoding the block of the bitstream corresponding to the full pixel matrix Contains.

In yet another embodiment, a circuit is provided. The circuit includes an integrated circuit chip configured to decode video data. The integrated circuit chip includes circuitry for receiving a bit stream of data associated with a frame of video data. The integrated circuit chip has a circuit configuration for decoding a bit stream of data into a transform (eg, DCT) domain representation. A circuit configuration for identifying the type of the transformation matrix and a circuit configuration for performing inverse motion compensation by hybrid factorization and integer approximation techniques are mounted on an integrated circuit chip.

In another embodiment, a video decoder is provided. The video decoder includes a variable length decoder (VLD) configured to extract coefficient values and motion vector data from an incoming bit stream. V
It has an anti-quantization block in communication with the LD. The anti-quantization block is configured to rescale the coefficient values. A downstream branch in communication with the anti-quantization block is provided. This downstream branch is configured to decode the error coefficients into the spatial domain.
Includes an upstream branch in communication with the anti-quantization block. This upstream branch uses internal transformations (eg, DC
T) It is configured to maintain domain transformation. The upstream branch is further configured to generate a spatial domain output that can be added to the decoded error coefficients to reconstruct the current block.

Other aspects and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention.

[0022]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention is described as a system, apparatus, and method for minimizing the amount of memory required for compressed domain video decoding. However, it will be apparent to one skilled in the art, in light of the following description, that the present invention may be practiced without some or all of the details described below. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention. FIG. 1 has been described in the section of “Prior Art”. As used herein, the term "about" refers to +/- 10% of the reference value.

The embodiments described herein provide a data structure that allows for a reduction in the memory used in decoding video data in the compressed domain. In one embodiment, the video decoding pipeline is reordered so that the current frame is retained and inverse motion compensation is performed in the frequency domain, or compression domain. The hybrid data structure is
Enables data manipulation in the compressed domain without significant computational cost or loss of data. In one embodiment, the hybrid data structure takes advantage of the fact that there are very few non-zero discrete cosine transform (DCT) coefficients in the encoded block. Therefore, the non-zero
Since only the DCT coefficient is retained, the memory requirement can be reduced. As described in more detail below, the hybrid data structure includes a fixed size array and a variable size overflow vector. The variable size overflow vector holds the non-zero DCT coefficients of the coded block beyond the capacity of the fixed size array.

FIG. 2 is a schematic diagram of a video decoder arranged to perform inverse motion compensation, according to one embodiment of the present invention. Here, the bit stream 122 is received by the video decoder 120. The first two stages, the variable length decoder (VLD) stage 12
4 and the dequantization (DQ) stage 126 decodes the compressed bitstream into a DCT domain representation. The DCT domain representation is for use in the motion compensation (MC) stage 134,
It is held in a memory (MEM) 130 also called a frame buffer. After a motion compensation feedback loop including MC 128 and MEM 134, a run length decoder (RLD) stage 132 and an inverse DCT (IDCT) stage 13
4 is executed. Therefore, the internal representation of the decoded block remains a compressed domain. Because there are only a few non-zero DCT coefficients in the coded block, this feature can be exploited by developing a MEM 130 data structure that holds only the non-zero DCT coefficients of each block in the frame.
Can be used. As will be explained in more detail below, the memory compression enabled by the hybrid data structure reduces the memory usage by 5 without loss of video quality.
It can be reduced by 0%. The human visual system is a high-level DCT
Since it is more sensitive to the lower DCT coefficients than the coefficients, thresholding is used to filter out higher DCTs and trade off memory usage versus fluctuating power or peak signal to noise ratio requirements, as described below. Developed a scheme.

A description will now be given of fully-compressed area video decoding which is optimized so as to enable high-speed and memory-efficient decoding. In one embodiment, a video decoder from TELENOR, a public domain H.263 compliant decoder, was used for the tests referred to herein. It should be noted that some of the embodiments described below are referred to as H.263 bit streams.
It is not limited to operations on 3 bit streams. That is, Motion Picture Expert Group (MPEG) 1/2/4, H.
Any DCT-based compressed bit stream with video data, such as 261 can be employed. Discrete cosine transform (DC) enables efficient processing in the compressed domain
T) There are many fast inverse motion compensation algorithms for region representation. Note that the memory compression method of storing non-zero DCT coefficients in an encoded block is processing in a compression area, so that the required memory amount can be reduced. Further, to demonstrate various performance trade-offs in speed and memory optimization, the performance of a video decoder employing the inverse motion compensation technique and the compression domain processing using memory compression described herein is reduced by three. In dimensions, that is, the complexity,
Evaluate in terms of memory efficiency and PSNR.

[0026]

(Equation 1)

[0027]

(Equation 2)

[0028]

(Equation 3)

[0029]

(Equation 4)

(Equation 5)

Low bit rate video, ie, about 6 per second
Video data having a bit rate of less than 4 kilobits is used for video conferencing applications and for applications such as cellular phones, personal digital assistants (PDAs), and wireless video on other handheld and battery-powered devices. . The H.263 standard is a schematic standard specifying a bitstream syntax and algorithm for decoding low bit rates video. Algorithms include transform coding, motion estimation / compensation, coefficient quantization, and run-length coding. Apart from baseline specification, version 2 of the standard also supports 16 negotiable options that provide improved coding performance and provide error resilience.

Video encoded at a low bit rate is
Visible distortion can occur. This is especially the case for videos that have been categorized as action-rich, that is, blocks of active movement. As mentioned earlier, the embodiments described herein refer to the H.263 standard, but any suitable video codec standard may be used with the embodiments. For reference, H.26
Some of the functional features of the three standards are described below, but are not intended to limit the invention to use with the H.263 standard. One feature of the H.263 standard is that there is no group of pictures (GOP) and higher layers in this standard. If the baseline coded sequence consists only of a single intra-frame (I-frame) followed by a long sequence of inter-frames (P-frames), the long P-frame sequence is removed because temporal redundancy is removed between consecutive frames. Achieve higher compression ratios. However, motion estimation / motion compensation (ME / MC) introduces a time dependency, so that errors generated during the lossy encoding process accumulate during the decoding process. If there are not enough I-frames, the decoder cannot break this error accumulation. Because the H.263 standard has a forced update mechanism, the encoder must encode the macroblock as an intrablock at least once every 132 times during the encoding process. FIG. 4 is a diagram illustrating the effect of the forced update mechanism. As shown in FIG. 4, the video PS
The NR fluctuates randomly, but does not drift in any direction in the later frames of the row.

FIG. 5 is a schematic diagram illustrating the determination of the value of a half pixel in the H.263 standard. As is well known,
The H.263 standard employs half-pixel interpolation for motion compensation. In this standard, half-pixel interpolation uses 0.5 resolution (that is, <7.
5, 4.5>). The encoder can specify interpolation only in the horizontal direction, only in the vertical direction, or in both the horizontal and vertical directions. As shown in FIG. 5, the half-pixel values are found by bilinear interpolation of the integer pixel positions surrounding the half-pixel position. Pixel position A 150-1, pixel position B 150-2, pixel position C 150-3, and pixel position D 150-4 represent integer pixel positions, whereas pixel position e 152-1 and pixel position f 152-2, pixel position g 15
2-3 represents a half pixel position. E for horizontal interpolation
= (A + B + 1) >> 1 and the vertical interpolation is f = (A + C + 1) >> 1
Can be expressed as G = (A + B
+ C + D + 2) >> 2.

FIGS. 6A and 6B are schematic diagrams of a baseline spatial video decoder and a compressed domain video decoder, respectively. The block diagram in FIG. 6B is obtained by partially rearranging the functional blocks of the spatial domain video decoder in FIG. 6A. In particular, RLD 132 and IDCT 134 have moved after the MC 28 feedback loop. This arrangement allows the internal representation of the video to be preserved in the compressed domain. In the arrangement of FIG. 6B, the compression domain post-processing module can be inserted immediately after the MC128 feedback loop. Certain video operations, such as compositing, scaling, and deblocking, are faster in the compressed domain than in the spatial domain. However, from a video codec perspective, a spatial encoder does not perfectly match a compressed domain decoder. As shown in FIG. 6B, the compressed domain video decoder differs from the spatial domain video decoder of FIG. 6A at some points along the decoding pipeline. Rather than just reordering blocks, the differences represent non-linear operations such as clipping and rounding. The point having these nonlinearities is
Generate a video with different PSNR measurements between the two regions.

[0034]

Those skilled in the art will recognize that the MEM 130 is a frame buffer that holds frames before motion compensation. In the spatial domain decoder, the frame buffer is
Allocate memory sufficient to accommodate the incoming frame size to hold the (Y, Cr, Cb) values. For example,
About 20 for CIF video sampled at 4: 2: 0
There must be 0 kilobytes of memory. MEM130
Since is the only maximum memory usage in a video decoder, the hybrid data structure and inverse motion compensation defined herein allow for reduced memory usage in the compression domain decoding pipeline. In one embodiment, two to three times the memory compression is achieved without significant loss of image quality in the decoded video.

FIG. 7 is a block diagram illustrating block conversion during the video encoding and decoding process according to one embodiment of the present invention. The sequence of transforms above the dotted line 170 describes the spatial compression method used by the video encoder for blocks in I-frames or blocks in P-frames after motion compensation / motion estimation. Pixel block 172 is a complete 8x8 matrix. At this point, any compression or truncation in the spatial domain directly affects the visible image quality of the reconstructed block. However, after the DCT transform, the transformed matrix 174 is compact and has large terms at low frequencies. The quantization step further compacts the block by zeroing out the higher frequency smaller terms in block 176. The zigzag scan highlighted in block 176 orders the DCT coefficients from low to high frequencies.
Run-length encoding is a binary element,
For example, in a compact list of runs and levels,
The zero coefficient is ignored, and only the non-zero DCT coefficients are represented in the run length representation 178. Thus, memory compression can be achieved in the DCT domain by developing efficient data structures and methods for holding and accessing run-length representations of non-zero DCT coefficients.

In one embodiment, the semi-compressed (SC) representation is
One such memory-efficient run-length representation. The run-length representation of the non-zero DCT coefficients is similar to the run-length representations 178 and 180 of FIG. However, there are two variants. Each binary element (run, level) is described by its composite 16-bit value.

RL = binary rrrrllllllllllll '(9) The least significant 12 bits (llllllllllll') are in block 1
84 defines the value of the anti-quantized DCT coefficient.
Block 184 is derived from quantized block 182. Note that the block 184 is an example of a DCT area expression. Those skilled in the art will appreciate that the value of the DCT
It is clear that the range is from 048 to 2047.
Block 186 of FIG. 7 is a reconstructed block of block 172 after performing the IDCT operation on block 184.
The four most significant bits (rrrr ') define the value of the run. The run represents the relative position of the non-zero DCT coefficients relative to the position of the ending non-zero DCT coefficients based on a zigzag scan within the 8x8 block. Since the run of non-zero coefficients can exceed 15, an escape sequence is defined to make the run smaller. The escape sequence RL = 'F0' is defined to represent a run with 15 zero coefficients followed by a zero amplitude coefficient.

To reduce memory requirements, data structures must be developed to hold and access SC representations. We considered the following data structures: arrays, linked lists, vectors, and hybrids. In developing these structures, we took into account the balance between the need for memory compression and the need to keep computations low. This will be further described below with reference to Table 1. The SC representation achieves the targeted memory compression, but the computational complexity of the decoder greatly increases in three fields depending on the specific data structure. First, the use of the 2-byte representation makes it impossible to use the value of (run, level) immediately. Accessing and modifying these values requires a function to pack and unpack bits each time. Second, motion compensation is complicated by the compact run-length representation.
Third, adding a prediction error to a prediction requires a sort and merge operation.

FIG. 8 is a schematic diagram illustrating the use of another index to find the starting position of each 8.times.8 block of the run-length representation. If a single list 190, also called a vector, is used to hold the run-length representation of all 8x8 blocks 192-1 to 192-4 in the frame, then to access a particular DCT block during motion compensation, Another index is needed to find the starting position. That complicates motion compensation.

FIGS. 9A and 9B illustrate the classification and merging operations required to add the prediction error to the prediction, for an array-based data structure and a list data structure, respectively. In FIG. 9A, the array-based data structure only requires the addition of the value at the corresponding array index. However, array-based data structures have no memory compression effect. In FIG. 9B, the list (or vector) data structure requires further classification and merging operations. That is, the merging algorithm must have insertion and deletion functions. It is very expensive for data structures, such as vectors, in terms of computational complexity. More specifically, if the indexes are equal, DCT coefficients can be added or divided. For example, (0,20) +
(0,620) = (0,640). If the wrong index precedes the prediction, DCT coefficients are inserted. For example, (0,
-3) is inserted. If the addition of DCT value results in 0, DCT
The coefficients are deleted. For example, (1,13) + (4, -13) = (1,0).

Table 1 compares memory compression ratios and calculation costs of various data structures. Although the array-based data structure does not incur any additional computational costs other than the 64 additions required to update the prediction, the array of DCT coefficients is
Since each DCT coefficient requires 2 bytes instead of 1 byte to hold, memory compression cannot be achieved compared to an array of pixels. Vectors in linked lists or semi-compressed (SC) representations can have up to 2.5 times the memory compression compared to arrays of pixels. However, the cost of inserting / deleting vectors is expensive, so neither solution is optimal. In particular, inserting and deleting in the middle of a vector is expensive. Also, since an internal pointer is created for each element of the list, the memory overhead of the linked list is expensive.

[Table 1]

With the hybrid data structure of the SC expression,
Table 1 provides an optimal balance between competing interests. 9A and 9B.
In order to take advantage of the high compression ratio of the vector structure, a hybrid data structure was developed. The hybrid data structure is composed of a fixed-size array that holds a fixed number of DCT coefficients for each block, and a variable-size overflow vector that holds DCT coefficients exceeding the fixed-size array allocation in these blocks. Note that the fixed size array can be configured to hold an arbitrary number of DCT coefficients suitable for each block. Where DCT
The number of coefficients is less than 64. Of course, the larger the fixed size array, the more memory compression is reduced. In one embodiment, the fixed size array is configured to hold eight DCT coefficients per block.

FIG. 10 is a schematic diagram of a hybrid data structure including an array structure and a vector structure that enables memory compression and computational efficiency, according to one embodiment of the present invention. DCT blocks 200-1, 200-2, 200-n
Contains a zero DCT coefficient and a non-zero DCT coefficient. In addition,
DCT blocks 200-1 to 200-n represent the DCT domain representation as described earlier in FIG. further,
Blocks 200-1 to 200-n are blocks of frames of video data, eg, block 184 of FIG.
Associated with Blocks 200-1 to 200-n
Are identified and inserted into the fixed-size array 202 data structure. The fixed size array 202 includes fixed size blocks 204-1 to 204-4.
-N. In one embodiment, each block 204-1 through 204-2
04-n is large enough to hold eight DCT coefficients in an 8 × 8 data structure. It should be noted that the present invention is not limited to a block configured to hold eight DCT coefficients, but may employ any suitable size. As described above, as the capacity of the block increases, the amount of memory compression decreases in proportion thereto.

In FIG. 10, if any of the DCT blocks 200-1 to 200-n contains more than eight non-zero coefficients, the fixed-size blocks 2
Non-zero DCT coefficients exceeding the capacity of 04-1 to 204-n are included in the overflow vector 206. The overflow vector 206 is configured as a variable size overflow vector. That is, the overflow vector is dynamic. For example, block 200-
1 includes nine non-zero DCT coefficients A1 to A9. Here, the DCT coefficients A1 to A8 are fixed size blocks 204-1.
, But the DCT coefficient A9 is copied to the overflow vector 206. Block 200-2 is 10
Number of non-zero DCT coefficients B1 to B10. Where DCT
The coefficients B1 to B8 are copied to the fixed size block 204-2, while the DCT coefficients B9 and B10 are copied to the overflow vector 206 and the like for each block of the frame. The index table 208 contains entries that identify the corresponding fixed-size blocks 204-1 to 204-n for the entries of the overflow vector 206. Since each entry is one byte, the size of the index table can be ignored. Therefore,
In frames of data corresponding to DCT blocks 200-1 to 200-n, to produce image 210,
Fixed size array 202 and overflow vector 2
06 are combined. Significant memory savings can be made. That is, DCT blocks 200-1 to 200-2
00-n most often decreases from 64 zero and non-zero coefficients to eight or less non-zero coefficients held in fixed size blocks 204-1 through 204-n. Of course, some non-zero coefficients may be provided, and if the non-zero coefficient exceeds 8, it will be retained in the overflow vector 206.

FIGS. 11A-11C are graphs illustrating factors evaluated in determining the capacity and overflow vector of a fixed size block of a fixed size array of a hybrid data structure, according to one embodiment of the present invention. . In FIG. 11A, the average number of non-zero DCT coefficients per luma block for two typical CIF sequences is depicted by lines 220 and 222. The number of non-zero coefficients per block ranges from three to seven. That is, out of the 64 coefficients, only 2 to 7 coefficients on average are non-zero coefficients.
FIG. 11B illustrates using the information of FIG. 11A as a guideline, as increasing the fixed size array reduces the size of the overflow vector, thereby minimizing vector insertion and deletion costs. . Here, line 220-1 corresponds to the CIF sequence of line 220 in FIG. 11A, while line 222-1 corresponds to the CIF sequence of line 222 in FIG. 11A. One skilled in the art will recognize that as the size of the fixed size array increases, memory compression decreases. In addition, FIG. 11C shows that the load factor of the array has also decreased, leaving the array almost empty. In one embodiment, a fixed size array was chosen that holds eight DCT coefficients per block. Again, line 220-2 is shown in FIG.
Line 222-2 corresponds to the CIF sequence of line 222 in FIG. 11A. This choice minimizes the size of the overflow vector, keeps the DCT coefficient at about 200, and reduces the loading factor to about 9% and about 1
Maintain between 5%. It will be apparent to those skilled in the art that the fixed size array is not limited to eight coefficients per block, but that the number of coefficients per block may be any suitable number. Further, the individual blocks of the fixed size array can be of any suitable configuration. For example, a block with the ability to hold eight coefficients
For example, a block that can be arranged as an 8x1 block, a 4x2 block, but has the ability to hold 9 coefficients, such as a 9x1 block, 3x3
Can be arranged as blocks.

FIG. 12 is a flowchart of the operation of a method for reducing the memory requirements required to decode a bitstream, according to one embodiment of the present invention. The method begins at operation 230 where a video bitstream is received. In one embodiment, the bitstream is a low rate bitstream. For example,
Video streams from H.263, Motion Pictures Expert
Group (MPEG-1 / 2/4), H.261, Joint Photographic Exp
It can be associated with video coding standards, such as the ert Group (JPEG). The method then proceeds to operation 23
Proceeding to 2, the frame of the bitstream is decoded into a discrete cosine transform (DCT) domain representation of each block of data associated with the frame. here,
The video is processed in the first two stages of the decoder, such as the decoder shown in FIGS. That is, the video data is processed in a variable length decoder stage and an anti-quantization stage to decode the compressed bit stream into a DCT domain representation. The DCT area representation is in a compressed state format. The frame is decoded one block at a time. The method then proceeds to operation 234 where non-zero coefficients of the DCT domain representation are identified. Here, of the 64 DCT coefficients associated with the DCT domain representation of the data block, a relatively small number of the 64 DCT coefficients are generally non-zero coefficients.

Continuing with FIG. 12, the method then proceeds to operation 236 to assemble the hybrid data structure. The hybrid data structure includes a fixed size array and a variable size overflow vector. One typical hybrid data structure is a fixed-size array including a plurality of fixed-size blocks and a variable-size overflow vector shown in FIG. The method then proceeds to operation 238, where the non-zero coefficients of the DCT domain representation are inserted into the hybrid data structure. As described in FIG. 10, the non-zero coefficients of the DCT domain representation of the video data block are associated with fixed size blocks in the fixed size array. When the number of non-zero coefficients exceeds the capacity of the fixed-size block associated with the video data block, the remaining non-zero coefficients are kept in a variable-size overflow vector. In one embodiment, the index table maps the data in the overflow vector to appropriate fixed-size blocks in a fixed-size array. Thus, the hybrid data structure and the retention of non-zero coefficients reduce memory requirements. More specifically, the memory requirement can be reduced by 50% without loss of video quality.

Each of the data frames
The non-zero coefficients of the DCT domain representation are kept in a hybrid data structure. The data of the retained frames is then combined and decompressed for display. In one embodiment, once the next frame has been decoded into a DCT domain representation to be retained in the hybrid data structure, the data associated with the previous frame in the hybrid data structure is flushed. As will be described further below, reverse motion compensation is performed on the data held in the compressed domain. In reverse motion compensation, integer approximation is used for perfect pixel reverse motion compensation, and factorization is used for half-pixel reverse motion compensation.

The main components of the spatial H.263 video decoder include run-length decoding, inverse DCT, and inverse motion compensation. Using a timing profiler, the performance of a TELENO.H.263 video decoder running on a 1.1 GHz Pentium® 4 processor is measured at baseline data. Decrypting the baseline data and ignoring the system call, the profiler returns 1
The overall time required to decode 44 frames is measured and the timing characteristics of each component are described in detail. Table 2 shows the timing profile of the spatial H.263 video decoder, specifically showing the timing results for selected functions.

[Table 2]

Table 3 shows that the non-optimized compressed area H.26
3 is a timing profile of a video decoder. One typical decoder pipeline configuration is the decoder shown in FIG.

[Table 3]

As shown in Table 2, the spatial domain video decoder takes about 1.2 seconds to decode 144 frames. Most of this time, for example, WINDOW ^TM
It uses an image display function that converts the color value of each frame from YUV to RGB for display on a suitable operating system. Functions such as run-length decoding, inverse DCT, and inverse motion compensation take up about 25% of the total time required to decode the video. Reverse motion compensation is particularly fast in the spatial domain. Here, full pixel motion compensation simply sets the pointer to a location in the memory or frame buffer and copies the data block, whereas half pixel motion compensation sets the pointer in memory and uses a shift operator. Interpolate values. In contrast, Table 3 shows some of the timing results of the non-optimized compressed domain video decoder. An unoptimized compressed domain decoder takes about 13.67 seconds to decode the same 144 frames.

[0053]

(Equation 6)

[0054]

[Table 4]

The multiplication of an 8 × 8 matrix requires 512 multiplications and 448 additions, respectively. As is known,
Matrix multiplication is computationally expensive. Table 5 shows a comparison of optimization schemes such as matrix multiplication, matrix factorization, and shared blocks of macroblocks with hybrid schemes of compressed domain video pipelines such as the pipelines shown in FIGS. To provide acceptable image quality on handheld devices supporting video formats, such as a common intermediate format, where each data frame includes 352 lines and has 288 pixels per line, the compressed domain video decoding pipeline is Decoding should be at a rate of about 15-25 frames per second (fps).

[Table 5]

One of the enhancements in the compressed domain video decoding pipeline is that block alignment
This is to reduce the number of TMi operations in equation (10). The block alignment is performed, for example, as follows: 144 columns of one column are decoded, and the block alignment ratio is measured at 36.7% of all blocks. FIG. 13 is a schematic diagram illustrating three examples of block alignment that reduces matrix multiplication. Block alignment 240
((W = 8, h = 4)), block alignment 242 ((w =
4, h = 8)) and block alignment 244 ((w = 8,
h = 8)). Example 24 of these
At 0, 242, and 244, when the overlap with the corresponding block becomes zero, the TMi operation disappears. However, in the DCT area (compression area), the block alignment does not save when half-pixel interpolation is specified. The equation for half-pixel motion compensation in the compressed domain is as follows. In the example of (w = 8, h = 8), half-pixel interpolation still requires four TMi operations, as shown in equations 12 and 13. Table 6 shows the half-pixel conversion matrix C _hpij
Is provided for reference in defining.

(Equation 7)

[Table 6]

Even with perfectly aligned blocks,
Half-pixel interpolation causes overlap with neighboring pixels. FIG. 14 is a schematic diagram of half-pixel interpolation of a perfectly aligned DCT block. Half-pixel interpolation results in overlap of neighboring blocks by one pixel width and one pixel height.

The processing speed of the compressed area decoding pipeline can be increased by rearranging the functional blocks of the decoder of FIG. In Tables 2 and 3, the processing time of the inverse DCT block is much shorter (3 ms) in the spatial domain than in the compressed domain (632 ms). In the spatial domain, the inverse DCT is used for intra blocks and error coefficients before the feedback loop. Specifically, intra blocks and error coefficients represent less than 15% of all blocks of the video. In the other 85%, the inverse DCT function is simply omitted. In the compression domain, an inverse DCT is used for 100% of the blocks in each frame of the video at the last stage of the pipeline.

FIG. 15 is a schematic diagram illustrating the reordering of functional blocks of a compressed domain video decoder that enhances the processing of video data, according to one embodiment of the present invention.
Here, the functional blocks are rearranged, and the compression area pipeline is divided into two places. First split is VLD
Occurs at point (i) 252 after 124 and DQ 126. In the upstream branch, the pipeline maintains the internal DCT domain representation of memory compression 128. In the downstream branch, the pipeline decodes the error coefficients into the spatial domain,
Move RLD and IDCT forward. The second split occurs at point (ii) 254 during motion compensation (MC). During motion compensation, a spatial domain output may be generated according to equation (7). The output can be added directly to the error coefficient to reconstruct the current block at point (iii) 256 for display on display 136. internal
DCT block 250 is inserted in the feedback loop to maintain the DCT representation. RLD13 at point (i) 252
2 with IDCT 134 and DCT at point (ii) 254
Requires less computation than the IDCT block in the last stage of the pipeline in FIG. Table 7 shows the possible combinations of FIG. 15 in addition to the other optimization schemes described herein.
It has been demonstrated that the rearrangement shown in FIG.

[Table 7]

In one embodiment, inverse motion compensation is accelerated by reducing the number of multiplications required for the basic TM operation of Equations (11, 13). Instead of computing a full 8x8 matrix multiplication, the DCT matrix S is factored into sparse matrix columns, as shown in equation 14. The sparse matrix of equation (17) includes an order matrix (A ₁ , A ₂ , A ₃ , A ₄ , A ₅ , A ₆ ) and a diagonal matrix (D, M). Substituting this factorization in equation (15) can be derived fully factored equations TM _i in equation (16). This has fewer multiplications than equation (11,13).

(Equation 8)

Thus, matrix multiplication is replaced by matrix permutation. However, as shown in equation (16), a completely factorized expression of the term TM _i , does not always speed up inverse motion compensation. Basically, since the multiplication was traded for memory access, too many memory accesses actually slow down the decoding process. Therefore, regrouping of the matrices is performed to balance between these competing functions. The matrix S (= G ₀ G ₁ ) has two terms, G ₀ = DA ₁ A ₂ A ₃ (mixed permutation and multiplication) and G ₁ = MA ₄ A ₅ A
₆ (mixture of permutation and addition). To form the equation factored for inverse motion compensation in equation (24), fixed matrices J _i , K _i are defined and equations (10 and 1)
Substituted in 2).

(Equation 9) Similarly, in half-pixel interpolation,

(Equation 10)

By realizing high-speed multiplication by the fixed matrices J _i and K _i , the speed can be further improved. Fixed matrices contain structural repetitions. For example, the matrix J ₆ is defined as follows.

(Equation 11)

Here, a = 0.7071, b = 0.9239, c = 0.3827
And Assuming u = {u ₁ , ..., u ₈ } and v = {v ₁ , ..., v ₈ }, to calculate u = J ₆ v, calculate the sequence of equations according to the following steps:

(Equation 12)

[0064]

In order to further speed up the reverse motion compensation, the multiplication necessary for the basic TM operation of the equations (11, 13) is deleted. Matrix of full and half pixels, C
_ij and C _Hpij is approximated to binary closest to powers of 2 ^-5. When these matrices are approximated to binary numbers, the equation (1
To solve the inverse motion compensation of (0, 12), matrix multiplication can be performed by using basic integer operations such as right shift and addition. For example, in the case of h = 1, examine the complete pixel matrix C ₁₁ as follows. Note that the other matrices are similarly approximated.

(Equation 13) Rounding each element of the matrix to the value closest to a power of 2 yields a matrix (47).

[Equation 14]

Since the DCT element is in the range of [-2048 to 2047], most values are driven to zero by directly shifting the DCT coefficient. To maintain the intermediate result precision scales each DCT coefficient 2 ⁸ throughout the decoding pipeline. Since this scaling factor is introduced in the quantization and anti-quantization steps, no extra computation occurs.

Further, high-speed matrix multiplication is realized by grouping terms according to the rule of sum of products (see equations (48 to 50)).

(Equation 15)

[0068]

(Equation 16)

[0069]

The hybrid factorization / integer approximation of the transformation matrix TM that is selectively applied based on the video motion, while maintaining acceptable image quality, is at the same time preferable to about 15
A frame rate of ２５25 fps is realized. As mentioned earlier, integer approximation techniques not only reduce the complexity of the decoder, but also reduce the PSNR of the decoded video. At the same time, the factorization technique can maintain good PSNR but does not reduce the complexity of the decoder to meet the preferred frame rate. By integrating the low computational complexity of the integer approximation technique with the high accuracy of the factorization technique, a compressed domain video decoding pipeline to support low rate video bitstreams can be obtained.

Two types of transformation matrices, namely the equations
Perfect pixel motion compensation TM shown in (11)_iAnd the equation (1
Half pixel motion compensation TM shown in 3)_hpiHave explained
Was. TM _iUsing an approximation matrix for
Compared to the case of using an 8 × 8 floating-point matrix, the calculation amount is
8% is enough. However, the half-pixel transformation matrix TM_hpiClose to
When using similar techniques directly, TM_hpiUse the approximate matrix of
And half-pixel motion compensation reduces PSNR (Table 8)
Cause visible distortion in the video decoded together
Was observed. There are two error sources. One of them is half
Pixel conversion matrixTM_hpiRespond more sensitively to approximation techniques
And In Table 8, TM_hpiIs TM_iBesides, from many sections
Is a composite matrix. The second one is first shown in FIGS. 6A and 6B.
As described above, the nonlinear processing at the time of half-pixel interpolation is
Combined with the error introduced by the approximation technique,
Accumulation of visible errors in the motion range from degrees to high
Let

The error problem can be solved by selectively applying the factorization technique to the half-pixel matrix. As discussed above, the factorization technique maintains floating point precision, thereby minimizing the errors described above. For example, the factorization technique reduces the matrix multiplication with TM _hpi to a sequence of equations similar to that shown in equations (25-45). These equations maintain 32-bit floating point precision, so no approximation error is introduced. Further, since the factorization technique decodes the DCT block into the spatial domain during motion compensation, the optimization described in FIG. 15 can be combined with the optimization just described. Table 5 demonstrates that the target frame rate of 15 fps can be achieved with the hybrid method, while Table 8 shows that the PNSR of the hybrid method achieves an acceptable PSNR.

[Table 8]

FIG. 16 is a flowchart of the operation of a method for performing reverse motion compensation in a compressed domain, according to one embodiment of the present invention. The method begins with operation 260 of receiving a video data frame in a compressed bitstream. In one embodiment, the bitstream is a low rate bitstream. For example,
The bitstream may be associated with a known video coding standard such as MPEG4, H.263, H.261. The method then proceeds to operation 262, where the blocks of the frame of the bitstream are decoded into a discrete cosine transform (DCT) domain representation. Where the video is
It is processed in the first two stages of the decoder, such as the decoder shown in FIGS. That is, the video data is processed in a variable length decoder stage and an anti-quantization stage to decode the compressed bit stream into a DCT domain representation. The DCT area representation is in a compressed format. The method then proceeds to operation 264, where data associated with the DCT domain representation is kept in a hybrid data structure. A suitable hybrid data structure is the hybrid data structure described in FIGS. In one embodiment, the hybrid data structure is, for example, a cellular telephone, a PD
A. Reduce the memory requirements of portable electronic devices such as web tablets, pocket personal computers, etc., having display screens for displaying video data.

Continuing with FIG. 16, the method proceeds to operation 266 where inverse motion compensation is performed on the data associated with the DCT domain representation in the compressed domain. Here, the hybrid factorization / integer approximation technique described in Tables 5 and 9 is selectively applied to inverse motion compensation. The method then proceeds to operation 268, where a hybrid factorization / integer approximation technique identifies a type of transform matrix associated with the block of video data being processed. In one embodiment, the type of the transform matrix is detected by the information in the bit set of the bit stream that is currently being decoded. If the transform matrix is a half-pixel matrix, the method proceeds to operation 270, where a factorization technique is used to decode the bitstream. In one embodiment, equations 25-
As explained at 45, the factorization technique reduces the matrix multiplication to a series of equations. That is, matrix multiplication is replaced by matrix permutation. Decision operation 2
If it is determined at 68 that the transform matrix is a complete pixel matrix, the method proceeds to operation 272, where an integer approximation technique is used to decode the bitstream. Here, matrix multiplication can be performed using basic integer arithmetic to solve the inverse motion compensation, as described earlier in Equations 46-58. Therefore, the hybrid factorization / integer approximation technique is selectively applied to achieve a frame rate that allows for the memory reduction achieved with the hybrid data structure described above while having acceptable image quality. As a result, processing in the compression area is executed.

FIG. 17 is a schematic diagram of the selective application of the hybrid factorization / integer approximation technique according to one embodiment of the present invention. Display screen 280 is configured to display an image defined by the low bit rate video. For example, when the display screen 280 is displayed on a PDA, a cellular phone, a pocket personal computer, a web tablet,
Can be associated with portable electronic devices. Ball 28
2 is moving vertically in the video. Block 284
Is located around the moving object and is considered an altitude or moderate motion area, and varies from frame to frame. Block 286 represents the background and remains substantially the same from frame to frame. Therefore,
Upon decoding of the compressed bitstream, block 284 of the data frame is frame-to-frame and associated with the advanced motion region, while block 286 remains substantially the same frame-to-frame. The block 284 associated with the advanced motion region requires a higher degree of accuracy during the compounding technique, i.e., factorization, but the block 286 is substantially unchanged and therefore requires less computational interpolation, i.e., an integer. Allowed by approximation. Therefore, the factorization technique is applied to the high to medium motion region block 284 and the integer approximation method is applied to the background block 286. As explained earlier, whether the block is associated with a high degree of motion, ie, whether to apply factorized half-pixel motion compensation, or whether the block is background data,
Information embedded in the bitstream is detected to determine whether to apply full pixel motion compensation by integer approximation. In one embodiment, the motion vectors shown in FIGS. 2, 6B, and 15 specify whether the motion compensation is half-pixel or full-pixel motion compensation.

The embodiment described above may be executed by software or hardware. One skilled in the art will appreciate that the decoder can be implemented as a semiconductor chip that includes logic gates configured to implement the functions described above. For example, to implement a video decoder in hardware, a hardware description language (HDL) such as VERILOG, for example, is used to synthesize the layout and firmware of the logic gates to achieve the necessary functions described in this document. Can be adopted.

FIG. 18 illustrates a hybrid factorization / integer approximation for efficiently decoding bitstream data while utilizing a hybrid data structure to minimize memory requirements, according to one embodiment of the present invention. 1 is a simplified schematic diagram of a portable electronic device having a decoder circuit configuration configured to apply the technique. Portable electronic device 2
90 is a central processing unit (CPU) 294, a memory 292,
A display screen 136, including a decoder circuitry 298, are all in communication with each other on a bus 296. Decoder circuitry 298 includes logic gates configured to provide the functions described above to reduce the amount of memory required to perform the video processing and inverse motion compensation in the compressed domain. It will be apparent to one skilled in the art that the decoder circuitry 298 may have memory on the chip containing the decoder circuitry, or the memory may be located off chip.

FIG. 19 is a block diagram of one embodiment of the present invention.
8 is a more detailed schematic diagram of the decoder circuit configuration of FIG. The incoming bit stream 122 is received by the variable length decoder (VLD) circuitry 300 of the decoder 298. Those skilled in the art will understand that the decoder circuit configuration 298 may be provided on a semiconductor chip disposed on a printed wiring board. VLD circuitry 300 provides a motion vector signal to motion compensation circuitry 306. The video processing memory 308 stores the anti-quantization circuit configuration 30 in the compression domain.
2 holds the internal representation of the video from DCT circuit configuration 3
04 is the internal DC of the video from the motion compensation circuitry 306
Maintain the T representation. Run-length decoding (RLD) circuit 31
The 0 and inverse discrete cosine transform (IDCT) circuitry 312
Decompress the video data so that it can be displayed on the display screen 136. The blocks of the circuit configuration described here provide functions similar to those of the blocks / stages described with reference to FIGS.

[0079]

In summary, the inventions described so far provide a compressed domain video decoder that reduces the amount of video memory and performs inverse motion compensation in the compressed domain. Non-zero DC of reference frame to define current frame
A memory reduction is achieved by a hybrid data structure configured to hold and manipulate T coefficients. The hybrid data structure includes a fixed size array having a fixed size block associated with each block of the video data frame. To accommodate non-zero coefficients beyond the capacity of the fixed size block, the hybrid data structure is provided with a variable size overflow vector. The amount of memory compression achieved by the compressed domain video decoder is up to twice that of the spatial domain video decoder. The reverse motion compensation of the compressed domain video decoder is optimized to achieve about 15-25 frames per second with video of acceptable quality. Hybrid blocks / integer approximation are selectively applied to the block being decoded. The criteria for determining which interpolation of the factorization / integer approximation technique to apply is based on a transformation matrix. That is, factorization is applied to a half-pixel matrix, whereas integer approximation is applied to a complete pixel matrix. Note that in one embodiment, the compression domain pipeline described in this document can be incorporated into an MPEG-4 simple profile video decoder.
Further, according to the embodiments, it is possible to pursue a wide variety of applications such as, for example, power scalable decoding of a battery-operated (CPU constrained type) device and integration of a video conference system.

In view of the above embodiments, it can be seen that the present invention can employ various computer-implemented operations involving data stored in computer systems. These operations include physical manipulations of physical quantities. Usually, but not always, these quantities are
Takes the form of electrical or magnetic signals that can be transformed, combined, compared, or otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing.

The invention described above can be implemented with other computer system configurations, such as handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention can be implemented in distributed computing environments where tasks are performed by remote processing devices that are linked via a communications network.

The present invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium can be any data storage device that holds data and that data can later be read by a computer system. Examples of computer readable media are hard drives, network attached storage (NA
S), read-only memory, random access memory, C
There are D-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical and non-optical data storage devices. The computer readable media can also be distributed in networked computer systems so that the computer readable code is retained and executed in a distributed fashion. The computer readable medium may be an electromagnetic carrier containing computer code.

While the invention has been described in more detail for a clearer understanding, it will be apparent that changes and modifications can be made within the scope of the appended claims. Accordingly, the embodiments of the present application are illustrative and not restrictive. Also, the present invention is not limited to the details described herein, but can be modified within the scope and equivalents of the claims. In the claims, elements and / or steps do not imply any particular order of operation, unless explicitly stated in the claims.

[Brief description of the drawings]

The invention will be more readily understood from the following detailed description taken in conjunction with the accompanying drawings. Similar structural elements are indicated by similar reference numerals.

FIG. 1 is a schematic diagram of a video decoder for decoding video data and performing motion compensation in the spatial domain.

FIG. 2 is a schematic diagram of a video decoder configured to perform inverse motion compensation in a compression domain, according to one embodiment of the present invention.

FIG. 3 is a schematic diagram illustrating inverse motion compensation performed in the spatial domain.

FIG. 4 is a graph illustrating peak signal-to-noise ratio (PSNR) of multiple frames to demonstrate the effectiveness of a forced update mechanism associated with the H.263 standard.

FIG. 5 is a schematic diagram illustrating determination of a half-pixel value in the H.263 standard.

FIG. 6A: Schematic diagram of a baseline space video decoder. B: Schematic diagram of a compressed domain video decoder according to one embodiment of the present invention.

FIG. 7 is a block diagram illustrating block conversion during a video encoding and decoding process according to one embodiment of the present invention.

FIG. 8 is a schematic diagram illustrating the use of a separate index to find the starting position of each 8 × 8 block in a run-length representation.

FIGS. 9A and 9B are diagrams for explaining classification and merging operations necessary for adding a prediction error to prediction in an array-based data structure and a list data structure, respectively.

FIG. 10 is a schematic diagram of a hybrid data structure including an array structure and a vector structure that enables memory compression and computational efficiency, according to one embodiment of the present invention.

FIGS. 11A, 11B and 11C are graphs illustrating factors evaluated in determining the capacity and overflow vector of fixed size blocks of a fixed size array of a hybrid data structure, according to one embodiment of the present invention.

FIG. 12 is a flowchart illustrating the operation of a method for reducing memory requirements for decoding a bitstream, according to one embodiment of the present invention.

FIG. 13 is a schematic diagram illustrating three examples of block alignment for reducing matrix multiplication.

FIG. 14 is a schematic diagram of half-pixel interpolation of a perfectly aligned DCT block.

FIG. 15 is a schematic diagram illustrating the reordering of functional blocks of a compressed domain video decoder to enhance processing of video data, according to one embodiment of the present invention.

FIG. 16 is a flowchart illustrating operations of a method for performing inverse motion compensation on a compressed domain, according to one embodiment of the invention.

FIG. 17 is a schematic diagram of the selective application of a hybrid factorization / integer approximation technique according to one embodiment of the present invention.

FIG. 18 applies a hybrid factorization / integer approximation technique to efficiently decode a bit data stream while utilizing a hybrid data structure to minimize memory requirements, according to one embodiment of the present invention. FIG. 1 is a simplified schematic diagram of a portable electronic device having a decoder circuit configuration configured to perform the following.

FIG. 19 is a more detailed schematic diagram of the decoder circuit configuration of FIG. 18, according to one embodiment of the present invention.

[Explanation of symbols]

100,120 decoder 102,122 bitstream 104,124 Variable decoder 106,132 run length decoder 108,126 anti-quantization 110,134 inverse discrete cosine transform 112,128 motion compensation 114,130 memory 116,136 display 160 half pixel interpolation

   ────────────────────────────────────────────────── ─── Continuation of front page    (72) Inventor Vasdev Baskaran             United States Sani, California             -Vale North Mahui Avenue             190 F term (reference) 5C059 KK08 MA00 MA05 MA23 MC11                       MC38 ME05 NN15 NN21 PP04                       SS10 SS20 TA61 TC00 UA05                       UA33                 5J064 AA03 BA09 BB05 BC01 BC02                       BC08 BC09 BC14 BC23 BD03

Claims (65)

[Claims] 1. A method for reducing the amount of memory required to decode a bitstream, comprising: receiving a video bitstream; decoding a frame of the bitstream into a transform domain representation; Assembling a hybrid data structure including a fixed size array and a variable size overflow vector, and inserting the non-zero coefficients of the transform domain representation into the hybrid data structure. how to. 2. The method of claim 1, wherein the video bitstream is a low-rate video bitstream. 3. The method of claim 1, wherein decoding the frames of the bitstream into a transform domain representation comprises processing the bitstream through a variable length decoder and an anti-quantization block. The described method. 4. The method of claim 1, wherein the fixed size array includes fixed size blocks. 5. The method of claim 4, wherein the fixed size block is configured to hold eight non-zero coefficients of the transform domain representation. 6. The operation of inserting non-zero coefficients of the transform domain representation into a hybrid data structure maps, for each block of a frame, coefficients in a fixed size array to corresponding coefficients in a variable size overflow vector. The method of claim 1, comprising: 7. A method for decoding video data, comprising: receiving a frame of video data in a compressed bitstream, decoding blocks of the frame into a transform domain representation in a compression domain, and defining a hybrid data structure. Holding data associated with the transform domain representation in the hybrid data structure, performing inverse motion compensation on the data associated with the transform domain representation in the compression domain, and performing inverse motion compensation on the data; A method comprising decompressing data for display. 8. The method of claim 7, wherein the hybrid data structure includes a fixed size array of fixed size blocks and a variable size overflow vector. 9. The operation of maintaining data associated with the transform domain representation in a hybrid data structure comprises: identifying non-zero coefficients of the transform domain representation; and storing the non-zero coefficients until the capacity of the fixed-size block is reached. Holding in a fixed size block of the fixed size array of the hybrid data structure, and after reaching the fixed size block capacity, holding non-zero coefficients exceeding the fixed size block capacity in the overflow vector. 7. The method according to claim 7, wherein
The method described in. 10. The method of claim 7, wherein the compressed bit stream is a low rate bit stream. 11. The operation of performing inverse motion compensation on data associated with the transformed domain representation in a compressed domain includes applying a hybrid factorization and integer approximation technique to data associated with the transformed domain representation. The method of claim 7, wherein: 12. A computer-readable medium having program instructions for reordering a low-rate bit stream for retention in a hybrid data structure, the method comprising identifying non-zero transform coefficients associated with an encoded block of a data frame. Program instructions for arranging the non-zero transform coefficients in a fixed size array, and a program for determining whether the quantity of the non-zero transform coefficients exceeds the capacity of the fixed size array Instructions for holding non-zero transform coefficients exceeding the capacity of the fixed size array in a variable size overflow vector; and for translating the non-zero transform coefficients from a compressed domain to a spatial domain. And computer readable program instructions. Deer. 13. The computer-readable medium of claim 12, wherein the fixed size array includes a plurality of fixed size blocks. 14. The computer-readable medium of claim 13, wherein the fixed-size blocks are each configured to hold eight non-zero transform coefficients. 15. The computer of claim 12, further comprising, for each block of a data frame, program instructions for mapping the coefficients of the fixed size array to corresponding coefficients of a variable size overflow vector. Readable media. 16. The computer-readable medium of claim 12, further comprising program instructions for performing inverse motion compensation on the non-zero transform coefficients using hybrid factorization and integer approximation techniques. 17. A circuit, comprising: a video decoder integrated circuit chip, wherein the video decoder integrated circuit chip receives a bit stream of data associated with a frame of video data.
A circuit configuration for decoding the bit stream of data into a transform domain representation; and a circuit for placing the non-zero transform coefficients of the transform domain representation into a hybrid data structure array in a memory associated with the video decoder. A circuit comprising: a configuration; and a circuit configuration for decompressing the non-zero transform coefficients of the transform domain representation for display. 18. The circuit according to claim 17, wherein said bit stream is an H.263 bit stream. 19. The circuit according to claim 17, wherein said memory is separate from said video decoder integrated circuit chip. 20. The circuit of claim 17, further comprising circuitry for performing inverse motion compensation by hybrid factorization and integer approximation techniques. 21. The circuit according to claim 17, wherein said memory is a static random access memory. 22. An apparatus configured to display a video image, comprising: a central processing unit (CPU); a random access memory (RAM); a display screen configured to display the image; A decoder circuit configured to convert the bit stream to a transform domain representation, wherein the decoder circuitry stores the non-zero transform coefficients of the transform domain representation in a memory associated with the decoder circuitry. The decoder circuit configuration includes a circuit configuration for selectively applying a hybrid factorization / integer approximation technique at the time of inverse motion compensation, further comprising: the CPU, the RAM, the display screen; And a bus in communication with the decoder circuit configuration. 23. The device according to claim 22, wherein the device is a portable electronic device. 24. The device of claim 23, wherein the portable electronic device is selected from the group consisting of a personal digital assistant, a cellular phone, a web tablet, and a pocket personal computer. 25. The apparatus of claim 22, wherein the hybrid data structure includes a fixed size array having a plurality of fixed size blocks and a variable size overflow vector. 26. The apparatus of claim 25, wherein the plurality of fixed size blocks are each configured to hold eight non-zero transform coefficients. 27. The apparatus of claim 26, wherein more than eight non-zero transform coefficients are retained in the variable size overflow vector. 28. The apparatus of claim 22, wherein the decoder circuitry includes an on-chip memory configured to hold data associated with the hybrid data structure. 29. A circuit for selectively applying a hybrid factorization / integer approximation technique during the inverse motion compensation, comprising: converting blocks of a frame of a video image into active motion and inactive motion. When associated with one, the circuit configuration for identification, and the factorization technique is applied to blocks associated with active motion areas, while integer approximation is applied to blocks associated with non-active motion areas. Apparatus for performing inverse motion compensation by applying a technique. 30. The apparatus of claim 22, wherein the video bit stream is a low-rate video bit stream. 31. A method for performing inverse memory compensation, comprising: receiving a video bitstream; identifying a transform matrix type selected from a group consisting of a half-pixel matrix and a full pixel matrix; If the type is a half-pixel matrix, apply a factorization technique to decode the bitstream corresponding to the half-pixel matrix, and if the transformation matrix type is a full-pixel matrix, a bitstream corresponding to the full-pixel matrix Applying an integer approximation technique to decode. 32. The method of claim 31, wherein the video bit stream is a low rate video bit stream. 33. The operation of applying a factorization technique to decode a bitstream corresponding to the half-pixel matrix includes factorizing the half-pixel matrix into columns of a sparse matrix, wherein the sparse matrix is The method of claim 31, comprising an order matrix and a diagonal matrix. 34. The operation of applying an integer approximation technique to decode a bit stream corresponding to the complete pixel matrix includes approximating each element of the complete pixel matrix with a binary number. The method according to claim 31. 35. The method of claim 34, wherein each element is rounded to the nearest square. 36. A method for decoding video data, comprising: receiving a frame of video data in a compressed bitstream, decoding the blocks of the frame into a transform domain representation in a compressed domain, Holding the associated data in a hybrid data structure, performing inverse motion compensation on the data associated with the transform domain representation in the compressed domain, wherein performing the inverse motion compensation comprises a portion of the video data frame. Determining a type of a transformation matrix associated with and applying hybrid factorization and integer approximation techniques to improve inverse motion compensation. 37. The compressed bit stream may be H.26
37. The method of claim 36, wherein the method is associated with a standard selected from the group consisting of: H.261, Motion Picture Expert Group. 38. The method of claim 36, wherein the hybrid data structure includes a fixed size array and a variable size overflow vector. 39. The method of claim 36, wherein the type of the transformation matrix is selected from a group consisting of a half pixel matrix and a full pixel matrix. 40. The method of claim 3, wherein the half-pixel matrix is associated with a high motion region of the image, while the full pixel matrix is associated with a minimum motion region of the image.
10. The method according to 9. 41. The operation of applying a hybrid factorization and integer approximation technique to improve the inverse motion compensation comprises applying a factorization technique to a matrix associated with a block corresponding to a high motion region of the frame; The method of claim 36, comprising applying an integer approximation technique to the remaining blocks. 42. The method of claim 36, wherein said compressed bit stream is a low rate bit stream. 43. A computer readable medium having program instructions for performing inverse motion compensation in a compressed domain, the program instructions for identifying a transformation matrix, wherein the transformation matrix is a half-pixel matrix and a full pixel matrix. A program instruction to determine if it is one of them, a program instruction to apply a factorization technique to decode the block of the bitstream corresponding to the half-pixel matrix, and a bit corresponding to the complete pixel matrix. Computer readable media comprising program instructions for applying an integer approximation technique for decoding blocks of the stream. 44. The computer-readable medium of claim 43, wherein the program instructions for performing the reverse motion compensation are executed in a compression domain. 45. The apparatus of claim 43, further comprising program instructions for extracting motion vector data, wherein the motion vector data identifies a transformation matrix as one of a half pixel matrix and a full pixel matrix. The computer readable medium of claim 1. 46. The computer-readable medium of claim 43, further comprising program instructions for arranging non-zero transform coefficients associated with the encoded data frame block into a hybrid data structure. 47. A program instruction for applying an integer approximation technique for decoding a block of a bit stream corresponding to the complete pixel matrix includes a program instruction for approximating each element of the complete pixel matrix with a binary number. 44. The computer readable medium of claim 43, wherein: 48. Program instructions for applying a factorization technique for decoding a block of a bitstream corresponding to the half-pixel matrix include program instructions for factorizing the half-pixel matrix into columns of a sparse matrix. 44. The computer-readable medium of claim 43, wherein the sparse matrix includes an order matrix and a diagonal matrix. 49. A circuit, comprising: an integrated circuit chip configured to decode video data, wherein the integrated circuit chip receives a bit stream of data associated with a frame of video data. A circuit configuration for decoding the bit stream of data into a transform domain representation; a circuit configuration for identifying a type of transform matrix; and a circuit configuration for performing inverse motion compensation by hybrid factorization and integer approximation techniques. A circuit comprising a circuit configuration. 50. The circuit of claim 49, wherein said integrated circuit chip further has a circuit configuration for arranging non-zero transform coefficients of said transform domain representation in a hybrid data structure. 51. The circuit according to claim 49, wherein said bit stream is a low rate bit stream. 52. A circuit configuration for performing inverse motion compensation by the hybrid factorization and integer approximation technique includes:
50. The circuit of claim 49, wherein the circuit is configured to apply a factorization technique to the half-pixel transformation matrix and apply an integer approximation technique to the full pixel transformation matrix. 53. The circuit of claim 49, further comprising a memory in communication with said integrated circuit chip. 54. The circuit of claim 49, wherein said factorization and integer approximation techniques are used on data in a compression domain. 55. A video decoder, comprising: a variable length decoder (VLD) configured to extract coefficient values and motion vector data from an incoming bit stream, and in communication with the variable length decoder. An anti-quantization block, wherein the anti-quantization block is configured to rescale the coefficient values, and has a downstream branch in communication with the anti-quantization block, wherein the downstream branch has an error An upstream branch in communication with the dequantized block, wherein the upper branch is configured to decode coefficients into a spatial domain, and the upper branch is configured to maintain an internal transform domain representation; Is configured to generate a spatial domain output that can be added to the decoded error coefficients to reconstruct a current block. Video decoder according to claim. 56. The video decoder according to claim 55, wherein said video decoder is implemented as software. 57. The video decoder according to claim 55, wherein said video decoder is implemented as hardware. 58. The video decoder according to claim 55, wherein said incoming bit stream is a low rate bit stream. 59. The video decoder according to claim 55, wherein the upstream branch includes a feedback loop, and the feedback loop includes a frame buffer, a motion compensation block, and a discrete cosine transform block. 60. The video decoder of claim 55, wherein the downstream branch includes a run length decoding block and an inverse compensation block. 61. The video decoder according to claim 55, wherein the inverse motion compensation operation is performed in a compression domain. 62. The method of claim 55, wherein the non-zero coefficients of the transform domain representation are arranged in a hybrid data structure in a memory associated with a video decoder to reduce memory requirements. Video decoder. 63. The video decoder of claim 62, wherein the hybrid data structure includes a fixed size array and a variable size overflow vector. 64. The inverse motion compensation includes hybrid factorization and integer approximation techniques.
2. The video decoder according to 1. 65. The hybrid factorization and integer approximation technique applies a factorization technique to a half-pixel transformation matrix,
65. The video decoder of claim 64, wherein the video decoder is configured to apply an integer approximation technique to the complete pixel transformation matrix.