JP2001527352A - Partial decoding of compressed video sequences - Google Patents

Abstract

SUMMARY A compressed video sequence, generated using a transform (eg, a DCT), is partially decoded (202) and converted to low-frequency transform coefficients (eg, a DCT) from an encoded bitstream. Coefficient only) is recovered (204). A block of image data is then generated using the low frequency coefficients, and the block of image data is subjected to motion compensated frame addition (206) to generate a partially decoded image. . The partially decoded image can be used for further processing, such as histogram analysis (208) for video analysis. The present invention still achieves satisfactory results while avoiding the need to perform computationally expensive inverse transformation operations.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001] The present invention relates to video processing, and more particularly to decoding of compressed video sequences.

[0002] This application claims the benefit of filing date priority of US Provisional Application No. 60 / 068,774, filed December 23, 1997.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT The United States Government has certain rights, at least in part, in the present invention in accordance with Government Contract No. MDA-904-95-C-3126.

2. Description of the Related Art A video image (or frame) in a digital video sequence is usually represented by an array of picture elements, or pixels, where each pixel is represented by one or more different elements. For example, in a monochrome density gradation image, each pixel is represented by an element having a value corresponding to the intensity of the pixel. In the RGB color format, each pixel is represented by a red component (R), a green component (G), and a blue component (B). Similarly YU
In the V color format, each pixel has an intensity (or luminance) component Y and two colors (
Or luminous intensity) elements U, V. In the 24-bit versions of these color formats, each pixel element is represented by an 8-bit value.

[0005] A typical video sequence consists of a series of consecutive frames called shots, where the frames of one shot correspond to the same basic scene. One shot is a continuous sequence of frames from one camera. At present, there is a great interest in analyzing digital video into its constituent shots. The growing availability of video material and the rapidly increasing need for indexing video databases has fueled interest in video analytics. Video analysis is also useful for video editing and compression.

One way to identify different shots in a digital video sequence is to
There is an analysis of the histogram corresponding to the video frame. The frame histogram represents the distribution of element values for one frame of video data. For example, a 32 bin histogram is generated for an 8-bit Y component of a video frame represented in a 24-bit YUV color format. Here the first b
in indicates how many pixels in the frame have a Y value of 0 to 7, and the second bin
Indicates how many pixels have a Y value between 8 and 15. Similarly, the last 32nd bin indicates how many pixels have Y values of 248 to 255.

[0007] Multidimensional histograms can also be generated based on combinations of bits from different elements. For example, a three-dimensional (3D) histogram can be generated for the YUV data using the four most significant bits (MSBs) of the Y component and the three MSBs of the U and V components. At this time, each bin in the 3D histogram corresponds to the number of pixels in a frame having the same 4 MBSs of Y, the same 3 MSBs of U, and the same 3 MSBs of V.

[0008] One of the general features of video sequences is that in certain histograms,
The histograms for frames in a given shot are usually more similar than the histograms for frames in shots corresponding to different scenes. Thus, one way to analyze digital video into its constituent shots is to look for dislocations between shots by comparing the histograms generated for the frames in the video sequence. In general, frames with similar histograms are more likely to correspond to the same scene than frames with different histograms.

[0009] Video sequences are typically stored and / or transmitted in compressed digital form. In compressed digital form, the original pixel data is processed to take advantage of the amount of redundancy that occurs both within and between video frames. There are many well-known algorithms for video data compression. Some of these algorithms employ a transform such as discrete cosine transform (DCT), which converts the pixel intensity data into transform coefficients in the spatial frequency domain. The DCT transform is applied directly to the pixel values or, if motion estimation is employed, the inter-frame corresponding to the difference between the pixel data in the current frame and the pixel data motion compensated from the reference frame. Apply to the pixel difference. The difference calculation between frames is performed together with or without motion estimation and motion correction. After applying the DCT transform to the appropriate pixel data, the resulting DCT coefficients are quantized and then run-length (RL) coded using, for example, a zigzag RL pattern. The RL encoded data is then optionally further encoded for storage and / or transmission as a compressed video data stream. The motion vectors derived during the motion estimation and used in the difference operation between the motion compensated frames are also encoded into the compressed video data stream. For multi-element video data such as RGB or YUV data, the planes of different pixel element values for each video frame are usually compressed separately.

When recovering a video sequence from a compressed video stream, the encoding process is reversed to restore a complete video image that can be displayed. For example, in the above-described compression algorithm, the compressed data is RL-decoded and dequantized as needed to recover uncoded DCT coefficients. Next, an inverse DCT transform is applied to the decoded DCT coefficients to recover pixel data. If the pixel data corresponds to the difference between the frames, an addition using the motion vector (decoded from the compressed video stream) is performed between the motion-compensated frames. As a result, a decoded pixel intensity value is generated for the current frame of the decoded video sequence.

One method of analyzing a compressed video sequence into its constituent shots is to completely restore the compressed video stream to fully recover the decoded video frames, and then perform a histogram analysis on the restored video sequence. Is to apply. This method is computationally expensive and slow. For example, the reason for this is that it takes an extremely long time to perform calculations such as inverse DCT.

Another technique for analyzing a compressed video sequence generated using a compression algorithm based on a transform, such as a DCT transform, is based on a low resolution decoding scheme. In this low-resolution decoding scheme, all DCT coefficients except for the lowest spatial frequency DCT coefficient in the compressed video stream (that is, the DC coefficient of DCT transform) are ignored, and only the DC coefficient is used for each frame of the compressed video stream. Generate a low-resolution decoded image. Here, each pixel of the low-resolution decoded image is a DC coefficient for a pixel block in a corresponding frame of the original video sequence. Thereafter, a histogram analysis is performed on the low resolution image.

FIGS. 1A and 1B show an original frame 102 in an original video sequence and a low-resolution frame 104 in a corresponding low-resolution decoded video sequence, respectively. Original frame 102 has 480 rows and 512 rows
Has columns of pixels. According to one possible compression algorithm, the original frame 102 is divided into (8 × 8) pixel blocks and the original frame 1
02 has an (8 × 8) block of 60 rows and 64 columns. Motion estimation is performed for each (8 × 8) block, and a motion vector is specified. The motion vector associates each (8 × 8) block with a corresponding (8 × 8) block in a reference frame (not shown). A difference operation between the motion-compensated frames is applied, and for each (8 × 8) block of the original frame 102,
A pixel difference block between (8 × 8) frames is generated. An (8 × 8) DCT transform is applied to each (8 × 8) inter-frame pixel difference block and the DCT coefficients (8 × 8
) Block generated. Here, the DC coefficient is usually located in the upper left corner. Since each (8 × 8) block of DCT coefficients is stored and / or transmitted as a compressed video stream, it is then quantized, run-length coded with motion vectors,
Alternatively, it is further encoded.

According to the low resolution decoding scheme, run-length decoding is applied to the compressed video data to analyze the compressed video stream (eg, to identify transitions between shots) to recover the quantized DCT coefficients. I do. If appropriate, the quantized DC coefficients from each set of DCT coefficients are dequantized and added between the motion compensated frames to the decoded DC coefficients (appropriately scaled by a factor of 8). Using the decoded motion vectors), generate the low resolution frame 104 of FIG. Here, each pixel of the low-resolution frame 104 corresponds to only the restored DC coefficient. Thus, each (8 × 8) block in the original frame 102 is represented by a single pixel in the low resolution frame 104, so that the low resolution frame 104 has 60 (ie, 480/8) rows and 64 (ie, 512/8) columns. Of pixels. Since the DC coefficient of the DCT transform is equal to the average intensity value of 64 pixels in the corresponding (8 × 8) pixel block, the low-resolution frame 104
2 is a low resolution approximation.

Next, histogram analysis is applied to the series of low resolution frames, and the compressed video sequence is analyzed into its constituent shots. This low-resolution decoding scheme avoids the computationally expensive inverse DCT processing, and thus reduces the complexity of the compressed video sequence compared to applying histogram analysis to the fully decoded image generated using the inverse DCT processing. Analysis is faster and cheaper. Unfortunately, the resolution of frames decoded using this conventional low resolution decoding scheme is very low, and without accurate analysis results, a positive error (i.e., a shift between shots that is not true in the original video sequence). And / or negative errors (i.e. miss the true dislocations in the original video sequence).

SUMMARY OF THE INVENTION The present invention is directed to a scheme for partially decoding a compressed video stream for applications such as video analysis. According to one embodiment, the compressed video stream is decoded to recover one or more low frequency transform coefficients for each original image data block. A block of low frequency image data is generated from each set of low frequency transform coefficients corresponding to each original image data block. A difference operation between the motion compensated frames is applied to each block of the low frequency image data to generate a partially decoded image for each frame of the compressed video stream.

DETAILED DESCRIPTION According to embodiments of the present invention, a partially decoded image is generated by partially decoding a transform-based compressed video stream, and then partially decoded. The following processing is applied to the resulting image, such as histogram analysis for video analysis. Partially decoded images are generated by forming blocks using only the decoded DC transform coefficients obtained from the compressed video stream.

FIG. 2 shows a flowchart of a process according to one embodiment of the present invention. The compressed video stream is partially decoded to recover the DC coefficients of the encoded transform coefficients (step 202 in FIG. 2). For example, if the compressed video stream is (8 × 8
3.) If encoded using a DCT transform, the decoding (eg, RL decoding or unquantization) of the compressed video stream in step 202 corresponds to each (8 × 8) pixel block in the original video sequence. Decoding DC means decoding the compressed video stream only enough to recover the DCT coefficients from the bit stream.

A block of image data is then generated by duplicating the corresponding DC coefficient for each pixel of the block, using only DC coefficients (step 204).
. In one embodiment, each block is a sub-block, where the sub-block is smaller than the corresponding area of the original video sequence used to generate the transform coefficients. For example, if the DCT transform is applied to (8 × 8) blocks of the original video sequence, the sub-block generated in step 204 will be (2 × 2) or (4 × 4) (although other sizes may be used). Available). Alternatively, the block of duplicated DC coefficients can take the same size (eg, 8 × 8) as the transform.

If appropriate, the addition between the motion compensated frames is then performed to produce a partially decoded image (step 206). Thereafter, the partially decoded image is subjected to additional processing such as histogram analysis for video analysis (step 208). If the block of duplicate DC coefficients is smaller than the original transform size, the decoded motion vectors used in the addition between motion compensated frames must be scaled accordingly. (4x4)
One advantage of using (2 × 2) sub-blocks is that motion vectors can be scaled by a factor of 2 or 4, respectively, by simply shifting bits without performing division.

FIGS. 3A and 3B illustrate an original frame 302 in an original video sequence and a partially decoded version of a corresponding partially decoded video sequence, according to an implementation of the process of FIG. Each of the frames 304 is shown. The original frame 302 is a (480 × 512) frame like the frame 102 in FIG. The partially decoded image 304 has a (4 ×
4) It is generated from a block and therefore has 240 rows and 256 columns of pixels.

FIG. 3C shows a (4 × 4) block 306 of duplicated DC coefficients representing sub-blocks used to generate the partially decoded image 304 of FIG. 3B.
Is shown. For a sub-block with duplicated DC coefficients, each pixel contains the same information, but the addition between the motion compensated frames is performed using appropriately scaled decoded motion vectors. , The corresponding sub-blocks in the resulting partially decoded image usually do not contain duplicates of the same data. This is because most motion vectors have elements other than integral multiples of four.

The present inventors believe that this partial decoding scheme allows for better video analysis of the compressed video stream than does the prior art low resolution decoding scheme (ie, positive and / or negative errors are eliminated). Less). Although some additional computational load is required, such as the addition between motion-compensated frames for larger decoded images, the partial decoding scheme of the present invention is less than the prior art low resolution decoding scheme. They share the advantage of avoiding the inverse transformation. Thus, depending on processing limitations, the present invention can provide better results in that it can be made more affordable even if the computational cost increases.

FIGS. 4A to 4D show P for four different DC element block sizes.
For a test sequence of 1500 frames encoded using the x64 video compression scheme, the histogram differences between the frames are plotted against the number of frames and represented in a graphical representation (Y-axis scale is arbitrary). . FIG. 4A corresponds to the processing of the prior art, where a single DC value (ie, (1)
Each (8 × 8) pixel block is represented by (× 1) block). FIG. 4 (B
) Corresponds to the process according to the invention, wherein each (8 × 8) pixel block is represented by a (2 × 2) block of duplicated DC values in a partially decoded image. Similarly, FIGS. 4 (C) and 4 (D) correspond to the process according to the invention, in which (4 × 4) of the duplicated DC values in the partially decoded image, respectively.
Each (8 × 8) pixel block is represented by a block and an (8 × 8) block. High peaks in these figures indicate scene changes in the video sequence.

FIGS. 4A-4D show that as the number of blocks of duplicated DC values increases from (1 × 1) to (8 × 8), the background noise level decreases as a result. It is shown. This is one of the advantages of the present invention over the conventional low-resolution system shown in FIG.

Although the present invention has been described in the context of a two-dimensional (8 × 8) DCT transform, those skilled in the art will appreciate that the present invention can be applied to two-dimensional DCT transforms of sizes other than (8 × 8) or one-dimensional D
It will be appreciated that other DCT transforms, such as a CT transform, or other transforms, such as a one-dimensional or second-order original Slant transform or a Haar transform, can be used as well. Similarly, the present invention can be applied to motion estimation and motion correction (
Although we have described the situation where it is performed on 8 × 8) pixel data blocks, motion analysis can be performed on blocks of other sizes,
It will be appreciated that these other sizes may be different from the transform size. For example, in a general video compression method, motion estimation and motion correction are performed on a (16 × 16) pixel data block even if the transform is an (8 × 8) DCT transform.

Further, while the present invention has been described in terms of generating a partially decoded image using only DC transform coefficients, those skilled in the art will appreciate that, as an alternative, two or more low frequency It will be appreciated that transform coefficients (including DC coefficients) may be used to generate the partially decoded image.

Similarly, while the present invention has been described in the context of performing video analysis based on histogram analysis, those skilled in the art will recognize that the present invention may be used in other applications where low resolution is acceptable,
For example, generation of images in images, fast forward playback of video sequences, target recognition,
It can also be used for motion detection and the like.

The present invention can be embodied in the form of a method and an apparatus for performing these methods. The present invention can also be embodied in the form of program code embodied in a specific medium, such as a floppy disk, CD-ROM, hard disk drive, or other device-readable storage medium. At this time, if the program code is loaded on a machine such as a computer and executed by the machine,
A machine is an apparatus for performing the present invention. The invention can also be embodied in the form of program code, said program code being loaded on a machine and / or stored on a storage medium as executed by the machine, or It can be transmitted through any transmission medium, such as electrical wiring, cables, optical fibers, electromagnetic radiation, and the like. At this time, when the program code is loaded on a machine such as a computer and executed by the machine, the machine becomes an apparatus for performing the present invention. When executed on a general-purpose processor, the program code segments combine with the processor to provide a unique device that performs similar operations as certain logic circuits.

Further, those skilled in the art will appreciate the details described and illustrated, to illustrate the essence of the invention;
It will be understood that various changes may be made in the materials and arrangement of parts without departing from the principles and scope of the invention as set forth in the following claims.

[Brief description of the drawings]

Other aspects, features, and advantages of the invention are set forth in the description which follows, the appended claims,
And will be more fully apparent from the following figures.

1A and 1B show an original frame in an original video sequence and a low-resolution frame in a corresponding low-resolution decoded video sequence, respectively.

FIG. 2 illustrates a process flow diagram according to one embodiment of the present invention.

FIGS. 3A and 3B show an original frame in an original video sequence and a partially decoded frame in a corresponding partially decoded video sequence, respectively, according to the process of FIG. 2; (C) is a duplicated DC representing a sub-block used to generate the partially decoded image of FIG. 3 (B).
Shows a (4 × 4) block of coefficients.

4A-4D show histogram differences between frames for a 1500 frame test sequence encoded using the Px64 video compression scheme for four different DC element block sizes. Is shown (a scale on the Y axis is arbitrary).

──────────────────────────────────────────────────続 き Continuation of front page (81) Designated country EP (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE ), OA (BF, BJ, CF, CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG), AP (GH, GM, KE, LS, MW, SD, SZ, UG, ZW), EA (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), AL, AM, AT, AU, AZ, BA, BB, BG, BR, BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, FI, GB, GE, GH, GM, HR, HU, ID, IL, IS, JP, KE, KG, KP , KR, KZ, LC, LK, LR, LS, LT, LU, LV, MD, MG, MK, MN, MW, MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, TR, TT, UA, UG, UZ, VN, YU, ZWF terms (reference) 5C059 KK01 KK40 MA23 PP04 RF09 SS20 TA76 TB02 TC04 TD10 TD11

Claims (10)

[Claims] (A) decoding a compressed video stream to recover one or more low frequency transform coefficients for each block of original image data; and (b) corresponding to each block of original image data. Generating a block of low-frequency image data from each set of low-frequency transform coefficients; and (c) applying a difference operation between motion-compensated frames to each block of low-frequency image data. Generating a partially decoded image for each frame of the video stream. A method for partially decoding a transform-based compressed video stream. 2. The invention of claim 1, further comprising applying a histogram analysis to the partially decoded image to analyze the compressed video stream. 3. The invention according to claim 1, wherein each block of the low-frequency image data is smaller than the transform size. 4. The invention according to claim 1, wherein the transform is a discrete cosine transform (DCT). 5. The method according to claim 4, wherein the transform is an (8 × 8) DCT transform, and each block of the low-frequency image data is either (2 × 2) or (4 × 4). The described invention. 6. The method of claim 1, wherein each block of the low frequency image data is smaller than a size of the transform, and further comprising applying a histogram analysis to the partially decoded image to analyze the compressed video stream. Item 6. The invention according to Item 5. 7. Step (a) comprises decoding the compressed video stream to recover only DC transform coefficients for each block of original image data, and step (b) comprises: The invention of claim 1, further comprising the step of generating a block of image data. 8. The invention of claim 7, wherein each block of low frequency image data is generated by duplicating a corresponding DC transform coefficient. 9. A means for decoding the compressed video stream to recover one or more low frequency transform coefficients for each block of the original image data; and (b) corresponding to each block of the original image data. Means for generating blocks of low-frequency image data from each set of low-frequency transform coefficients to be applied; and (c) means for applying a difference operation between motion-compensated frames to each block of low-frequency image data. Means for generating a partially decoded image for each frame of the compressed video stream, wherein the apparatus partially decodes the transform-based compressed video stream. 10. A computer readable medium having stored thereon a plurality of instructions, the instructions comprising, when executed by a processor, a method for partially decoding a conversion-based compressed video stream. Instructions for causing the processor to execute, the means comprising: (a) decoding the compressed video stream and recovering one or more low frequency transform coefficients for each block of the original image data; Generating a block of low-frequency image data from each set of low-frequency transform coefficients corresponding to each block of image data; and (c) a difference between frames whose motion has been corrected for each block of low-frequency image data. Applying an operation, wherein for each frame of the compressed video stream the partially decoded image Computer-readable medium, comprising the steps of: generating a.