JP2004266794A - Multi-resolution video coding and decoding - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide diversified techniques and tools for multi-resolution video coding and decoding. <P>SOLUTION: A video encoder performs multi-resolution video coding. For example, the encoder adaptively changes frame sizes to reduce blocking artifacts at low bit-rates. A video decoder performs corresponding multi-resolution decoding. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

  The present invention relates to multi-resolution video coding and decoding. For example, a video encoder adaptively changes the video frame size to reduce blocking artifacts at low bit rates.

  Digital video consumes (exhausts) large amounts of storage and transmission capacity. A typical raw digital video sequence contains 15 or 30 frames per second. Each frame may contain tens or hundreds of thousands of pixels (also called pels). Each pixel represents a very small element of the image. In raw form, computers typically represent pixels in 24 bits. Thus, the bits per second or bit rate of a typical raw digital video sequence can be 5 million bits / second or more.

  Most computers and computer networks lack the resources to process live digital video. For this reason, engineers use compression (also called coding or encoding) to reduce the bit rate of digital video. The compression can be lossless, in which the quality of the video is not degraded, but the bitrate reduction is limited by the complexity of the video. Alternatively, the compression can be lossy, in which the quality of the video is poor, but the bitrate reduction is more dramatic relative to the result in lossless compression. Decompression reverses compression.

  In general, video compression techniques include intra-frame (intra-frame) compression and inter-frame (inter-frame) compression. Intra-frame compression techniques compress individual frames, which are commonly referred to as I-frames, key frames, or reference frames. Inter-frame compression techniques compress a frame with reference to a preceding and / or subsequent frame, usually referred to as a predicted frame, a P frame or a B frame.

  Many intra-frame and inter-frame compression techniques are block-based. A video frame is divided into blocks for encoding. For example, an I frame is divided into 8 × 8 blocks, and these blocks are compressed. Alternatively, the P frame is divided into 16 × 16 macroblocks (eg, four 8 × 8 luminance blocks and two 8 × 8 chrominance blocks) and these macroblocks are compressed. Different implementations can use different block configurations.

US Pat. No. 6,499,060 US Patent No. 6,418,166

  Standard video encoders suffer a dramatic performance degradation when the target rate falls below a certain threshold. In block-based video compression and decompression, quantization and other irreversible processing stages generally result in blocking artifacts, i.e., distortions that appear as perceptible discontinuities between blocks. At low bit rates, the high frequency information for a block of an I-frame can be severely distorted or completely lost. Similarly, high frequency information about the residuals of a block of P-frames (portions of P-frames not predicted by motion estimation or other predictions) can be severely distorted or completely lost. As a result, significant blocking artifacts can occur in the "low-pass" region, causing a significant decrease in the quality of the reconstructed video.

  Some previous encoders have attempted to reduce the perceptibility of blocking artifacts by processing the reconstructed frames with a deblocking filter. The deblocking filter smooths boundaries between the blocks. Although a deblocking filter can improve perceived video quality, it has several disadvantages. For example, smoothing is performed only on the reconstructed output at the decoder. Thus, even when in-loop deblocking is in use, the effects of deblocking cannot be factored into the motion estimation, motion compensation or transform coding process for the current frame. On the other hand, the smoothing (ie, out-of-loop deblocking) of the current frame by the post-processing filter can be too extreme, and the smoothing process introduces unnecessary computational complexity.

  Given the critical importance of video compression and decompression to digital video, it is not surprising that video compression and decompression are well-developed areas. However, whatever the advantages of previous video compression and decompression techniques, they do not have the advantages of the following techniques and tools.

  In summary, the detailed description is directed to various techniques and tools for multi-resolution video coding. For example, a video encoder adaptively changes the video frame size to reduce blocking artifacts at low bit trades. In doing so, the encoder reduces blocking artifacts, but may increase blurring, which is less perceptible and less unpleasant than blocking artifacts. Various techniques and tools can be used in combination or independently.

  In one aspect, a video encoder encodes video in any of a number of spatial resolutions. The encoder encodes at least one frame in the sequence of multiple video frames with a first spatial resolution and encodes at least one other frame with a second spatial resolution. The second spatial resolution is different from the first spatial resolution and the encoder selects the second spatial resolution from a set of multiple spatial resolutions to reduce blocking artifacts in the sequence of video frames. I do.

  In other aspects, the encoder encodes a first portion of the frame with a first spatial resolution and encodes a second portion of the frame with a second spatial resolution. The second spatial resolution is different from the first spatial resolution.

  In other aspects, the video encoder includes, in a bitstream, a first code indicating a first spatial resolution for a first frame encoded with the first spatial resolution, the second spatial resolution Include in the bitstream a second code indicating a second spatial resolution for the second frame encoded in. The second spatial resolution is different from the first spatial resolution and the encoder selects the second spatial resolution from a set of multiple spatial resolutions to reduce blocking artifacts in the sequence of video frames. I do.

  In other aspects, the encoder includes a first signal in the bitstream that indicates a first spatial resolution for a first portion of the frame, and a second signal that indicates a second spatial resolution for a second portion of the frame. 2 in the bitstream. The second spatial resolution is different from the first spatial resolution.

  In another aspect, the decoder receives the multi-resolution signal in a sequence header for a video sequence of multiple encoded frames. The multi-resolution signal indicates whether multiple frames are encoded with multiple spatial resolutions. If the multiple frames are encoded with multiple spatial resolutions, the decoder decodes the first encoded frame with the first spatial resolution and converts the second encoded frame into a second spatial resolution. Decode with spatial resolution.

  In other aspects, the decoder decodes a first portion of the encoded frame with a first spatial resolution and decodes a second portion of the encoded frame with a second spatial resolution. I do. The second spatial resolution is different from the first spatial resolution.

  In another aspect, an encoder or decoder receives pixel data for a video image and adaptively changes the spatial resolution of the video image, which converts the resampled pixel data. , 6 tap down sampling filter or 10 tap up sampling filter.

  Additional features and advantages will be apparent from the following detailed description of various embodiments, which proceeds with reference to the accompanying figures.

  Detailed embodiments of the present invention are directed to multi-resolution video encoding and decoding. For example, a video encoder adaptively changes the video frame size to reduce blocking artifacts at low bit trades. In doing so, the encoder reduces blocking artifacts but may increase blurring, which is less perceptible and less undesirable than blocking artifacts.

  In some embodiments, the encoder encodes the input frames with different spatial resolutions using multi-resolution encoding techniques and tools. For example, in one implementation, the encoder may encode the frame with the largest original resolution, encode with a half downsampled resolution in the horizontal direction, or encode in half the vertical direction. Or down-sampled resolution in the horizontal and vertical directions. Alternatively, the encoder may reduce or increase the resolution of frames encoded by some other factor relative to the original resolution, or may encode the frame by a factor relative to the current resolution. Reduce or increase the resolution of the rendered frame, or set the resolution using some other technique. The decoder decodes the encoded frame using a corresponding technique.

  In some embodiments, the encoder selects a spatial resolution for a frame on a frame-by-frame basis or some other basis. The decoder performs the corresponding adjustment.

  In some embodiments, the encoder selects a spatial resolution by evaluating certain criteria (eg, bit rate, frame content, etc.).

  Various techniques and tools can be used in combination or independently. Particular embodiments implement one or more of the described techniques and tools. Different techniques and tools can be combined, used independently or with other techniques and tools.

I. Computing Environment FIG. 1 illustrates a general-purpose example of a suitable computing environment (100) in which the described embodiments can be implemented. The computing environment (100) is not intended to imply any limitation as to the scope of use or functionality of the invention, since the invention may be implemented in a variety of general purpose or special purpose computing environments. It is.

  Referring to FIG. 1, the computing environment (100) includes at least one processing unit (110) and memory (120). In FIG. 1, this most basic configuration (130) is included within the dashed line. The processing unit (110) executes computer-executable instructions and may be a real or virtual processor. In a multi-processing system, the multi-processing unit executes computer-executable instructions to increase processing power. The memory (120) can be volatile memory (eg, registers, cache, RAM), non-volatile memory (eg, ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory (120) stores software (180) implementing multi-resolution encoding and / or decoding techniques.

  A computing environment may have additional features. For example, the computing environment (100) includes storage (140), one or more input devices (150), one or more output devices (160), and one or more communication connections (170). An interconnection mechanism (not shown), such as a bus, controller, or network, interconnects the components of the computing environment (100). Typically, operating system software (not shown) provides an operating environment for other software executing in the computing environment (100) and coordinates the activities of the components of the computing environment (100).

  The storage (140) can be removable or non-removable, including a magnetic disk, magnetic tape or cassette, CD-ROM, CD-RW, DVD, or used to store information, Any other media that can be accessed within the naming environment (100) is included. The storage (140) stores instructions for software (180) that implements multi-resolution encoding and / or decoding techniques.

  The input device (150) may be a keyboard, a mouse, a touch input device such as a pen or a trackball, a voice input device, a scanning device, a network adapter, or another device that provides input to the computing environment (100). Can be For video, the input device (150) may be a TV tuner card, camera video interface, or similar device that accepts video input in analog or digital form, or a CD-ROM / that provides video input to the computing environment. It can be a DVD reader. The output device (160) can be a display, a printer, a speaker, a CD / DVD writer, a network adapter, or another device that provides output from the computing environment (100).

  The communication connection (170) allows communication over one communication medium to another computing entity. Communication media conveys information such as computer-executable instructions, compressed video information, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired or wireless techniques implemented in electrical, optical, RF, infrared, acoustic, or other carriers.

  The invention can be described in the general context of computer readable media. Computer readable media are any available media that can be accessed within a computing environment. By way of example, and not limitation, computer readable media within computing environment (100) includes memory (120), storage (140), communication media, and combinations of any of the above.

  The invention can be described in the general context of computer-executable instructions contained in program modules that execute within a computing environment on a real or virtual processor of interest. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The functionality of a program module may be combined or divided among multiple program modules as desired in various embodiments. Computer-executable instructions for a program module can execute in a local or distributed computing environment.

  For the sake of presentation, the detailed description will use the terms “set,” “choose,” “encode,” and “decode,” Computer operations in a virtual environment. These terms are a high-level abstraction of operations performed by a computer, and should not be confused with acts performed by humans. The actual computer operation corresponding to these terms will vary depending on the implementation.

II. Examples of Video Encoders and Decoders The techniques and tools in various embodiments may be implemented in video encoders and / or decoders. Video encoders and decoders can include different modules within them, and multiple different modules can be associated with and communicate with each other in many different ways. The modules and relationships described below are exemplary.

  Depending on the implementation and the type of compression desired, encoder or decoder modules can be added, omitted, divided into multiple modules, combined with other modules, and / or replaced with similar modules. . In alternative embodiments, an encoder or decoder having a different module and / or configuration of other modules performs one or more of the described techniques.

  The exemplary encoder and decoder are block-based and use a 4: 2: 0 macroblock format, where each macroblock is composed of four 8 × 8 luminance blocks (sometimes one 16 × 16 macroblock). And two 2x8 chrominance blocks. Alternatively, the encoder and decoder are object-based and use different macroblocks or block formats, or in sets of pixels of different sizes or configurations than 8x8 blocks and 16x16 macroblocks Perform the operation.

A. Video Encoder Embodiment The encoder receives a sequence of video frames, including the current frame, and produces as output compressed video information. The encoder compresses the predicted frame and the key frame. Many of the components of the encoder are used to compress both key and predicted frames. The exact operations performed by these components can vary depending on the type of information being compressed.

  Predicted frames (also called P frames, B frames in bidirectional prediction, or inter-coded frames) are expressed in terms of prediction (or difference) from one or more other frames Is done. The prediction residual is the difference between what was predicted and the original frame. In contrast, key frames (I-frames, also called intra-coded frames) are compressed independently of other frames.

  Referring to FIG. 2 in some embodiments, an encoder (200) that encodes a current frame (205) includes a resolution transformer (210) for multi-resolution encoding. The resolution converter (210) receives the current frame (205) as input and outputs multi-resolution parameters (215) as well as the converted frame. If the current frame (205) is a predicted frame, the resolution transformer (230) receives as input a reference frame (225) for the current frame (205) and outputs a transformed reference frame. .

  Resolution converters (210) and (230) communicate with other encoding modules (240), which were provided by resolution converters (210) and (230). An output (245) (eg, pixel block data, motion vector, residual, etc.) is generated based in part on the multi-resolution coding information (eg, multi-resolution parameters (215)).

  Other encoder modules (240) may include, for example, a motion estimator, a motion compensator, a frequency transformer, a quantizer, a frame store, and an entropy encoder.

  If the current frame (205) is a forward predicted frame, the motion estimator estimates the motion of the current frame (205) macroblock or other set of pixels with respect to the reference frame (225). The reference frame in this case is the previous reconstructed frame buffered in the frame store. In an alternative embodiment, the reference frame (225) is a later frame, or the current frame (205) is bidirectionally predicted. The motion compensator applies the motion information to the reconstructed previous frame to form a motion compensated current frame. However, the prediction is rarely perfect, and the difference between the motion compensated current frame and the original current frame (205) is the prediction residual. Alternatively, the motion estimator and motion compensator apply another type of motion estimation / compensation.

  The frequency converter converts spatial domain video information into frequency domain (ie, spectrum) data. For block-based video frames, the frequency transformer applies a discrete cosine transform (DCT) to a block of pixel data or prediction residual data to generate a block of DCT coefficients. Alternatively, the frequency transformer applies another conventional frequency transform, such as a Fourier transform, or uses wavelet or sub-band analysis.

  The quantizer then quantizes the block of spectral data coefficients. The quantizer applies uniform scalar quantization to the spectral data using a step size that changes on a frame-by-frame basis or on another basis. Alternatively, the quantizer may apply another type of quantization to the spectral data coefficients, such as non-uniform, vector, or non-adaptive quantization, or may apply spatial quantization in encoder systems that do not use frequency transform. Quantize or directly quantize region data. In addition to adaptive quantization, the encoder (200) may use frame dropping, adaptive filtering, or other techniques for rate control.

  When the reconstructed current frame is needed for subsequent motion estimation / compensation, the module of the encoder (200) reconstructs the current frame (205), which typically encodes the frame. Performs the reverse of the technique used to The frame store buffers the reconstructed current frame for use in predicting the next frame.

  The entropy encoder compresses the output of the quantizer as well as certain side information (eg, motion information, quantization step size, etc.). Common entropy coding techniques include arithmetic coding, differential coding, Huffman coding, run length coding, LZ coding, dictionary coding, and combinations or variations of the above. Entropy encoders typically use different coding techniques for different types of information and can select among a number of code tables within a particular coding technique.

  For additional details on other encoder modules (240) in some embodiments, see US patent application Ser. No. 09 / 849,502 entitled “DYNAMIC FILTERING FOR LOSSY COMPRESSION” filed May 3, 2001. See U.S. patent application Ser. No. 09 / 201,278, U.S. Pat. Each of these disclosures is incorporated herein by reference.

B. Video Decoder Embodiment Referring to FIG. 3, a decoder (300) receives information about a sequence of compressed video frames and generates an output including the reconstructed frames. The decoder (300) decompresses the predicted frame and the key frame. Many of the components of the decoder (300) are used to decompress both key frames and predicted frames. The exact operations performed by these components can vary depending on the type of information being compressed.

  In some embodiments, the decoder (300) for reconstructing the current frame (305) includes a resolution transformer (310) for multi-resolution decoding. The resolution transformer (310) takes as input the decoded frame (315) and outputs a reconstructed current frame (305).

  If the current frame (305) is a predicted frame, the resolution transformer (330) receives as input the multi-resolution parameter (315) and a reference frame (325) for the current frame (305). I do. The resolution converter (330) outputs the reference frame information to another decoder module (340). Another decoder module (340) uses the reference frame information with the motion vectors, residuals, etc. (345) received from the encoder to decode the current frame (305).

  Other decoder modules 340 may include, for example, buffers, entropy decoders, motion compensators, frame stores, inverse quantizers, and inverse frequency transformers.

  The buffer receives information (345) about the compressed video sequence and makes the received information available to an entropy decoder. Buffers typically receive information at a rate that is fairly constant over time and include a jitter buffer to smooth short-term variations in bandwidth or transmission. Buffers can also include playback buffers and other buffers. Alternatively, the buffer receives information at a varying rate.

  An entropy decoder decodes the entropy coded quantized data, as well as the entropy coded side information (eg, motion information, quantization step size, etc.), and typically performs Apply the inverse of encoding. Entropy decoders often use different decoding techniques for different types of information and can select among multiple code tables within a particular decoding technique.

  If the frame being reconstructed is a forward predicted frame, the motion compensator applies the motion information to the reference frame to form a prediction of the frame being reconstructed. For example, the motion compensator uses a macroblock motion vector to find a macroblock in a reference frame. The frame buffer stores the previous reconstructed frame for use as a reference frame. Alternatively, the motion compensator applies another type of motion compensation. The prediction by the motion compensator is rarely perfect, so the decoder also reconstructs the prediction residual.

  When the decoder needs the reconstructed frame for subsequent motion compensation, the frame store buffers the reconstructed frame for use in predicting the next frame.

  The inverse quantizer inversely quantizes the entropy-decoded data. In general, inverse quantizers apply uniform scalar inverse quantization to entropy-decoded data using a step size that changes on a frame-by-frame or other basis. Alternatively, the inverse quantizer applies another type of inverse quantization to the data, such as non-uniform, vector, or non-adaptive inverse quantization, or does not use the spatial frequency data with an inverse frequency transform. Dequantize or directly in the decoder system.

  The inverse frequency transformer converts the quantized frequency domain data into spatial domain video information. For block-based video frames, the inverse frequency transformer applies inverse DCT (IDCT) to the block of DCT coefficients to generate pixel data or prediction residual data for a key frame or prediction frame, respectively. . Alternatively, the frequency transformer applies another conventional inverse frequency transform, such as a Fourier transform, or uses wavelets or sub-band synthesis.

  For additional details on other decoder modules (340) in some embodiments, see US Patent Application Serial No. 09 / 849,502, filed May 3, 2001, entitled "DYNAMIC FILTERING FOR LOSSY COMPRESSION. See U.S. patent application Ser. No. 09 / 201,278, U.S. Pat.

III. Multi-Resolution Video Coding and Decoding In multi-resolution coding, an encoder encodes input frames with different spatial resolutions. The encoder selects the spatial resolution for the frame on a frame-by-frame or other basis. In some embodiments, the encoder selects a spatial resolution based on the following observations.
1. As the bit rate decreases, the advantages in coding at lower spatial resolutions increase.
2. As the quantization step size increases, the advantage in coding with lower spatial resolution increases.
3. Because downsampling discards high frequency information, downsampling is sometimes less suitable for frames with perceptually significant high frequency content (eg, "strong edges", text, etc.).
4. If the frame has low frequency characteristics, or if the frame has high frequency content such as noise, downsampling may be appropriate.

  In some embodiments, the encoder uses the bit rate of the current frame, the quantizer step size, and the orientation / magnitude for high frequency energy to select a spatial resolution. For example, if the magnitude of the horizontal high frequency component of the current frame is large but the magnitude of the vertical high frequency component is small, the encoder selects vertical down sampling. In another embodiment, the encoder uses information from the reference frame (instead of or in combination with information from the current frame) to select a spatial resolution. Alternatively, the encoder may omit some or all of the above criteria, use other criteria instead of some of the above criteria, or select spatial resolution using additional criteria can do.

  After the encoder has selected a spatial resolution for the current frame, the encoder resamples the original frame to the desired resolution before encoding it. If the current frame is a predicted frame, the encoder also resamples the reference frame for the predicted frame to match the new resolution of the current frame. The encoder then sends the resolution selection to the decoder. In one implementation, a 6 tap filter is used for downsampling and a 10 tap filter is used for upsampling, which combine to enhance the quality of the reconstructed video Designed to be. Alternatively, other filters are used.

  FIG. 4 shows a technique (400) for multi-resolution encoding of a frame. An encoder, such as the encoder (200) of FIG. 2, sets the resolution for the frame (410). For example, the encoder considers the criteria listed above or other criteria.

  The encoder then encodes frame (420) with the resolution. If the encoding is complete (430), the encoder ends. If not, the encoder sets the resolution for the next frame (410) and continues encoding. Alternatively, the encoder sets the resolution to some level other than the frame level.

  In some embodiments, the encoder encodes predicted frames as well as intra frames. FIG. 6 shows a technique (600) for multi-resolution encoding of intra and predicted frames.

  First, the encoder checks whether the current frame to be encoded is an I frame or a P frame (610). If the current frame is an I-frame, the encoder sets a resolution for the current frame (620). If the frame is a P-frame, the encoder sets the resolution for the reference frame (630) before setting the resolution for the current frame (620).

  After setting the resolution for the current frame (620), the encoder encodes the current frame with the resolution (640). If the encoding is complete (650), the encoder ends. Otherwise, the encoder continues encoding.

  In some implementations, the encoder selectively encodes the frame with one of the following resolutions. That is, 1) the maximum resolution, 2) the resolution downsampled by a factor of 2 in the horizontal direction, 3) the resolution downsampled by a factor of 2 in the vertical direction, or 4) the horizontal and vertical directions. , A resolution downsampled by a factor of 2. Alternatively, the encoder may reduce or increase the resolution by some other factor (eg, not a power of two), have additional available resolutions, or use some other technique. Use or set resolution. The encoder sets the resolution for each frame to the original image size. Alternatively, the encoder sets the resolution for the frame to the resolution of the previous frame or the previous resolution setting, i.e., the encoder progressively resolves relative to the previous resolution. Change progressively.

  The decoder decodes the encoded frames and, if necessary, up-samples the frames before displaying them. Like the resolution of an encoded frame, the resolution of a decoded frame can be adjusted in a number of different ways.

  FIG. 5 shows a technique (500) for multi-resolution decoding of a frame. A decoder, such as decoder (300) in FIG. 3, sets the resolution for the frame (510). For example, a decoder obtains resolution information from an encoder.

  The decoder then decodes the frame with the resolution (520). If the decoding is complete (530), the decoder ends. If not, the decoder sets the resolution for the next frame (510) and continues decoding. Alternatively, the decoder sets the resolution to some level other than the frame level.

  In some embodiments, the decoder decodes predicted frames as well as intra frames. FIG. 7 shows a technique (700) for multi-resolution decoding of intra and predicted frames.

  First, the decoder checks whether the current frame to be decoded is an I frame or a P frame (710). If the current frame is an I-frame, the decoder sets the resolution for the current frame (720). If the frame is a P-frame, the decoder sets the resolution for the reference frame (730) before setting the resolution for the current frame (720).

  After setting the resolution for the reference frame (720), the decoder decodes (740) the current frame with that resolution. If the decoding is complete (750), the decoder ends. Otherwise, the decoder continues decoding.

  The decoder typically decodes the frame with the resolution used at the encoder, for example, one of the resolutions described above. Alternatively, the resolution available at the decoder is not exactly the same as that used at the encoder.

A. Signaling The encoder uses bit stream signaling to provide the decoder with enough information to decode the multi-resolution encoded frame. For example, the encoder may indicate whether a sequence of frames is encoded using multi-resolution encoding and / or may output a signal indicating the resolution of the encoded frames within the sequence to 1 It can be sent in the form of one or more flags or codes. Alternatively, the encoder enables / disables multi-resolution encoding at some level other than the sequence level and / or sets the resolution at some level other than the frame level.

  FIG. 8 shows a technique (800) for transmitting a signal when encoding a sequence of frames by multi-resolution encoding. The encoder takes as input the next sequence header (810) and determines whether multi-resolution encoding should be enabled for this sequence (820).

  If the encoder is not using multi-resolution encoding for this sequence, the encoder sets the signal of the sequence accordingly (830) and encodes the frames in the sequence (840). ).

  If the encoder is using multi-resolution encoding, the encoder sets the signal of the sequence accordingly (850). The encoder then codes the frames in the sequence according to a signal indicating a scaling factor for the horizontal and / or vertical resolution of the frame (eg, as described above with reference to FIG. 4 or FIG. 6). Become

  If the encoding is complete (870), the encoder ends. Otherwise, the encoder encodes the next sequence.

  In some embodiments, the encoder sends one bit that indicates whether multi-resolution encoding is enabled for the sequence of frames. Then, for the frames in the sequence, the codes in the fields specified for each of the I and P frames specify the scaling factor for that frame's resolution relative to the maximum resolution frame. In one implementation, the code is a fixed length code. Table 1 shows how the scaling factor is encoded in the field labeled RESPIC FLC.

  Alternatively, the encoder uses another method (eg, a variable length code) that signals adjustments to the frame resolution. Depending on the implementation and the number of possible resolutions, the encoder may use additional or fewer signal codes, or may use different codes for horizontal and vertical resolution . Further, depending on the relative probabilities of possible resolutions, the encoder can adjust the length of the code (eg, assign a shorter code to the most likely resolution). In addition, the encoder can use the signal for other purposes. For example, an encoder may use a signal (eg, a fixed or variable length code) to indicate which filters should be used for resampling in situations where multiple filters are available. . The encoder can use such a signal to indicate which of the available predefined or custom filters should be used in the resampling.

  By transmitting the signal, the encoder provides the decoder with information useful for decoding the multi-resolution encoded frame. The decoder parses the signal to determine how the encoded frame should be decoded. For example, the decoder interprets the code transmitted by the encoder to determine whether the sequence of frames has been encoded using multi-resolution encoding and / or The resolution of the fragmented frame can be determined.

  FIG. 9 shows a technique (900) for receiving and interpreting a signal when decoding a sequence of encoded frames with multi-resolution decoding. The decoder takes as input the next encoded sequence header (910) and checks the signal associated with this sequence to determine if the encoder used multi-resolution encoding for this sequence. A determination is made (920).

  If the encoder did not use multi-resolution encoding, the decoder decodes the frames in the sequence (930). On the other hand, if the encoder used multi-resolution encoding, the decoder parses (940) a signal indicating the horizontal and / or vertical resolution scaling factor for that frame. The decoder then decodes the frame accordingly (eg, as described above with reference to FIG. 5 or FIG. 7) (950).

  If the decoding is complete (960), the decoder ends. Otherwise, the decoder decodes the next sequence.

B. Down-Sampling and Up-Sampling The following sections describe the up-sampling and down-sampling processes in some implementations. Other implementations use different up-sampling, down-sampling or filtering techniques. For example, in an alternative embodiment, the encoder may encode the frame using a non-linear filter or a spatially varying filter bank.

  Table 2 shows the variable definitions used by the encoder and / or decoder for down / up sampling of the frame. These definitions are used in the following description of the pseudo code for the down-sampling and up-sampling embodiments.

Table 2: Variable definitions for down sampling / up sampling in some implementations
N _u = number of samples in the up-sampled (maximum resolution) line
N _d = number of samples in the down-sampled (semi-resolution) line
X _u [n] = upsampled sample values at positions n, except n = 0,1,2 _... N u -1
X _d [n] = sample value down-sampled at position n, where n = 0, 1, 2,..., N _d −1

  The term "line" refers to a set of samples in a horizontal-row or vertical-column in the Y, Cr or Cb component plane. In the following example, the up-sampling and down-sampling operations are equal for both rows and columns, and are therefore illustrated using one-dimensional line samples. When vertical and horizontal upsampling or downsampling is performed, the horizontal lines are resampled first, followed by the vertical lines. Alternatively, horizontal and vertical filtering is performed simultaneously on blocks of pixels using different filters.

  Table 3 shows the pseudo code for resampling the luminance lines, and Table 4 shows the pseudo code for resampling the chrominance lines.

Table 3: Pseudo code for luminance line resampling in some implementations
N _d = N _u / 2 (provided that, N _u is the number of samples in the luminance line up Rizorushon)
if ((N _d & 15)! = 0)
N _d = N _d + 16- (N _d & 15)

Table 4: Pseudocode for chrominance line resampling in some embodiments
N _d = N _u / 2, where _Nu is the number of samples in the chrominance line of maximum resolution
if ((N _d & 7)! = 0)
N _d = N _d + 8- (N _d & 7)

  Resampling sets the number of samples for the downsampled line. Then (in encoders operating with 4: 2: 0 or similar macroblocks) resampling adjusts the number of samples in the line, which is a multiple of the macroblock for the luminance line (ie a multiple of 16). ), Or a multiple of the block for the chrominance line (ie, a multiple of 8).

1. Down Sampling Filter Down sampling of the line produces an output according to the pseudo code in Table 5.

Table 5: Pseudo code for line downsampling in some implementations
if (N _d ! = (N _u / 2))
{
for (i = N _u ; i <N _d * 2; i ++)
x _u [i] = x _u [N _u -1]
}
downsamplefilter_line (x _u [])
for (i = 0; i <N _d ; i ++)
x _d [i] = x _u [i * 2]
The code for the 6 tap filter used in downsample_filterline () is shown in FIG.

  In FIG. 10, RND_DOWN is set to a value of 64 when the image is filtered in the horizontal direction, and is set to a value of 63 when the image is filtered in the vertical direction.

2. Up-sampling filter Up-sampling of the line produces an output according to the pseudocode in Table 6.

Table 6: Pseudo code for line upsampling in some implementations
for (i = 0; i <N _u ; i ++)
{
x _u [i] = x _d [i * 2]
x _u [i + 1] = 0
}
upsamplefilter_line (x _u [])

  A code example for a 10 tap filter used in upsample_filterline () is shown in FIG. In FIG. 11, RND_UP is set to 15 when the image is filtered in the horizontal direction, and is set to 16 when the image is filtered in the vertical direction.

  Other filter pairs can also be used for resampling. The pair of filters can be adjusted to the content of the video and / or the target bit rate. The encoder can send the filter selection as side information to the decoder.

C. Calculation of New Frame Dimensions The pseudo code in Table 7 illustrates how the encoder calculates new frame dimensions for down-sampled frames.

Table 7: Pseudo code for calculating new frame dimensions after down sampling
X = horizontal dimension, number of samples in original resolution
Y = vertical dimension, number of samples in original resolution
x = new horizontal resolution
y = new vertical resolution
hscale = horizontal scaling factor (0 = maximum resolution, 1 = half resolution)
vsscale = vertical scaling factor (0 = maximum resolution, 1 = half resolution)
x = X
y = Y
if (hscale == 1)
{
x = X / 2
if ((x & 15)! = 0)
x = x + 16- (x & 15)
}
if (vscale == 1)
{
y = Y / 2
if ((y & 15)! = 0)
y = y + 16- (y & 15)
}

  In an implementation using the technique shown in Table 7, the encoder computes a new frame dimension, which downsamples the original dimension by a factor of two, and then the new dimension Round up to the nearest integer multiple of size (multiple of 16). For chrominance lines, the encoder rounds the dimensions up to integer multiples of the block size (multiples of 8). Rounding up the new dimension allows the down-sampled frame to be encoded by a video encoder / decoder using 4: 2: 0 or a similar macroblock format.

D. Alternatives (alternatives)
With or in addition to the various alternatives described above, the encoder and decoder can operate as follows.

  The multi-resolution framework can be extended to several levels of down-sampling for individual frames or a series of frames. By using several levels of downsampling, the quality of the reconstructed frame can be improved when the encoder encodes the high resolution frame at a relatively low bit rate.

  The encoder may use a multi-rate filtering technique that resamples the frame to a resolution other than the resolution achieved by adjusting the resolution related to the original resolution by a factor of two. For example, fractional-rate sampling may offer a better trade-off between retaining high frequency details and reducing blocking artifacts at the cost of increased complexity. it can.

  The encoder can apply different levels of resampling to different parts of the frame. For example, the encoder can encode regions of the frame that have little high frequency content with a down-sampled resolution, while the encoder identifies areas of the frame that have strong high frequency content. It can be encoded in the original resolution. In addition, the encoder can apply different filters to resample different portions of the frame, or for vertical and horizontal resampling. The encoder may use signaling to indicate different resampling levels and / or different filters used to resample different portions of the frame.

  Although the principles of the present invention have been described and illustrated with reference to various above-described embodiments, it will be understood that the above-described embodiments may be modified in construction and detail without departing from such principles. It is to be understood that the programs, processes, or methods described herein are not related to or limited to any particular type of computing environment, unless otherwise noted. Various types of general-purpose or special-purpose computing environments may be used with or perform operations in accordance with the teachings described herein. Elements of the above-described embodiments shown in software can be implemented in hardware, and vice versa.

  In view of the large number of possible embodiments to which the principles of the present invention can be applied, the present invention includes all such possible embodiments that fall within the scope and spirit of the appended claims and their equivalents. Claim an embodiment.

FIG. 2 is a block diagram of a suitable computing environment in which the described embodiments can be implemented. FIG. 2 is a block diagram of a video encoder in which the described embodiments can be implemented. FIG. 3 is a block diagram of a video decoder in which the described embodiments can be implemented. 5 is a flowchart illustrating a generalized technique for multi-resolution encoding of a frame. 5 is a flowchart illustrating a generalized technique for multi-resolution decoding of frames. 5 is a flowchart illustrating a technique for multi-resolution encoding of intra and predicted frames. 5 is a flowchart illustrating a technique for multi-resolution decoding of intra and predicted frames. 4 is a flow diagram illustrating a technique for transmitting a signal when encoding a sequence of frames with multi-resolution encoding. 5 is a flowchart illustrating techniques for receiving and interpreting a signal when decoding a sequence of encoded frames with multi-resolution decoding. FIG. 4 is a pseudo code listing for a down sampling filter in one implementation. FIG. 5 is a pseudo code listing for an up-sampling filter in one implementation.

Explanation of reference numerals

200 encoder 205 current frame 210 resolution converter 215 multi-resolution parameter 225 reference frame 230 resolution converter 240 other encoder module 245 motion vector, residual, etc. 300 decoder 305 current frame 310 resolution Transformers 315 Multi-resolution parameters, decoded frames 325 Reference frames 330 Resolution transformers 340 Other decoder modules 345 Motion vectors, residuals, etc.

Claims (61)

In a video encoder, a computer-implemented method for encoding video in any of a number of spatial resolutions, the method comprising:
Encoding at least one frame in a sequence of video frames with a first spatial resolution; and encoding at least one other frame in the sequence of video frames with a second spatial resolution With encoding
The second spatial resolution is different from the first spatial resolution, and the encoder selects the second spatial resolution from a set of spatial resolutions and selects the second spatial resolution in the sequence of video frames. A method comprising reducing blocking artifacts.   The second spatial resolution comprises a vertical resolution having a vertical scaling factor and a horizontal resolution having a horizontal scaling factor, wherein the vertical scaling factor is different from the horizontal scaling factor. The method of claim 1.   The method of claim 1, wherein the encoder switches to the second spatial resolution based on a resolution, at least in part, on a bit rate basis.   The method of claim 1, wherein the encoder switches to the second spatial resolution based at least in part on high frequency content criteria.   The method of claim 1, wherein the encoder switches to the second spatial resolution based at least in part on a quantization step size criterion.   The method of claim 1, wherein frames encoded with the second spatial resolution are down-sampled from the first spatial resolution using a down-sampling filter.   The method of claim 6, wherein the frames encoded with the second spatial resolution are down-sampled using fractional ratio down-sampling.   The method of claim 6, wherein the down sampling filter comprises an even number of filter taps.   The method of claim 6, wherein the down sampling filter is a 6 tap filter.   The method of claim 6, wherein the frames encoded with the second spatial resolution are down-sampled in a horizontal direction before being down-sampled in a vertical direction.   The method of claim 6, further comprising calculating a new dimension for the frame encoded with the second spatial resolution.   The method of claim 1, wherein frames encoded in the second spatial resolution are up-sampled from the first spatial resolution using an up-sampling filter.   The method of claim 12, wherein the up-sampling filter is a 10 tap filter.   The method of claim 1, wherein the encoding of at least one other frame with a second spatial resolution is indicated by a signal code.   The method further includes including a flag in a header for the sequence of the plurality of video frames, wherein the flag indicates that the plurality of frames in the sequence are encoded using a plurality of spatial resolutions. The method of claim 1, wherein   The method of claim 1, wherein the first and second spatial resolutions are selected from a group comprising two or more spatial resolutions.   The method of claim 1, further comprising encoding frames in the sequence of video frames with a third spatial resolution.   The method of claim 1, wherein the at least one other frame comprises a bidirectionally predicted frame.   A computer-readable medium storing computer-executable instructions for causing a computer programmed to perform the method of claim 1. A multi-resolution method for encoding a video, the method comprising:
Encoding a first portion of a frame in the sequence of video frames with a first spatial resolution; and encoding a second portion of the frame in the sequence of video frames into a second With encoding in spatial resolution,
The method of claim 2, wherein the second spatial resolution is different from the first spatial resolution.   The method of claim 20, wherein the first portion of the frame includes high frequency content, and wherein the first spatial resolution is a maximum resolution.   The encoding for encoding the first part includes applying a first filter to the first part, and the encoding for encoding the second part includes applying another filter to the first part. The method of claim 20, comprising applying to a second part.   The method of claim 1, further comprising communicating the first spatial resolution by a first code in a bitstream, and communicating the second spatial resolution by a second code in the bitstream. Item 21. The method according to Item 20,   The method of claim 20, wherein the frames in the sequence are bidirectionally predicted frames. In a video encoder, a computer-implemented method for conveying a spatial resolution in an encoded video sequence, the method comprising:
Including, in a bitstream, a first code indicating a first spatial resolution for a first frame encoded with a first spatial resolution; and a second code encoded with a second spatial resolution. Comprising, in said bitstream, a second code indicating a second spatial resolution for the frame of
The second spatial resolution is different from the first spatial resolution, and the encoder selects the second spatial resolution from a set of spatial resolutions and selects the second spatial resolution in the sequence of video frames. A method comprising reducing blocking artifacts. The second spatial resolution comprises a horizontal resolution and a vertical resolution, and the act of including the second code comprises:
Including in the bitstream a horizontal scaling factor code indicating a scaling factor of the horizontal resolution, and including a vertical scaling factor code indicating a scaling factor of the vertical resolution in the bitstream. The method of claim 25, wherein:   The method of claim 25, wherein the second code is a frame level fixed length code.   26. The method of claim 25, wherein the second code is a frame-level variable length code having a code length, wherein the code length changes based on a probability of occurrence of the second spatial resolution. The described method.   The method of claim 25, further comprising including a sequence code in the bitstream that indicates whether multi-resolution encoding is used in the encoded video sequence.   The method of claim 25, further comprising including a filter code in the bitstream that indicates a resampling filter to be used for decoding.   A computer-readable medium storing computer-executable instructions for causing a computer programmed to perform the method according to claim 25. A method for conveying spatial resolution in a bitstream of an encoded video sequence, comprising:
Including in said bitstream a first signal indicating a first spatial resolution of a first portion of a frame in said encoded video sequence; and said frame in said encoded video sequence. Including in said bitstream a second signal indicative of a second spatial resolution of a second portion of
The method of claim 2, wherein the second spatial resolution is different from the first spatial resolution. Including in the bitstream a first filter code indicating a first filter to be used to decode the first part, and being used to decode the second part Comprising, in said bitstream, a second filter code indicating a second filter to be applied,
33. The method of claim 32, wherein the second filter is different from the first filter.   The method of claim 32, wherein the frames in the encoded video sequence are bidirectionally predicted frames. A video encoder,
Means for setting a plurality of spatial resolutions for encoding a plurality of video frames in the video sequence;
Means for encoding the plurality of video frames in the video sequence.
The means for encoding the plurality of video frames includes: encoding a first video frame of the plurality of video frames in a first spatial resolution; and encoding the first video frame of the plurality of video frames. Two video frames with a spatial resolution different from the first spatial resolution, wherein said means for setting a plurality of spatial resolutions comprises: converting the spatial resolutions into a plurality of spatial resolutions. A video coder comprising means for selecting from a set of solutions to reduce blocking artifacts in said video sequence. A multi-resolution method for decoding video, comprising:
A multi-resolution signal in a sequence header of a video sequence of a plurality of encoded frames, wherein the signal indicates whether the plurality of frames in the video sequence are encoded in a plurality of spatial resolutions. Receiving a multi-resolution signal indicating the first encoded frame of the plurality of encoded frames when the plurality of frames are encoded in a plurality of spatial resolutions. Decoding in a first spatial resolution, and decoding a second encoded frame of the plurality of encoded frames in a second spatial resolution.   The method of claim 36, wherein the second spatial resolution differs from the first spatial resolution by a factor of two.   The second spatial resolution includes a vertical resolution having a vertical scaling factor and a horizontal resolution having a horizontal scaling factor, wherein the vertical scaling factor is different from the horizontal scaling factor. 37. The method of claim 36.   The method of claim 36, wherein the second spatial resolution is determined adaptively based at least in part on a bit rate criterion.   37. The method of claim 36, wherein the second spatial resolution is determined adaptively based, at least in part, on high frequency content criteria.   The method of claim 36, wherein the second spatial resolution is determined at least in part based on a quantization step size criterion.   The method of claim 36, wherein the second encoded frame is down-sampled from an original resolution.   43. The method of claim 42, wherein the second encoded frame is down-sampled from the original resolution using fractional ratio down-sampling. Decoding said second encoded frame, wherein said decoding results in a decoded frame;
Further comprising, after decoding the second encoded frame, performing up-sampling on the decoded frame to produce a decoded frame with a maximum resolution. 43. The method of claim 42, wherein   The method of claim 44, wherein performing the up-sampling comprises applying a 10-tap filter to the decoded frame.   The method of claim 36, wherein a plurality of signal codes in a bitstream convey the first and second spatial resolutions.   The method of claim 36, wherein the second encoded frame is a bidirectionally predicted frame.   A computer-readable medium storing computer-executable instructions for causing a computer programmed with the computer to perform the method of claim 36. A multi-resolution method for decoding video, comprising:
Decoding a first portion of an encoded frame in a video sequence with a first spatial resolution; and decoding a second portion of the encoded frame with a second spatial resolution Be prepared to
The method of claim 2, wherein the second spatial resolution is different from the first spatial resolution.   The method of claim 49, wherein the encoded frames in the video sequence are bidirectionally predicted frames. A video decoder,
Means for receiving, in a sequence header of the video sequence, a multi-resolution signal indicating whether or not a plurality of video frames in the video sequence are encoded with a plurality of spatial resolutions;
Means for setting a plurality of spatial resolutions for decoding the plurality of video frames in the video sequence;
Means for decoding the plurality of video frames in the video sequence. A computer implemented method in a computer system, comprising:
Receiving pixel data of a video image; and calculating resampled pixel data using a resampling filter comprising a 6 tap down sampling filter, the pixel data of the video image Adapting and changing the spatial resolution.   53. The method of claim 52, wherein the calculating comprises horizontal and vertical resampling, wherein the horizontal resampling is performed before the vertical resampling. 53. The method of claim 52, wherein the calculating comprises rounding the number of chrominance samples to a multiple of eight, and rounding the number of luminance samples to a multiple of sixteen.   53. The method of claim 52, further comprising repeating the calculating for each of a plurality of video images in a video image sequence.   53. A computer-readable medium storing computer-executable instructions for causing the computer system to perform the method of claim 52 during video encoding. A computer implemented method in a computer system, comprising:
Receiving pixel data for a video image; and calculating resampled pixel data using a resampling filter with a 10 tap upsampling filter; Adaptively changing the spatial resolution of the image.   58. The method of claim 57, wherein the calculating comprises horizontal and vertical resampling, wherein the horizontal resampling is performed before the vertical resampling. 58. The method of claim 57, wherein the calculating comprises rounding the number of chrominance samples to a multiple of eight, and rounding the number of luminance samples to a multiple of sixteen.   The method of claim 57, further comprising repeating the calculating for each of a plurality of video images in a video image sequence. 58. A computer-readable medium storing computer-executable instructions for causing the computer system to perform the method of claim 57 during video decoding.

Images