Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
  • H04N19/615—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding using motion compensated temporal filtering [MCTF]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/117—Filters, e.g. for pre-processing or post-processing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/136—Incoming video signal characteristics or properties
  • H04N19/137—Motion inside a coding unit, e.g. average field, frame or block difference
  • H04N19/139—Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/146—Data rate or code amount at the encoder output
  • H04N19/152—Data rate or code amount at the encoder output by measuring the fullness of the transmission buffer
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/157—Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/63—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using sub-band based transform, e.g. wavelets
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/63—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using sub-band based transform, e.g. wavelets
  • H04N19/635—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding using sub-band based transform, e.g. wavelets characterised by filter definition or implementation details
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
  • H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
  • H04N19/86—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving reduction of coding artifacts, e.g. of blockiness
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/13—Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]

Definitions

• the present invention relates to video coding, and more particularly, to a method for improving image quality and efficiency for video coding using a smoothed predicted frame.
• Multimedia data requires a large capacity of storage media and a wide bandwidth for transmission since the amount of multimedia data is usually large in relative terms to other types of data. Accordingly, a compression coding method is required for transmitting multimedia data including text, video, and audio. For example, a 24-bit true color image having a resolution of 640*480 needs a capacity of 640*480*24 bits, i.e., data of about 7.37 Mbits, per frame.
• a compression coding method is a requisite for transmitting multimedia data including text, video, and audio.
• Data redundancy is typically defined as: (i) spatial redundancy in which the same color or object is repeated in an image; (ii) temporal redundancy in which there is little change between adjacent frames in a moving image or the same sound is repeated in audio; or (iii) mental visual redundancy taking into account that human eyesight and perception are not sensitive to high frequencies. Data can be compressed by removing such data redundancy.
• Data compression can largely be classified into lossy/lossless compression, according to whether source data is lost, intraframe/interframe compression, according to whether individual frames are compressed independently, and symmetric/asymmetric compression, according to whether time required for compression is the same as time required for recovery.
• data compression is defined as real-time compression when a compression/ recovery time delay does not exceed 50 ms and as scalable compression when frames have different resolutions.
• lossless compression is usually used for text or medical data.
• lossy compression is usually used for multimedia data.
• an ultrahigh- speed communication network can transmit data of several tens of megabits per second while a mobile communication network has a transmission rate of 384 kilobits per second.
• video coding methods such as Motion Picture Experts Group (MPEG)-I, MPEG-2, H.263, and H.264
• motion compensation based on motion estimation and compensation
• spatial redundancy is removed by transform coding.
• a scalable video coding technique means a video coding method having scalability.
• Scalability indicates the ability to partially decode a single compressed bitstream, that is, the ability to perform a variety of types of video re ⁇ production.
• Scalability includes spatial scalability indicating a video resolution, Signal to Noise Ratio (SNR) scalability indicating a video quality level, temporal scalability indicating a frame rate, and a combination thereof.
• SNR Signal to Noise Ratio
• MCTF motion compensated temporal filtering
• FIG. 1 shows the configuration of a conventional scalable video encoder.
• FIG. 2 il ⁇ lustrates a temporal filtering process using 5/3 Motion-Compensated Temporal Filtering (MCTF).
• MCTF Motion-Compensated Temporal Filtering
• the scalable video encoder includes a motion estimator 110 estimating motion between input video frames to determine motion vectors, a motion- compensated temporal filter 140 compensating the motion of an interframe using the motion vectors and removing temporal redundancies within the interframe subjected to motion compensation, a spatial transformer 150 removing spatial redundancies within an intraframe and the interframe within which the temporal redundancies have been removed and producing transform coefficients, a quantizer 160 quantizing the transform coefficients in order to reduce the amount of data, a motion vector encoder 120 encoding a motion vector in order to reduce the number of bits required for the motion vector, and a bitstream generator 130 generating a bitstream using the quantized transform coefficients and the encoded motion vectors.
• the motion estimator 110 calculates a motion vector to be used in compensating the motion of a current frame and removing temporal redundancies within the current frame.
• the motion vector is defined as a displacement from the best-matching block in a reference frame with respect to a block in a current frame.
• HVSBM Hierarchical Variable Size Block Matching
• a frame having an N*N resolution is first downsampled to form frames with lower resolutions such as N/2*N/2 and N/4*N/4 resolutions. Then, a motion vector is obtained at the N/4*N/4 resolution and a motion vector having N/2*N/2 resolution is obtained using the N/4*N/4 resolution motion vector. Similarly, a motion vector with N*N resolution is obtained using the N/2*N/2 resolution motion vector.
• the final block size and the final motion vector are determined through a selection process.
• the motion-compensated temporal filter 140 removes temporal redundancies within a current frame using the motion vectors obtained by the motion estimator 110. To accomplish this, the motion-compensated temporal filter 140 uses a reference frame and motion vectors to generate a predicted frame and compares the current frame with the predicted frame to thereby generate a residual frame.
• the temporal filtering process will be described in more detail later with reference to FIG. 2.
• the spatial transformer 150 spatially transforms the residual frames to obtain transform coefficients.
• the video encoder removes spatial redundancies within the residual frames using wavelet transform.
• the wavelet transform is used to generate a spatially scalable bitstream.
• the quantizer 160 uses an embedded quantization algorithm to quantize the transform coefficients obtained through the spatial transformer 150.
• the motion vector encoder 120 encodes the motion vectors calculated by the motion estimator 110.
• the bitstream generator 130 generates a bitstream containing the quantized transform coefficients and the encoded motion vectors.
• a MCTF algorithm will now be described with reference to FIG. 2.
• a group of picture (GOP) size is assumed to be 16.
• a scalable video encoder receives 16 frames and performs
• MCTF forward with respect to the 16 frames, thereby obtaining 8 low-pass frames and 8 high-pass frames. Then, in temporal level 1, MCTF is performed forward with respect to the 8 low-pass frames, thereby obtaining 4 low-pass frames and 4 high-pass frames. In temporal level 2, MCTF is performed forward with respect to the 4 low-pass frames obtained in temporal level 1, thereby obtaining 2 low-pass frames and 2 high- pass frames. Lastly, in temporal level 3, MCTF is performed forward with respect to the 2 low-pass frames obtained in temporal level 2, thereby obtaining 1 low-pass frame and 1 high-pass frame.
• the video encoder predicts motion between the two frames, generates a predicted frame by compensating the motion, compares the predicted frame with one frame to thereby generate a hig h- pass frame, and calculates the average of the predicted frame and the other frame to thereby generate a low-pass frame.
• the decoder decodes the frame LLLL 16 to reconstruct a video sequence with a frame rate that is one sixteenth of the frame rate of the original video sequence.
• the decoder decodes the frames LLLL 16 and LLLH8 to re ⁇ construct a video sequence with a frame rate that is one eighth of the frame rate of the original video sequence.
• the decoder reconstructs video sequences with a quarter frame rate, a half frame rate, and a full frame rate from a single bitstream.
• scalable video coding allows generation of video sequences at various resolutions, various frames rates or various qualities from a single bitstream, this technique can be used in a wide variety of applications.
• currently known scalable video coding schemes offer significantly lower compression efficiency than other existing coding schemes such as H.264.
• the low compression efficiency is an important factor that severely impedes the wide use of scalable video coding.
• a block-based motion model for scalable video coding cannot effectively represent a non-translatory motion, which will result in block artifacts in low-pass and high-pass subbands produced by temporal filtering and decrease the coding efficiency of the subsequent spatial transform. Block artifacts introduced in a reconstructed video sequence also hampers video quality.
• a closed-loop H. 264 encoder performs deblocking on a reconstructed frame obtained by decoding a previously encoded frame and encodes other frames using the deblocked frame as a reference.
• An H. 264 decoder performs decoding of a received frame for reconstruction, deblocks the reconstructed frame, and decodes other frames using the deblocked frame as a reference. Disclosure of Invention
• deblocking cannot be applied to open-loop scalable video coding that uses an original frame as a reference frame instead of a reconstructed frame obtained by decoding a previously encoded frame.
• the present invention provides temporal decomposition and inverse temporal de ⁇ composition methods using a smoothed predicted frame for video encoding and decoding and a video encoder and decoder.
• a temporal de ⁇ composition method for video encoding including: estimating the motion of a current frame using at least one frame as a reference and generating a predicted frame; smoothing the predicted frame and generating a smoothed predicted frame; and generating a residual frame by comparing the smoothed predicted frame with the current frame.
• a video encoder including a temporal decomposition unit removing temporal redundancies in a current frame to generate a frame in which temporal redundancies have been removed, a spatial transformer removing spatial redundancies in the frame in which the temporal redundancies have been removed to generate a frame in which spatial redundancies have been removed, a quantizer quantizing the frame in which the spatial redundancies have been removed and generating texture information, and a bitstream generator generating a bitstream containing the texture information
• the temporal de ⁇ composition unit comprises a motion estimator estimating the motion of the current frame using at least one frame as a reference, a smoothed predicted frame generator generating a predicted frame using the result of motion estimation and smoothing the predicted frame to generate a smoothed predicted frame, and a residual frame generator generating a residual frame by comparing the smoothed predicted frame with the current frame.
• an inverse temporal decomposition method for video decoding including generating a predicted frame using at least one frame obtained from a bitstream, smoothing the predicted frame and generating a smoothed predicted frame, and reconstructing a frame using a residual frame obtained from the bitstream and the smoothed predicted frame.
• a video decoder including a bitstream interpreter interpreting a bitstream and obtaining texture information and encoded motion vectors, a motion vector decoder decoding the encoded motion vectors, an inverse quantizer performing inverse quantization on the texture information to create frames in which spatial redundancies are removed, an inverse spatial transformer performing inverse spatial transform on the frames in which the spatial redundancies have been removed and creating frames in which temporal re ⁇ dundancies are removed, and an inverse temporal decomposition unit reconstructing video frames from the motion vectors obtained from the motion vector decoder and the frames in which the temporal redundancies have been removed, wherein the inverse temporal decomposition unit comprises a smoothed predicted frame generator generating predicted frames using the motion vectors for frames in which the temporal redundancies have been removed and smoothing the predicted frames to generate smoothed predicted frames and a frame reconstructor reconstructing frames using the frames in which the temporal redundancies have been removed and the smoothed predicted frames
• a video encoding method including downsampling a video frame to generate a low-resolution video frame, encoding the low-resolution video frame, and encoding the video frame using information about the encoded low-resolution video frame as a reference, wherein temporal decomposition in the encoding of the video frame comprises estimating motion of the video frame using at least one frame as a reference and generating a predicted frame, smoothing the predicted frame and generating a smoothed predicted frame, and generating a residual frame by comparing the smoothed predicted frame with the video frame.
• a video decoding method including reconstructing a low-resolution video frame from texture information obtained from a bitstream, and reconstructing a video frame from the texture information using the reconstructed low-resolution video frame as a reference, and wherein the reconstructing of the video frame comprises inversely quantizing the texture information to obtain a spatially transformed frame, performing inverse spatial transform on the spatially transformed frame and obtaining a frame in which temporal redundancies are removed, generating a predicted for the frame in which the temporal redundancies have been removed, smoothing the predicted frame to generate a smoothed predicted frame, and reconstructing a video frame using the frame in which the temporal redundancies have been removed and the smoothed predicted frame.
• FIG. 1 is a block diagram of a conventional scalable video encoder
• FIG. 2 illustrates a conventional temporal filtering process
• FIG. 3 is a block diagram of a video encoder according to a first embodiment of the present invention.
• FIG. 4 illustrates a temporal decomposition process according to a first embodiment of the present invention
• FIG. 5 illustrates a temporal decomposition process according to a second embodiment of the present invention
• FIG. 6 illustrates a temporal decomposition process according to a third embodiment of the present invention
• FIG. 7 is a block diagram of a video decoder according to a first embodiment of the present invention.
• FIG. 8 illustrates an inverse temporal decomposition process according to a first embodiment of the present invention
• FIG. 9 illustrates an inverse temporal decomposition process according to a second embodiment of the present invention.
• FIG. 10 illustrates an inverse temporal decomposition process according to a third embodiment of the present invention.
• FIG. 11 is a block diagram of a video encoder according to a second embodiment of the present invention.
• FIG. 12 is a block diagram of a video decoder according to a second embodiment of the present invention.
• FIG. 3 is a block diagram of a video encoder according to a first embodiment of the present invention.
• FIG. 3 shows a video encoder performing an update step, the video encoder may skip the update step.
• the video encoder includes a temporal decomposition unit 310, a spatial transformer 320, a quantizer 330, and a bitstream generator 340.
• the temporal decomposition unit 310 performs MCTF on input video frames on a group of picture (GOP) basis to remove temporal redundancies within the video frames.
• the temporal decomposition unit 310 includes a motion estimator 312 estimating motion, a smoothed predicted frame generator 314 generating a smoothed predicted frame using motion vectors obtained by the motion estimation, a residual frame generator 316 generating a residual frame (high-pass subband) using the smoothed predicted frame, and an updating unit 318 generating a low-pass subband using the residual frame.
• the motion estimator 312 determines a motion vector by calculating a displacement between each block in a current frame being subjected to temporal decomposition (hereinafter called a 'current frame') and a block in one or a plurality of reference frames corresponding to the block.
• a 'current frame' temporal decomposition
• the current frame includes an input video frame and a low-pass subband being used to generate a residual frame in a higher level.
• the smoothed predicted frame generator 314 uses the motion vectors estimated by the motion estimator 312 and blocks in the reference frame to generate a predicted frame. Instead of directly using the predicted frame, the video encoder of the present embodiment smoothes the predicted frame and uses the smoothed predicted frame in generating a residual frame.
• the residual frame generator 316 compares the current frame with the smoothed predicted frame to generate a residual frame (high-pass subband).
• the updating unit 318 uses the residual frame to update a low-pass subband. A process of generating high-pass subbands and a low-pass subband will be described later with reference to FIGS. 4-6.
• the frames in which temporal redundancies have been removed are sent to the spatial transformer 320.
• the spatial transformer 320 removes spatial redundancies within the frames in which the temporal redundancies have been removed.
• the spatial transform is performed using discrete cosine transform (DCT) or wavelet transform.
• DCT discrete cosine transform
• the frames in which the spatial redundancies have been removed are sent to the quantizer 330.
• the quantizer 330 applies quantization to the frames in which the spatial re ⁇ dundancies have been removed. Quantization for scalable video coding is performed using well-known algorithms such as Embedded ZeroTrees Wavelet (EZW), Set Par ⁇ titioning in Hierarchical Trees (SPIHT), and Embedded ZeroBlock Coding (EZBC).
• EZW Embedded ZeroTrees Wavelet
• SPIHT Set Par ⁇ titioning in Hierarchical Trees
• EZBC Embedded ZeroBlock Coding
• the quantizer 330 converts the frames into texture information that is then sent to the bitstream generator 340. After quantization, the texture information has a signal- to-noise ratio (SNR) scalability.
• SNR signal- to-noise ratio
• the bitstream generator 340 generates a bitstream containing the texture in ⁇ formation, motion vectors, and other necessary information.
• the motion vector encoder 350 losslessly encodes the motion vectors to be contained in the bitstream using arithmetic coding or variable length coding.
• FIG. 4 illustrates a temporal decomposition process according to a first embodiment of the present invention using 5/3 MCTF.
• the temporal decomposition using 5/3 MCTF is used to remove temporal redundancies in a current frame using immediately previous and future frames in the same level.
• Frames 1 through 8 in one GOP are temporally decomposed into one low-pass subband and seven high-pass subbands.
• the shadowed frames in FIG. 4 are frames that are obtained as a result of temporal decomposition and will be converted into texture information after being subjected to spatial transform and quantization.
• P and S re ⁇ spectively denote a predicted frame and a smoothed predicted frame.
• a temporal decomposition process involves 1) generating a predicted frame for using received eight frames making up a GOP, 2) smoothing the predicted frames, 3) generating residual frames using the smoothed predicted frames, and 4) generating a low-pass subband using the residual frames.
• a video encoder uses frame 1 and frame 3 as a reference to generate a predicted frame 2P. That is, motion estimation is required to generate the predicted frame 2P, during which a matching block corresponding to each block in frame 2 is found within the frame 1 and frame 3. Then, a mode is determined by comp aring costs for encoding a block currently being subjected to motion estimation (hereinafter called a 'current block') using a block in the frame 1 (backward prediction mode), a block in the frame 3 (forward prediction mode), both blocks in the frame 1 and frame 3 (bi-directional prediction mode), respectively. Meanwhile, the current block in the frame 2 may be encoded using information from another block in the frame 2 or its own information, which is called an intra-prediction mode.
• the matching blocks corresponding to the blocks in the frame 2 are gathered to generate the predicted frame 2P.
• the video encoder generates predicted frames 4P, 6P, and 8P using frame 3 and frame 5, frame 5 and frame 7, and frame 7 as a reference, respectively.
• the video encoder then smoothes the predicted frames 2P, 4P, 6P, and 8P to generate smoothed predicted frames 2S, 4S, 6S, and 8S, respectively. A smoothing process will be described in detail later.
• the video encoder respectively compares the smoothed predicted frames 2S 4S,
• the video encoder uses the residual frame 2H to update the frame 1, thereby generating a low-pass subband IL.
• the video encoder uses the residual frames 2H and 4H to update the frame 3, thereby generating a low-pass subband 3L.
• the video encoder respectively uses the residual frames 4H and 6H and the residual frames 6H and 8H to generate low-pass subbands 5L and 7L.
• frames in level 0 are decomposed into the low- pass subbands IL, 3L, 5L, and 7L and the residual frames 2H, 4H, 6H, and 8H in level 1.
• the low-pass subbands IL, 3L, 5L, and 7L in level 1 are decomposed into low-pass subbands IL and 5L and residual frames 3H and 7H in level 2.
• the low-pass subbands IL and 5L in level 2 are decomposed into a low-pass subband IL and residual frame 5H in level 3.
• the low-pass subband IL and the high-pass subbands 2H, 3H, 4H, 5H, 6H, 7H, and 8H are then combined into a bitstream, following spatial transform and quantization.
• FIG. 5 illustrates a temporal decomposition process not including an update step according to a second embodiment of the present invention.
• a video encoder obtains residual frames 2H, 4H, 6H, and 8H in level 1 using frames 1 through 8 in level 0 through a predicted frame generation process, a smoothing process, and a residual frame generation process.
• a difference from the first embodiment is that the frames 1, 3, 5, and 7 in level 0 are used as frames 1, 3, 5, and 7 in level 1, re ⁇ spectively, without being updated.
• the video encoder obtains frames 1 and 5 and residual frames 3H and 7H in level 2 using the frames 1, 3, 5, and 7 in level 1. Likewise, the video encoder obtains a frame 1 and a residual frame 5H in level 3 using the frames 1 and 5 in level 2.
• FIG. 6 illustrates a temporal decomposition process using a Haar filter according to a third embodiment of the present invention.
• a video decoder uses all processes, i.e., a predicted frame generation process, a smoothing process, a residual frame generation process, and an update process.
• a predicted frame is generated using only one frame as a reference.
• the video encoder can use either forward or backward prediction mode. That is, the encoder may not select a different prediction mode for each block (e.g., forward prediction for one block and backward prediction for another block) nor a bi ⁇ directional prediction mode.
• the video encoder uses a frame 1 as a reference to generate a predicted frame 2P, smoothes the predicted frame 2P to obtain a smoothed predicted frame 2S, and compares the smoothed predicted frame 2S with a frame 2 to generate a residual frame 2H.
• the video encoder obtains other residual frames 4H, 6H, and 8H.
• the video encoder uses the residual frames 2H and 4H to update the frame 1 and the frame 3 in level 0, thereby generating low-pass subbands IL and 3L in level 1, respectively.
• the video encoder obtains low-pass subbands 5L and 7L in level 1.
• the video encoder obtains low-pass subbands IL and 5L and residual frames 3H and 5H in level 2 using the low-pass subbands IL, 3L, 5L, and 7L. Finally, the video encoder obtains a low-pass subband IL and a residual frame 5H in level 3 using the low-pass subbands IL and 5L in level 2.
• the smoothing process is performed on a predicted frame. While no block artifact is present in an original video frame, block artifacts are introduced in a predicted frame. Thus, block artifacts will be present in a residual frame obtained from the predicted frame and a low-pass subband obtained using the residual frame. To reduce the block artifacts, the predicted frame is smoothed.
• the video encoder performs a smoothing process by deblocking a boundary between blocks in the predicted frame. Deblocking of a boundary between blocks in a frame is also used in the H.264 video coding standard. Since a deblocking technique is widely known in video coding ap ⁇ plications, the description thereof will not be given.
• a deblocking strength can be determined according to the degree of blocking. The deblocking strength can be determined upon several principles.
• a deblocking strength for a boundary between blocks in a predicted frame obtained by motion estimation between frames with a large temporal distance can be made higher than that between blocks in a predicted frame obtained by motion estimation between frames with a small temporal distance.
• a temporal distance between the current frame and reference frame in level 0 is 1 while a temporal distance between the current frame and reference frame in level 1 is 2.
• a deblocking strength for a predicted frame obtained at a higher level is higher than that for a predicted frame obtained at a lower level.
• There are various approaches to determining a deblocking strength according to a level One example is to linearly determine a deblocking strength as defined by Equation (1):
• D is a deblocking strength and Dl is a default deblocking strength that may vary according to a video encoding environment. For example, since a large number of block artifacts may occur at low bit-rate, the default deblocking strength D is large for the low bit-rate environment.
• D2 is an offset value for a deblocking strength for each level and T is a level.
• deblocking strengths D at level 0 and level 2 are Dl and Dl+D2*2, respectively.
• a deblocking strength can also be determined according to a mode selected for each block in a predicted frame.
• a deblocking strength for a boundary between blocks predicted using different prediction modes is made higher than that for a boundary between blocks predicted using the same prediction mode.
• a deblocking strength for a boundary between blocks with a large motion vector difference is made higher than that for a boundary between blocks with a small motion vector difference.
• a decoder smoothes a predicted frame by deblocking the predicted frame with the same deblocking strength as the encoder and reconstructs video frames using the smoothed predicted frame.
• video encoding according to the embodiment of the present invention provides improved video quality over the con ⁇ ventional scalable video encoding.
• FIG. 7 is a block diagram of a video decoder according to an embodiment of the present invention.
• the video decoder performs the inverse operation of an encoder.
• the video encoder removes temporal and spatial redundancies within video frames to generate a bitstream
• the video decoder restores spatial and temporal redundancies from a bitstream to reconstruct video frames.
• the video decoder includes a bitstream interpreter 710 interpreting an input bitstream to obtain texture information and encoded motion vectors, an inverse quantizer 720 inversely quantizing the texture information and creating frames in which spatial redundancies are removed, an inverse spatial transformer 730 performing inverse spatial transform on the frames in which the spatial redundancies have been removed and creating frames in which temporal redundancies are removed, an inverse temporal decomposition unit 740 performing inverse temporal decomposition on the frames in which the temporal redundancies have been removed and reconstructing video frames, and a motion vector decoder 750 decoding the encoded motion vectors.
• the video decoder further includes a post filter 760 deblocking the reconstructed video frames.
• the inverse temporal decomposition unit 740 includes an updating unit 742, a smoothed predicted frame generator 744, and a frame reconstructor 746.
• the updating unit 742 uses a high-pass subband to update a low-pass subband, thereby generating a low-pass subband in a lower level.
• the smoothed predicted frame generator 744 uses the low-pass subband obtained by updating to generate a predicted frame and smoothes the predicted frame.
• the frame reconstructor 746 uses the smoothed predicted frame and the high-pass subband to generate a low-pass subband in a lower level or reconstruct a video frame.
• the post filter 760 reduces the effect of block artifacts by deblocking a re ⁇ constructed frame.
• Information about post-filtering performed by the post filter 760 is provided by an encoder. That is, information determining whether to perform post- filtering on the reconstructed video frame is contained in a bitstream.
• FIG. 8 illustrates an inverse temporal decomposition process using 5/3 MCTF according to a first embodiment of the present invention.
• the inverse temporal de ⁇ composition process using 5/3 MCTF is performed to reconstruct a frame (a low-pass subband or video frame) using reconstructed frames immediately before and after a residual frame, i.e., immediately previous reconstructed frame (a low-pass subband or reconstructed video frame) and immediately next reconstructed frame.
• the inverse temporal decomposition is performed for each GOP including one low-pass subband and seven high-pass subbands. That is, a video decoder receives one low-pass subband and seven high-pass subbands to reconstruct 8 video frames.
• shadowed frames are frames obtained as a result of inverse spatial transform
• P and S respectively denote a predicted frame and a smoothed predicted frame
• H and L respectively denote a residual frame (high-pass subband) and a low-pass subband.
• An inverse temporal decomposition process includes 1) updating received eight subbands in the reverse order that encoding is performed, 2) generating predicted frames, 3) smoothing the predicted frames, and 4) generating low-pass subbands using the smoothed predicted frames or reconstructing video frames.
• the video decoder uses a residual frame 5H to update a low-pass subband IL in level 3 in the reverse order that encoding is performed, thereby generating a low-pass subband IL in level 2.
• the video decoder uses the low-pass subband IL in level 2 and motion vectors to generate a predicted frame 5P and smoothes the predicted frame 5P to generate a smoothed predicted frame 5S. Thereafter, the video decoder uses the smoothed predicted frame 5S and the residual frame 5H to reconstruct a low-pass subband 5L in level 2.
• the video decoder reconstructs low-pass subbands IL, 3L, 5L, and 7L in level 1 using the low-pass subbands IL and 5L and residual frames 3H and 7H in level 2.
• the video decoder uses the low- pass subbands IL, 3L, 5L, and 7L and residual frames 2H, 4H, 6H, and 8H in level 1 to reconstruct video frames 1 through 8. Meanwhile, when further needed according to the information contained in the bitstream, post filtering is performed on the video frames 1 through 8.
• FIG. 9 illustrates an inverse temporal decomposition process according to a second embodiment of the present invention.
• the inverse temporal decomposition process according to the present embodiment does not include an update step.
• a video frame 1 in level 3 is the same as reconstructed video frames 1 in levels 2, 1, and 0.
• a video frame 5 in level 2 is the same as re ⁇ constructed video frames 5 in levels 1 and 0, and video frames 3 and 7 in level 1 are the same as video frames 3 and 7 in level 0.
• the video decoder reconstructs a video frame 5 in level 2 using a video frame 1 and a residual frame 5H in level 3. Likewise, the video decoder re ⁇ constructs video frames 3 and 7 in level 1 using reconstructed video frames 1 and 5 and residual frames 3H and 7H in level 2. Lastly, the video decoder reconstructs video frames 1 through 8 in level 0 using reconstructed video frames 1, 3, 5, and 7 and residual frames 2H, 4H, 6H, and 8H in level 1.
• FIG. 10 illustrates an inverse temporal decomposition process using a Haar filter according to a third embodiment of the present invention.
• a video decoder uses all processes, i.e., an update process, a predicted frame generation process, a smoothing process, and a frame reconstruction process.
• the difference from the first embodiment is that a predicted frame is generated using only one frame as a reference.
• the video decoder can use either forward or backward prediction mode.
• the video decoder uses a low-pass subband IL and a residual frame 5H in level 3 to reconstruct low-pass subbands IL and 5L in level 2. Then, the video decoder uses the reconstructed low- pass subbands IL and 5L and residual frames 3H and 7H in level 2 to reconstruct low- pass subbands IL, 3L, 5L, and 7L in level 1. Lastly, the video decoder uses the low- pass subbands IL, 3L, 5L, and 7L and residual frames 2H, 4H, 6H, and 8H to re- construct video frames 1 through 8.
• a smoothing process performed in the embodiments shown in FIGS. 8-10 is performed according to the same principle as an encoding process.
• a deblocking strength increases when a temporal distance between a reference frame and a predicted frame increases.
• a deblocking strength for blocks predicted using a different motion estimation mode or having a large motion vector difference is high.
• Information about a deblocking strength can be obtained from a bitstream.
• FIG. 11 is a block diagram of a video encoder according to a second embodiment of the present invention.
• the video encoder is a multi-layer encoder having layers with different resolutions.
• the video encoder includes a downsampler 1105, a first temporal decomposition unit 1110, a first spatial transformer 1130, a first quantizer 1140, a frame reconstructor 1160, an upsampler 1165, a second temporal de ⁇ composition unit 1120, a second spatial transformer 1135, a second quantizer 1145, and a bitstream generator 1170.
• the downsampler 1105 downsamples video frames to generate low-resolution video frames that are then provided to the first temporal decomposition unit 1110.
• the first temporal decomposition unit 1110 performs MCTF on the low -resolution video frames on a GOP basis to remove temporal redundancies in the low-resolution video frames.
• the first temporal decomposition unit 1110 includes a motion estimator 1112 estimating motion, a smoothed predicted frame generator 1114 generating a smoothed predicted frame using motion vectors obtained by the motion estimation, a residual frame generator 1116 generating a residual frame (high-pass subband) using the smoothed predicted frame, and an updating unit 1118 generating a low-pass subband using the residual frame.
• the motion estimator 1112 determines a motion vector by calculating a displacement between each block in a low-resolution video frame being encoded and a block in one or a plurality of reference frames corresponding to the block.
• the smoothed predicted frame generator 1114 uses the motion vectors estimated by the motion estimator 1112 and blocks in the reference frame to generate a predicted frame. Instead of directly using the predicted frame, the present embodiment smoothes the predicted frame and uses the smoothed predicted frame in generating a residual frame.
• the residual frame generator 1116 compares the low -resolution video frame with the smoothed predicted frame to generate a residual frame (high-pass subband).
• the updating unit 1118 uses the residual frame to update a low-pass subband.
• the low- resolution video frames in which temporal redundancies have been removed (the low- pass and high-pass subbands) are then sent to the first spatial transformer 1130.
• the first spatial transformer 1130 removes spatial redundancies within the frames in which the temporal redundancies have been removed.
• the spatial transform is performed using discrete cosine transform (DCT) or wavelet transform.
• DCT discrete cosine transform
• the frames in which spatial redundancies have been removed using the spatial transform are sent to the first quantizer 1140.
• the first quantizer 1140 applies quantization to the low-resolution video frames in which the spatial redundancies have been removed. After quantization, the low- resolution video frames are converted into texture information that is then sent to the bitstream generator 1170.
• the motion vector encoder 1150 encodes the motion vectors obtained during motion estimation in order to reduce the number of bits required for the motion vectors.
• the frame reconstructor 1160 performs inverse quantization and inverse spatial transform on the quantized low-resolution frames, followed by inverse temporal de ⁇ composition using motion vectors, thereby reconstructing low-resolution video frames.
• the upsampler 1165 upsamples the reconstructed low-resolution video frames.
• the upsampled video frames are used as a reference in compressing video frames.
• the second temporal decomposition unit 1120 performs MCTF on input video frames on a GOP basis to remove temporal redundancies in the video frames.
• the second temporal decomposition unit 1120 includes a motion estimator 1122 estimating motion, a smoothed predicted frame generator 1124 generating a smoothed predicted frame using motion vectors obtained by the motion estimation, a residual frame generator 1126 generating a residual frame (high-pass subband) using the smoothed predicted frame, and an updating unit 1128 generating a low-pass subband using the residual frame.
• the motion estimator 1122 obtains a motion vector by calculating a displacement between each block in a video frame currently being encoded and a block in one or a plurality of reference frames corresponding to the block or determines whether to use each block in the upsampled frame obtained by the upsampler 1165.
• the smoothed predicted frame generator 1124 uses blocks in the reference frame and the upsampled frame to generate a predicted frame. Instead of directly using the predicted frame, the video encoder of the present embodiment smoothes the predicted frame and uses the smoothed predicted frame in generating a residual frame.
• the residual frame generator 1126 compares the smoothed predicted frame with the video frame to generate a residual frame (high-pass subband).
• the updating unit 1128 uses the residual frame to update a low-pass subband.
• the video frames in which temporal redundancies have been removed are then sent to the second spatial transformer 1135.
• the second spatial transformer 1135 removes spatial redundancies within the frames in which the temporal redundancies have been removed.
• the spatial transform is performed using discrete cosine transform (DCT) or wavelet transform.
• DCT discrete cosine transform
• the frames in which spatial redundancies have been removed using the spatial transform are sent to the second quantizer 1145.
• the second quantizer 1145 applies quantization to the video frames in which the spatial redundancies have been removed. After quantization, the video frames are converted into texture information that is then sent to the bitstream generator 1170.
• the motion vector encoder 1155 encodes the motion vectors obtained during motion estimation in order to reduce the number of bits required for the motion vectors.
• the bitstream generator 1170 generates a bitstream containing the texture in ⁇ formation and motion vectors associated with the low-resolution video frames and original-resolution video frames and other necessary information.
• FIG. 11 shows the multi-layer video encoder having two layers of different resolutions
• the video encoder may have three or more layers of different resolutions.
• a multi-layer video encoder performing different video coding schemes at the same resolution may also be implemented in the same way as in FIG. 11. For example, when first and second spatial transformers 1130 and 1135 respectively adopt DCT and wavelet transform, the multi-layer video encoder having layers of the same resolution does not require the downsampler 1105 nor the upsampler 1165.
• the multi-layer video encoder of FIG. 11 may be implemented such that either one of the first and second temporal transformers 1110 and 1120 generates a smoothed predicted frame and the other generates a typical predicted frame.
• FIG. 12 shows the configuration of a video decoder according to a second embodiment of the present invention as the counterpart of the video encoder of FIG. 11.
• the video decoder may also be configured to reconstruct video frames from a bitstream encoded by the modified multi-layer video encoder described above.
• the video decoder includes a bitstream interpreter 1210 in ⁇ terpreting an input bitstream to obtain texture information and encoded motion vectors, first and second inverse quantizers 1220 and 1225 inversely quantizing the texture in ⁇ formation and creating frames in which spatial redundancies are removed, first and second inverse spatial transformers 1230 and 1235 performing inverse spatial transform on the frames in which the spatial redundancies are removed and creating frames in which temporal redundancies are removed, first and second inverse temporal decomposition units 1240 and 1250 performing inverse temporal decomposition on the frames in which the temporal redundancies have been removed and reconstructing video frames, and motion vector decoders 1270 and 1275 decoding the encoded motion vectors.
• the video decoding involves a smoothing process for smoothing a predicted frame, and the video decoder further includes a post filter 1260 deblocking the reconstructed video frames.
• FIG. 12 shows that both the first and second inverse temporal decomposition units 1240 and 1250 generate smoothed predicted frames, either one of the first and second inverse temporal decomposition units 1240 and 1250 may generate a typical predicted frame.
• the first inverse quantizer 1220, the first inverse spatial transformer 1230, and the first inverse decomposition unit 1240 reconstruct low-resolution video frames, and the upsampler 1248 upsamples the reconstructed low-resolution video frames.
• the second inverse quantizer 1225, the second inverse spatial transformer 1235, and the second inverse temporal decomposition unit 1250 reconstructs video frames using an upsampled frame obtained by the upsampler 1248 as a reference.
• the video decoder does not require the upsampler 1248.
• the temporal decomposition and inverse temporal de ⁇ composition methods according to the present invention allow smoothing of predicted frame during open-loop scalable video encoding and decoding, thereby improving image quality and coding efficiency for video coding.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Applications Claiming Priority (4)

Publications (1)

Family

ID=35784112

Family Applications (1)

Country Status (2)

Families Citing this family (1)

Family Cites Families (4)

• 2005
  • 2005-07-08 WO PCT/KR2005/002191 patent/WO2006006796A1/en not_active Application Discontinuation
  • 2005-07-08 EP EP05765962A patent/EP1766986A1/de not_active Withdrawn

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events