KR20090059707A - Apparatus of scalable video encoding using closed-loop filtering and the method thereof - Google Patents

Abstract

A scalable video encoding apparatus using the closed loop filtering and a method thereof are provided to perform the video coding efficiently by guaranteeing the quality of images, respectively having different resolution. A scalable video encoding apparatus using the closed loop filtering comprises a space-time conversion unit(13), a quantization unit and a closed loop filtering unit(22). The space-time conversion unit removes the temporal and spatial redundancy by using a reference image to generate the first conversion image. The quantization unit produces image information coded by quantizing the first conversion image. The closed loop filtering unit sequentially performs the inverse quantization and inverse space-time conversion for the coded image information to generate a decoding image, and then provides the decoding image as the reference image.

Description

Apparatus and method for scalable video encoding using closed loop filtering {APPARATUS OF SCALABLE VIDEO ENCODING USING CLOSED-LOOP FILTERING AND THE METHOD THEREOF}

The present invention relates to video coding, and more particularly, to a scalable video coding method and apparatus therefor.

The present invention is derived from the research conducted as part of the IT growth engine technology development project of the Ministry of Information and Communication and the Ministry of Information and Communication Research.

Project Name: Technology for Advanced Terrestrial DMB Transmission]

As information and communication technology including the Internet is developed, not only text and voice but also video communication are increasing. Accordingly, there is a demand for a multimedia service capable of accommodating various types of information such as text, video, and music.

Multimedia data has a huge amount and requires a large storage medium and a wide bandwidth in transmission. Therefore, compression coding is used to transmit multimedia data including text, video, and audio.

For example, a 24-bit True Color image with a resolution of 640 * 480 requires a capacity of 640 * 480 * 24 bits (ie approximately 7.37 Mbits of data) per frame. do. When transmitting it at 30 frames per second, a 24-bit true color image with a resolution of 640 * 480 requires a bandwidth of 221 Mbit / sec, and approximately 1200 Gbits of storage space to store a 90-minute movie. This is necessary.

The basic principle of compressing data is to eliminate data redundancy. Spatial overlap is the repetition of the same color or object in an image. Temporal redundancy is when the adjacent frames in a video frame change little, or when the same note is repeated over and over in audio.

Motion compensated predictive coding in video coding eliminates this duplication. That is, temporal redundancy is eliminated by temporal filtering based on motion compensation, and spatial redundancy is eliminated by spatial transform.

The multimedia generated after the data is duplicated is transmitted through the transmission medium. Currently used transmission media are very diverse, ranging from ultra-high speed communication networks capable of transmitting tens of megabits of data per second to mobile communication networks having a transmission rate of 384 kbits per second.

In general, the encoder divides the input video into groups of pictures (GOPs), which are basic units of encoding, and performs encoding for each GOP.

FIG. 1 is a block diagram illustrating a typical open-loop scalable video encoder.

Referring to FIG. 1, the scalable video encoder 10 includes a spatial transform unit 1, a temporal transform unit 2, a buffer 5, an embedded quantizer 6, and an entropy encoder 7. .

The spatial transform unit 1 uses a wavelet transform to remove spatial redundancy of the input video sequence. The wavelet transform divides one frame into quarters. The wavelet transform replaces a reduced image (hereinafter referred to as an "L subband") with a quarter area that is almost similar to the entire image, to any one of the quadrants of the one frame, and L in the other three quadrants. It is replaced with information (hereinafter referred to as "H subband") that allows the entire image to be restored through the subband.

The spatial converter 1 generates a residual image by wavelet converting the input image. The generated residual image is represented by wavelet coefficients. The wavelet coefficient refers to the transition of the scale and time axis of the residual image from which spatial overlap of the input image is removed.

The temporal transform unit 2 includes a motion estimation unit 3 and a temporal filtering unit 4. The motion estimation unit 3 generates a motion vector by performing motion estimation on the [n] th frame of the GOP using the [n-1] th frame of the GOP (Group of Picture) stored in the buffer 5 as a reference frame. do. The [n] th frame is stored in the buffer 5 for motion estimation of the next frame. The temporal filtering unit 4 generates a temporal transformed image by removing temporal redundancy between frames using the generated motion vector.

The embedded quantization unit 6 quantizes the transformed image (ie, wavelet coefficient) generated from the temporal filtering unit 4. The entropy encoder 7 generates a bit stream by encoding the quantized wavelet coefficients and the motion vector generated by the motion estimator 3. The entropy encoder 7 generates a bitstream including coded image information (ie, quantized wavelet coefficients), motion vectors obtained from the motion estimation unit 3, and other necessary information.

As the embedded quantization scheme, a motion compensated temporal filtering (MCTF) scheme is used.

The scalable video encoder can reduce the amount of information required by quantization by performing quantization on transform coefficients through an embedded quantization scheme, and obtain signal-to-noise ratio (SNR) scalability through embedded quantization. have.

In fact, since motion vectors for constructing a reference image in each layer are similar but not identical, the encoder cannot use a motion estimate optimized for a low resolution image using the motion vectors for the highest resolution. Therefore, the image quality of the residual image of the lowest resolution is severely degraded. In addition, if a large number of bits are allocated in order to improve the image quality, the compression efficiency decreases.

The video encoder 10 supporting temporal scalability shown in FIG. 1 has an open-loop structure to implement signal-to-noise ratio scalability. In general, an image of a current frame is used as a reference frame of a next frame in a video encoding process.

That is, the encoder 10 of the open loop structure as shown in FIG. 1 uses the previous original picture frame n−1 as a reference frame for the current frame. Previous frame [n-1]) is used as the reference frame of the current frame. That is, errors accumulate as frames are repeated within one GOP. Therefore, in the open loop encoder, drift occurs in the reconstructed image.

SUMMARY OF THE INVENTION An object of the present invention is to provide a closed loop filtering method in scalable video coding in order to improve image quality deterioration caused by accumulation of an error generated by quantization between an encoder and a decoder.

In addition, another object of the present invention is to provide a video coding method and apparatus therefor having a good image quality at each resolution while reducing the redundancy of the coded image at each resolution.

The scalable video encoding apparatus according to an embodiment of the present invention receives a reference image corresponding to the original image, and uses a reference image to remove a spatial and temporal overlap of the original image to generate a spatiotemporal transform unit. ; A quantizer configured to quantize the transformed image to generate coded image information; And a closed loop filtering unit configured to sequentially generate the decoded image information through inverse quantization, inverse spatial and inverse temporal transformation, and provide the decoded image as the reference image.

The closed loop filtering unit may include an inverse quantization unit configured to inversely quantize the coded image information to generate the converted image; An inverse spatial transform unit which inversely transforms the converted image to generate the residual image; And an inverse temporal filtering unit generating the decoded image by inversely transforming the residual image.

In this embodiment, the spatio-temporal transform unit may include: a spatial transform unit configured to receive a reference image corresponding to the original image and to remove a spatial overlap of the original image using the reference image to generate a residual image; And a temporal converter configured to perform a motion estimation corresponding to the residual image, generate a motion vector according to the executed motion estimation, and remove a temporal overlap of the original image using the motion vector to generate a transformed image. .

The apparatus may further include an entropy encoder configured to generate a bit stream including the motion vector and the coded image information.

The method may further include a buffer that stores the decoded image and provides the decoded image as the reference image to the spatial transform unit.

The scalable video encoding apparatus of claim 1, further comprising a down sampling unit configured to low pass filter the original resolution image of the original image to generate a low resolution image.

The down sampling unit may include: a first down sampling unit configured to low pass filter the original resolution image to generate a first low resolution image; A second down sampling unit configured to low pass filter the first low resolution image to generate a second low resolution image; And an n-th down sampling unit configured to generate an n-th low resolution image by low-pass filtering the n-1 low-resolution image.

In this embodiment, the lowpass filtering is down sampling by a wavelet filter.

In this embodiment, the spatiotemporal transform units are individually processed according to different resolutions.

In the present embodiment, the converted image is generated by removing the spatial and temporal overlap of the original resolution image and the first and second low resolution images to generate the original resolution converted image, the first and second low resolution converted images, and The first and second low resolution converted images are integrated to generate an integrated low resolution converted image, and the integrated low resolution converted image and the original resolution converted image are integrated.

A scalable video encoding method according to an embodiment of the present invention comprises the steps of: generating the converted image by dequantizing coded image information; Generating a residual image by inverse spatially transforming the converted image; And generating a decoded image by inversely transforming the residual image to provide the decoded image as a reference image.

In this embodiment, (a) receiving the reference image corresponding to the original image, and using the reference image to remove the spatial and temporal overlap of the original image to generate a converted image; And (b) quantizing the transformed image to generate the coded image information.

In this embodiment, the step (a) may include: generating a residual image by removing spatial redundancy of the original image using the reference image; And performing a motion estimation corresponding to the residual image, generating a motion vector according to the executed motion estimation, and removing a temporal overlap of the original image using the motion vector to generate a transformed image.

In this embodiment, the step (b) further comprises generating a bit stream including the motion vector and the coded image information.

In this embodiment, the method may further include storing the decoded image and providing the decoded image as the reference image.

In this embodiment, the method further includes low-pass filtering the original resolution image of the original image to generate a low resolution image.

The generating of the low resolution image may include: generating a first low resolution image by low pass filtering the original resolution image; Low-pass filtering the first low resolution image to generate a second low resolution image; And in the same manner, generating an nth low-resolution image by lowpass filtering the n-1th low-resolution image.

In the present embodiment, the converted image is generated by removing the spatial and temporal overlap of the original resolution image and the first and second low resolution images to generate the original resolution converted image, the first and second low resolution converted images, and The first and second low resolution converted images are integrated to generate an integrated low resolution converted image, and the integrated low resolution converted image and the original resolution converted image are integrated.

As described above, the present invention can ensure the maximum quality of the image of each resolution by integrating the image for each resolution, thereby enabling efficient video coding.

In addition, the present invention can reduce the drift phenomenon of the image by reducing the cumulative error caused by quantization by closed-loop optimization.

The scalable video encoding apparatus according to an embodiment of the present invention receives a reference image corresponding to the original image, and uses a reference image to remove a spatial and temporal overlap of the original image to generate a spatiotemporal transform unit. ; A quantizer configured to quantize the transformed image to generate coded image information; And a closed loop filtering unit configured to sequentially generate the decoded image information through inverse quantization, inverse spatial and inverse temporal transformation, and provide the decoded image as the reference image, wherein the closed loop filtering unit comprises: An inverse quantizer configured to inversely quantize coded image information to generate the converted image; An inverse spatial transform unit which inversely transforms the converted image to generate the residual image; And an inverse temporal filtering unit generating the decoded image by inversely transforming the residual image.

Therefore, the present invention can ensure the best image quality of the image of each resolution by integrating the image for each resolution, it is possible to perform efficient video coding. In addition, the present invention can reduce the drift phenomenon of the image by reducing the cumulative error caused by quantization by closed-loop optimization.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings so that those skilled in the art may easily implement the technical idea of the present invention.

Embodiments in accordance with the present invention assume video coding to have three resolutions from one bitstream. That is, the first layer relates to coding and decoding of an image having an original resolution having the highest resolution, a second layer having an intermediate resolution, and a third layer having an lowest resolution. An embodiment according to the present invention describes a coding and decoding process of an image based on one frame to be coded.

FIG. 2 is a block diagram illustrating a scalable video encoder according to the present invention, FIG. 3 is a block diagram showing the motion compensation unit shown in FIG. 2, and FIG. 4 is a block diagram showing the embedded inverse quantum unit shown in FIG. It is also.

Referring to FIG. 2, the scalable video encoder 100 according to the present invention includes a first and second down sampling units 11 and 12, a spatial transform unit 13, a temporal transform unit 17, and an embedded quantization unit ( 21, a buffer 26, an entropy encoder 27, and a closed loop filter 22.

First, the scalable video encoder 100 according to the present invention divides the input video into a group of pictures (GOP) which is a basic unit of encoding, and performs encoding for each GOP.

The first down sampling unit 11 extracts the second layer (ie, the first low resolution image) from the first layer (ie, the original resolution image). The second down sampling unit 12 extracts a third layer (ie, a second low resolution image) from the second layer. The first and second down sampling units 11 and 12 according to the present invention use a wavelet 9-7 filter.

The spatial converter 13 includes first to third wavelet converters 14-16. The first to third wavelet transform units 14-16 of the spatial transform unit 13 which remove spatial redundancy for each resolution have the same structure. The original resolution image and the first and second low resolution images are converted into the first to third residual images through the first to third wavelet converters 14-16, respectively. That is, the first wavelet converter 14 converts the original resolution image into the first residual image, and the second wavelet converter 15 converts the first low resolution image into the second residual image and the third wave. The let converting unit 16 converts the second low resolution image into a third residual image.

The temporal transform unit 17 includes an image reference unit 20, a motion estimator 19, and a motion compensator 18. That is, the image reference unit 20 stores the first to third residual images and provides the first to third residual images to the motion estimation unit 19.

The motion estimation unit 19 performs motion estimation on the frame [n] of the current GOP by using the frames [n-1] of the current GOP reconstructed by the closed loop filtering unit 22 as the first to third reference images. And generate a motion vector. The motion estimation unit 19 performs motion estimation with reference to the input image and the images stored in the image reference unit 20. The motion vectors obtained through the motion estimation are provided to the motion compensation unit 18. The motion compensator 18 generates the first to third transformed images by comparing the reference frame obtained through the motion compensation with the input image.

The motion compensation unit according to the present invention will be described in detail with reference to FIG. 3.

1 and 3, the motion compensator 18 includes first to third temporal filtering units 181 to 183. The first temporal filtering unit 181 generates a first converted image by removing temporal overlap of the first residual image. The second temporal filtering unit 182 generates a first converted image by removing temporal overlap of the second residual image. The third temporal filtering unit 183 generates a first converted image by removing temporal overlap of the third residual image. The motion compensator 18 provides the first to third transformed images to the entropy encoder 27.

2, the embedded quantizer 21 quantizes the residual image (that is, the wavelet coefficients) transformed by the temporal transformer 17 and rearranges the wavelet coefficients according to the magnitude of the wavelet coefficients.

In the wavelet coefficients, the larger coefficient is more important than the smaller coefficient. The embedded quantizer 21 rearranges the wavelet coefficients according to the size and then sends the wavelet coefficient having the largest size first. That is, in order to implement scalability, the embedded quantization unit 21 encodes only pixels having a value larger than a threshold, and after processing all pixels, lowers the threshold and repeats again. Embedded quantization is performed through.

The closed loop filtering unit 22 includes an embedded inverse quantization unit 23 which inversely quantizes the coded image information to generate a transformed image, an inverse spatial transformer 24 which inversely spatially transforms the transformed image, and generates a residual image, and a residual An inverse temporal filtering unit 25 generates a decoded image by inversely transforming an image. That is, the closed loop filtering unit 22 generates a reconstructed frame by performing a decoding process on the quantized wavelet coefficients, and serves as a reference frame (ie, a reference picture) in a subsequent motion estimation process.

The integrated coded image information generated by the embedded quantization unit 21 is also input to the closed loop filtering unit 22.

The embedded inverse quantization unit 23 decodes the integrated coded image information (that is, the quantized wavelet coefficients) received from the embedded quantization unit 21 according to the size information. That is, the embedded inverse quantization unit 23 arranges the wavelet coefficients in a spatial order in the reverse order of the method used in the embedded quantization unit 21. Therefore, the coded image information is separated and inversely quantized by the embedded inverse quantization unit 23 to be first to third converted images according to respective resolutions.

The structure of the embedded dequantization unit according to the present invention will be described in detail with reference to FIG. 4.

1 and 4, the embedded inverse quantization unit 23 includes first to third embedded inverse quantization units 231 to 233. The first and second distributors 234 and 235 provide integrated coded image information to the first to third embedded inverse quantizers 231 to 233.

The first embedded inverse quantization unit 231 inversely quantizes the integrated coded image information to generate a first transformed image. The second embedded inverse quantization unit 232 inversely quantizes the integrated coded image information to generate a second converted image. The third embedded inverse quantization unit 233 inversely quantizes the integrated coded image information to generate a second transformed image. The embedded inverse quantization unit 23 provides the first to third transformed images to the inverse spatial transform unit 24.

Subsequently, referring to FIG. 2, the inverse spatial transform unit 24 performs operations in the spatial transform unit 13 in the reverse order. The inverse transform quantized by the inverse quantization unit 23 converts the inverse to restore the frame in the spatial domain. When the transform coefficient is a wavelet coefficient, the embedded inverse quantization unit 23 converts the wavelet coefficient according to the inverse wavelet transform and generates a temporal difference frame. Accordingly, the first to third transformed images become first to third residual images through the inverse spatial transform unit 24.

The inverse temporal filtering unit 25 uses the first to third reference images generated by the spatial filtering unit 25 to display the first to third residual images transmitted from the inverse spatial transform unit 24 to the first to third images. Convert to decoded video. That is, the first to third residual images become first to third decoded images of each resolution through the inverse temporal filtering unit 25. The image reconstructed by the inverse temporal filtering unit 25 becomes a final reconstructed image.

The buffer 26 stores the first to third decoded images generated by the inverse temporal filtering unit 25, and provides the first to third decoded images as reference images when performing motion estimation on the images.

The entropy encoder 27 converts the wavelet coefficient quantized by the embedded quantizer 21 and the motion vector information and other header information generated by the motion estimation unit 19 into a compressed bitstream suitable for transmission or storage. .

That is, the first to third residual images are eliminated and merged through the temporal transform unit 17 to form one integrated transformed image. The integrated transform image is quantized and coded by the embedded quantization unit 21. The coded image obtained by coding each input image is a bitstream through entropy encoding 22 together with the motion vector for each resolution obtained in the process of eliminating temporal overlap. The bitstream includes information about the coded image and motion vectors, and includes other necessary header information.

Entropy encoding methods include a predictive coding method, a variable-length coding method (Huffman coding is typical), or an arithmetic coding method.

The open loop method refers to the original video in the process of eliminating temporal redundancy in the encoding process, but refers to the decoded video in the process of eliminating inverse temporal redundancy in the decoding process, so that a so-called drift error phenomenon occurs. Occurs. In contrast, in the closed loop method, the drift error does not occur because the encoding and decoding processes refer to the decoded image in the process of eliminating temporal duplication.

In order to improve the problem of such an open loop coding scheme, the encoder according to the present invention entropy-codes quantized transform coefficients, decodes the transform coefficients inversely, and provides a reconstructed frame as a reference frame. Therefore, the encoder according to the present invention generates the same environment as the decoder process and eliminates the accumulated error.

5 is a block diagram illustrating a scalable video decoder according to the present invention. 2 and 5, the scalable video decoder 50 may include an inverse entropy encoder 51, an embedded inverse quantizer 52, an inverse spatial converter 53, an inverse temporal converter 54, and a buffer. And 55.

The inverse entropy encoder 51 performs the reverse process of the entropy encoder 27 of the encoder 100. In other words, the inverse entropy encoder 51 obtains a quantized transform coefficient from the input bitstream.

The embedded dequantizer 52 decodes the quantized wavelet coefficients (ie, coded image information) received from the entropy encoder 27, similarly to the embedded dequantizer 23 of the encoder 100, according to the size information. .

The inverse spatial transform unit 53 operates in the same manner as the inverse spatial transform unit 24 of the encoder 200.

The inverse temporal filtering unit 54 uses the previously reconstructed reconstructed image [n-1] as a reference image, and uses the motion vector transmitted from the inverse entropy encoder 51 to convert the temporal residual image into the final reconstructed image [ n]. The inverse temporal filtering unit 54 stores the last reconstructed reconstructed image [n] in the buffer 55 to be used as a reference image for the subsequent image.

As described above, optimal embodiments have been disclosed in the drawings and the specification. Although specific terms have been used herein, they are used only for the purpose of describing the present invention and are not intended to limit the scope of the invention as defined in the claims or the claims. Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible from this. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

1 is a block diagram illustrating a typical open loop scalable video encoder.

2 is a block diagram illustrating a scalable video encoder in accordance with the present invention.

3 is a block diagram illustrating a motion estimation unit shown in FIG. 2.

4 is a block diagram showing the embedded inverse quantum unit shown in FIG.

5 is a block diagram illustrating a scalable video decoder in accordance with the present invention.

Explanation of symbols on the main parts of the drawings

11: first down sampling unit 12: second down sampling unit

13: spatial transform unit 14: first wavelet transform unit

15: second wavelet converter 16: third wavelet converter

17: temporal conversion unit 18: motion compensation unit

19: motion estimation unit 20: image reference unit

21: embedded quantization unit 22: closed loop filtering unit

23: embedded inverse quantization unit 24: inverse spatial transform unit

25: inverse temporal filtering unit 26: buffer

27: entropy encoder 100: scalable video encoder

Claims (21)

A spatio-temporal transform unit for generating a first transformed image by removing spatial and temporal overlap of the original image using the reference image; A quantizer configured to quantize the first transformed image to generate coded image information; And And a closed loop filtering unit configured to sequentially dequantize the coded image information, inverse spatially, and inversely temporally to generate a decoded image, and provide the decoded image as the reference image. The method of claim 1, The closed loop filtering unit, An inverse quantizer configured to inversely quantize the coded image information to generate a second transformed image; An inverse spatial transform unit which inverse-space transforms the second transformed image to generate a residual image; And And a reverse temporal filtering unit generating the decoded image by inversely converting the residual image. The method of claim 1, The space-time conversion unit, A spatial transform unit which generates a residual image by removing spatial overlap of the original image by using the reference image; And A temporal converter configured to perform motion estimation corresponding to the residual image, generate a motion vector according to the executed motion estimation, and remove temporal overlap of the original image using the motion vector to generate the first transformed image Scalable video encoding apparatus comprising. The method of claim 3, wherein And an entropy encoder configured to generate a bit stream including the motion vector and the coded image information. The method of claim 3, wherein And a buffer for storing the decoded image and providing the decoded image as the reference image to the spatial transform unit. The method of claim 1, And a down sampling unit configured to low pass filter the original resolution image of the original image to generate a low resolution image. The method of claim 6, The down sampling unit, A first down sampling unit configured to low pass filter the original resolution image to generate a first low resolution image; A second down sampling unit configured to low pass filter the first low resolution image to generate a second low resolution image; And In the same manner, a scalable video encoding apparatus comprising an n-th down sampling unit configured to low pass filter an n-1 low resolution image to generate an nth low resolution image. The method of claim 6, And the low pass filtering is down sampling by a wavelet filter. The method of claim 7, wherein And the spatiotemporal transform unit proceeds separately according to different resolutions. The method of claim 9, The first converted video, Generating the original resolution converted image and the first and second low resolution converted images by removing the spatial and temporal overlap of the original resolution image and the first and second low resolution images, The first and second low resolution converted images are integrated to generate an integrated low resolution converted image, and And the integrated low resolution converted image and the original resolution converted image are integrated. The method of claim 2, And the first transformed image and the second transformed image are the same. Dequantizing the coded image information to generate a first transformed image; Generating a first residual image by performing inverse spatial transformation on the first converted image; Generating a decoded image by inversely transforming the first residual image; And providing the decoded image as a reference image. The method of claim 12, (a) generating a second converted image by removing spatial and temporal overlap of the original image using the reference image; And and (b) quantizing the second transformed image to generate the coded image information. The method of claim 13, In step (a), Generating a second residual image by removing spatial redundancy of the original image using the reference image; And Performing a motion estimation corresponding to the second residual image, generating a motion vector according to the executed motion estimation, and removing the temporal overlap of the original image using the motion vector to generate the second transformed image Scalable video encoding method comprising a. The method of claim 13, In step (b), And generating a bit stream including the motion vector and the coded image information. The method of claim 14, Storing the decoded image and providing the decoded image as the reference image. The method of claim 13, And low-pass filtering the original resolution image of the original image to generate a low resolution image. The method of claim 17, Generating the low resolution image, Low pass filtering the original resolution image to generate a first low resolution image; Low-pass filtering the first low resolution image to generate a second low resolution image; And In the same manner, a scalable video encoding method comprising low pass filtering an n-1 low resolution image to generate an nth low resolution image. The method of claim 18, The converted video, Generating the original resolution converted image and the first and second low resolution converted images by removing spatial and temporal overlap of the original resolution image and the first and second low resolution images; The first and second low resolution converted images are integrated to generate an integrated low resolution converted image, and And the integrated low resolution converted image and the original resolution converted image are integrated. The method of claim 12, And the first transformed image and the second transformed image are the same. The method of claim 14, And the first residual image and the second residual image are the same.

Images