KR100731884B1 - Method and Apparatus for a Temporal Scalable Encoding/Decoding Based on Multiple Reference Frames - Google Patents

Abstract

The present invention relates to a method and apparatus for temporal scalable encoding and decoding referencing a plurality of frames.

According to the present invention, when encoding / decoding video information to have temporal scalability, the same extended frame is determined by determining the frame to be referred to by the extended frame in units of macroblocks constituting the frame, rather than determining the frame to be referred to. Each of the constituent macroblocks can be individually selected as a reference frame. And, this has the effect of further improving the compression ratio than the prior art.

Description

Temporal scalable encoding and decoding method for referencing a plurality of frames, and apparatus therefor {Method and Apparatus for a Temporal Scalable Encoding / Decoding Based on Multiple Reference Frames}

1 is a schematic diagram of a digital broadcasting system;

2 is a schematic diagram of a temporal scalable encoding / decoding method for referring to a single frame according to the prior art;

3 is a view for explaining a frame structure used in the present invention,

4 is a schematic diagram of a temporal scalable encoding / decoding method for referring to a plurality of frames according to the present invention;

FIG. 5 is a diagram illustrating a process in which one frame refers to a plurality of frames in macroblock units; FIG.

6 is a configuration diagram of a temporal scalable encoding apparatus for referring to a plurality of frames according to the present invention;

7 is a configuration diagram of a temporal scalable decoding apparatus for referring to a plurality of frames according to the present invention;

8 is a diagram illustrating a process of transmitting frame information through a heterogeneous network.

<Description of Symbols for Main Parts of Drawings>

100, 110: digital broadcasting center 200: communication network

210: first communication network 220: second communication network

310: first terminal 320: second terminal

400: Frame 500, 510, 520, 530: Macroblock

600: encoding device 610: video information input unit

620: frame extraction unit 630: frame information generation unit

640: frame information output unit 700: decoding device

710: frame information input unit 720: frame generation unit

730: frame combination

The present invention relates to a method and apparatus for temporal scalable encoding and decoding referencing a plurality of frames. More particularly, the present invention relates to a video to have temporal scalability for digital broadcasting services such as satellite digital multimedia broadcasting (DMB). A temporal scalable encoding and decoding method and apparatus for referencing a plurality of frames that can improve a compression rate by allowing a reference to a plurality of frames when compressing information (data).

In general, since video information (data) has a large capacity, it is effectively compressed and transmitted or stored. As a standard of the video information (data) compression technology, moving picture MPEG (Moving Picture) of ISO / IEC (International Data Communication / International Electrotechnical Commission) Experts Group) and ITU-T (International Telecommunication Union-Telecommunication Session) H.26x series. In relation to digital broadcasting, "H.264 / MPEG4 part 10" proposed by JVT (Joint Video Team) jointly formed by "MPEG4 part2 video" or MPEG group of ISO / IEC and VCEG group of ITU-T. AVC "is mainly used.

Video compression technology takes advantage of the fact that the statistical characteristics of video data have a lot of spatial and temporal redundancy.In general, to reduce spatial redundancy, a discrete cosine transform (DCT) is performed on a frame-by-frame basis. In order to reduce temporal redundancy, differential pulse coded modulation (DPCM) is performed by searching for continuity and similarity between images from the reference image that has been encoded for each frame.

As encoding modes for video compression, there are an intra mode and an inter mode. Intra mode is a mode that converts and encodes based on continuity and similarity included in the original image itself. In addition, the inter mode is a mode for encoding based on continuity and similarity between images, which is a predictive mode that predicts motion from an existing image in time and compares it with the original image to encode a prediction error. It may be classified into a bi-predictive mode, which predicts a motion from an image existing before or after time, and compares it with an original image to encode a prediction error.

1 and 2, the outline of the digital broadcasting system and the temporal scalable encoding / decoding technology according to the prior art will be described briefly. As shown in FIG. 1, the digital broadcasting center 100 includes a satellite DMB network or a terrestrial DMB network. Digital broadcasting has been performed in such a manner that the video information is transmitted using the network 200 and the DMB receiving terminals 310 and 320 receive and reproduce the same. On the other hand, although not commercially available in current digital broadcasting, there is a technology to have temporal scalability in encoding / decoding so that frame playback speed can be selected for video playback for high quality broadcasting.

Referring to FIG. 2, the frames constituting the video include a plurality of basic frames B '(0), ..., B' (n-4), B '(n-2), and B for basic quality viewing. '(n), B' (n + 2), ...) and a plurality of extended frames T '(1), ..., T' ( n-3), T '(n-1), T' (n + 1), ...) and encodes the basic frame and the extended frame, respectively, to generate the basic frame information and the extended frame information. During playback, temporal scalability is achieved by decoding only basic frame information or by decoding and combining both basic frame information and extended frame information.

Looking at the processing technique for the frame for this purpose, the first frame (B '(0)) of the basic frame is processed in the intra mode from the original image, the subsequent basic frames (B' (2), ..., B ' (n-4), B '(n-2), B' (n), B '(n + 2), ...) are mainly used in inter mode by comparing continuity and similarity with previous or subsequent frames. If the continuity and similarity with the processed or neighboring frames is low (e.g., when the scene is switched), it is processed in intra mode.

Meanwhile, separate extended frame information is generated for high-definition viewing having a better frame playback speed than the reference frame. As shown in FIG. 2, extended frame T '(n + 1) is generally the previous basic frame B'. (n) or later (inter mode processing) with reference to base frame B '(n + 2) or with reference to previous extended frames T' (n-3), T '(n-1) Inter mode processing). The extended frames T '(1), ..., T' (n-3), T '(n-1), T' (n + 1), and the like generated in this way are respectively the basic frames B '(0). By inserting between B '(n-4), B' (n-2) and B '(n), the frame reproduction speed is doubled.

However, the temporal scalable encoding / decoding technique according to the prior art has determined a frame to refer to an extended frame on a frame basis in inter mode (eg, forward prediction mode, bidirectional prediction mode) processing, and thus there is a limit in improving the extrusion rate. there was.

The present invention was devised to solve the above problems of the prior art, and an object of the present invention is to individually refer to macroblocks constituting the same extended frame when compressing moving picture information (data) to have temporal scalability. The present invention provides a method and apparatus for temporally scalable encoding and decoding capable of improving a compression ratio by referring to a plurality of frames.

For this purpose, a temporal scalable encoding method for referring to a plurality of frames according to the present invention includes: a) a base frame for viewing a basic image with respect to moving image information, and an extension for high-definition viewing with a higher frame reproduction speed than the basic image. Extracting a frame; And b) generating basic frame information by encoding the basic frame in macroblock units and generating extended frame information by encoding the extended frame in macroblock units, wherein macroblocks constituting the same extended frame are individually referenced frames. And generating the basic frame information and extended frame information so that one frame refers to a plurality of frames by selecting.

In addition, the temporal scalable decoding method for referring to a plurality of frames according to the present invention includes: (i) basic frame information capable of reproducing a basic frame for viewing a basic image quality, and an extension for high-definition viewing having a higher frame reproduction rate than the basic image quality. Receiving extended frame information capable of playing a frame; Ii) generating a basic frame by decoding the basic frame information in units of macroblocks, and generating an extended frame by decoding the extended frame information in units of macroblocks, wherein macroblocks constituting the same extended frame each separately refer to a reference frame; Selecting one frame to generate the base frame and the extended frame by referring to a plurality of frames; And iii) combining the basic frame and the extended frame.

On the other hand, the temporal scalable encoding apparatus for referencing a plurality of frames according to the present invention, the video input unit to which the video information is input; A frame extracting unit for extracting a basic frame for viewing a basic quality from the video information and an extended frame for high quality viewing having a higher frame reproduction speed than the basic quality; By generating the basic frame information by encoding the basic frame in macroblock units and generating the extended frame information by encoding the extended frame in macroblock units, macroblocks constituting the same extended frame are individually selected by reference frames. A frame information generation unit generating the basic frame information and the extended frame information so that one frame refers to a plurality of frames; And a frame information output unit configured to output the basic frame information and extended frame information.

In addition, the temporal scalable decoding apparatus referencing a plurality of frames according to the present invention includes basic frame information capable of reproducing a basic frame for viewing a basic image quality and extended frames for high quality viewing having a higher frame reproduction speed than the basic quality. A frame information input unit for receiving extended frame information which can be reproduced; By generating the basic frame by decoding the basic frame information in macroblock units, and generating an extended frame by decoding the extended frame information in macroblock units, macroblocks constituting the same extended frame are individually selected by reference frames. A frame generator configured to generate the basic frame and the extended frame with reference to a plurality of frames by one frame; And a frame combination unit for combining the generated basic frame and the extended frame.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings and preferred embodiments. In the following description, the same or similar components are denoted by the same reference numerals as much as possible, and detailed descriptions of well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention are omitted.

Prior to the detailed description, the terms used in the present invention are simply defined, and the term "basic quality" refers to a minimum quality (or standard quality) provided for watching a video. For example, the resolution and Refers to picture quality of frame rate of QVGA (320 x 240) and 15 fps, respectively. In addition, the image quality having a higher frame reproduction speed than the basic image quality is referred to as "high image quality". Similarly, in the case of satellite DMB, the image quality of QVGA / 30fps, QVGA / 45fps, QVGA / 60fps, etc. is referred to.

3 is a view for explaining a frame structure used in the present invention. A frame representing one screen (image) is divided into a plurality of macroblocks. In FIG. 3, four macroblocks MB_0 (500), MB_1 (510), BM_2 (520), and MB_3 (530) are one. An example of configuring the frame 500 is shown.

The main technical idea of the present invention is that video information (data) is encoded / decoded for each macroblock constituting a frame as described above, which will be described in detail below.

4 is a schematic diagram of a temporal scalable encoding / decoding method for referring to a plurality of frames according to the present invention.

The input or transmitted video information is extracted as a base frame for viewing a basic picture for temporal scalability and an extended frame for high quality viewing for the base picture. These base frames form a so-called "Base Layer" and extension frames form a so-called "Enhancement Layer".

The first frame of the basic frame is generally processed in the intra mode from the original image, and subsequent frames are mainly processed in the inter mode by comparing the continuity and similarity with previous or subsequent frames, and the continuity and similarity with neighboring frames are compared. If it is low (eg when the scene is switched), it is processed in intra mode.

Although not shown in detail with reference to FIG. 4, assuming that B (0) is the first frame, B (0) is composed of four macroblocks, and each macroblock can be referred to as a frame (reference frame). Since it does not exist, it is processed in intra mode from the original image itself. On the other hand, B (n), which is the nth basic frame, is similarly composed of four macroblocks, and each macroblock has neighboring basic frames B (n-4), B (n-2), and B (n + 2). ) And continuity and similarity are compared with the macroblocks constituting the &lt; RTI ID = 0.0 &gt; etc., &Lt; / RTI &gt;

FIG. 5 illustrates a process in which one frame refers to a plurality of frames in macroblock units, and macroblocks constituting the n-th frame are compared with the macroblocks constituting the previous frame and continuity and similarity, respectively. The processing in the inter mode is shown with reference to the most similar macroblock. In FIG. 5, for example, a first macroblock of macroblocks constituting an nth frame refers to a macroblock constituting an n-2nd frame, and a second macroblock refers to a macroblock constituting an n-1th frame. For reference, the third macroblock refers to a macroblock constituting the n-4th frame, whereby macroblocks constituting the same frame (eg, the nth frame of FIG. 5) are individually selected by selecting a reference frame. Frames are encoded / decoded with reference to a plurality of frames.

To this end, in the basic frame information, frame information representing the position or order of the frame among all the basic frames, reference macroblock information representing a macroblock referred to by the macroblock during encoding / decoding, and reference macroblocks A motion vector for moving to a position and prediction error information for reconstructing the macroblock by performing motion compensation on the reference macroblock are included.

Meanwhile, separate extended frame information is generated for high-definition viewing with better frame reproduction speed than the basic quality. The extended frame is inserted between the reference frames to improve the frame playback speed per hour. For example, the extended frames T (n-3), T (n-1), and T (n + 1) shown in FIG. 4 are the basic frames B (n-4), B (n-2), and B (n, respectively). Inserted after) to double the frame playback speed.

The extension frames forming the enhancement layer are processed in an inter mode from the base frame of the base layer or an extension frame of its own (extension layer). Like the base frame, each extension frame is composed of macroblocks. For example, the extended frame T (n + 1) consists of four macroblocks, each of which refers to a macroblock constituting a neighboring basic frame B (n) or B (n + 2) or moves in time. It is processed in an inter mode with reference to a macroblock constituting an extended frame T (n-3) or T (n-1) located at.

Similarly, in the extended frame information, frame information, reference macroblock information representing a macroblock referenced by the corresponding macroblock during encoding / decoding, and movement for moving the reference macroblock to the correct position so that extended frames can be sequentially formed. A vector and prediction error information for reconstructing the macroblock by performing motion compensation on the reference macroblock are included.

6 is a block diagram of a temporal scalable encoding apparatus for referring to a plurality of frames according to the present invention. The encoding apparatus 600 according to the present invention includes a video input unit 610 to which video information (data) is largely input, a frame extractor 620 which extracts a basic frame and an extended frame from the video information, and the basic frame and an extended frame. Frame information generation unit 630 for generating basic frame information and extended frame information by encoding the macroblock in macroblock units, and a frame information output unit 640 for outputting the basic frame information and extended frame information.

The moving picture information or moving picture data is information (data) generated by a video camera, a movie shooting device, or the like, and, in principle, refers to information that has not yet been compressed (encoded). However, even if data compressed (encoded) information is decompressed (decoded) when inputted to the video input unit 610 or information capable of generating (restored) an original frame, it is included in the video information according to the present invention. something to do.

Video information input to the video input unit 610 is transmitted to the frame extractor 620, the frame extractor 620 is a high quality frame playback rate higher than the basic frame and basic quality for viewing the basic image from the video information Extract the extended frame for viewing. For example, the basic frame is extracted with respect to the video information at a predetermined basic period, and the extended frame except for the basic frame is extracted at a period faster than the basic period (for example, 1/2 times the basic period). The basic frame and the extended frame are transmitted to the frame information generator 630, and the frame information generator 630 encodes them in units of macroblocks to generate basic frame information and extended frame information. As described in detail above, macroblocks constituting the same frame at encoding each individually select a reference frame, and thus, frame information is generated with reference to a plurality of frames in one frame. On the other hand, the frame information generation unit 630, the basic frame information generation unit for generating the basic frame information by encoding the basic frame in macroblock units and the extended frame information for generating extended frame information by encoding the extended frame in macroblock units It is also preferably divided into a generation unit, and configured to generate basic frame information and extended frame information, respectively. Finally, the frame information output unit 640 outputs the basic frame information and the extended frame information generated in this way to the outside.

The detailed process of generating the basic frame information and the extended frame information from the basic frame and the extended frame may refer to the above-described encoding method, and thus redundant description thereof will be omitted.

So far, a temporal scalable encoding method and apparatus for referencing a plurality of frames have been described. Hereinafter, a method and apparatus for decoding basic frame information and extended frame information encoded by the above-described process will be described.

For convenience of description, a decoding method and apparatus for referring to a plurality of frames according to the present invention will be described with reference to FIG. 7.

The decoding apparatus 700 according to the present invention largely includes a frame information input unit 710, a frame generator 720, and a frame combination unit 730.

The frame information input unit 710 receives or inputs basic frame information capable of playing a basic frame for viewing a basic picture quality from the outside and extended frame information capable of playing an extended frame for a high quality viewing having a higher frame reproduction speed than the basic picture quality. Receive. The received (input) basic frame information and extended frame information are decoded in units of macroblocks by the frame generator 720, and are restored to the basic frame and the extended frame before encoding. That is, during decoding, the frame is restored based on the reference macroblock information, the motion vector, and the prediction error information included in each frame information. In this case, the macroblocks constituting the same extended frame are individually separated by the reference macroblock information. By selecting a reference frame (exactly a macroblock), one frame is decoded with reference to a plurality of frames. The frame generator 720 is divided into a basic frame generator for generating basic frames by decoding basic frame information in macroblock units and an extended frame generator for generating extended frames by decoding extended frame information in macroblock units. It is also preferred that each of them be configured to generate a base frame and an extended frame separately.

The generated basic frame and the extended frame are combined in the frame combination unit 730 to generate original video information (data). On the other hand, when combining the basic frame and the extended frame in the frame combination unit 730, depending on the type of the terminal that can play the user selection or video, only the basic frames can be output to watch the basic quality or the basic frame and the extended frame It is desirable to output all of them so that they can be selected to enable high quality viewing. That is, the frame combination unit 730 may be implemented to provide a video of a desired quality by combining only the basic frame or combining the basic frame and the extended frame by the image quality selection information input by the user. Can be.

Since the temporal scalable decoding method and apparatus referring to the plurality of frames described above are to decode basic frame information and extended frame information generated by the temporal scalable encoding method and apparatus referring to the plurality of frames described above, For details, refer to the contents of the encoding method and apparatus.

Meanwhile, as shown in FIG. 8, when the present invention is applied to digital broadcasting such as satellite DMB, basic frame information and extended frame information generated by the encoding apparatus are respectively transmitted to the user terminal through different communication networks. desirable. For example, basic frame information is transmitted through a first communication network specialized for digital broadcasting (eg, satellite DMB network, terrestrial DMB network, etc.) 210, and extended frame information is not a professional digital broadcasting network but is a commercially available system. By transmitting through the two communication networks (eg, wireless Internet, wired Internet, mobile communication network, etc.) 220, it is possible to reduce the load of the first communication network.

In addition, the DMB receiving terminals 310 and 320 receive only basic frame information according to its performance (function) or user selection to provide basic quality broadcasting or receive both basic frame information and extended frame information to broadcast in high quality. Can be provided.

Although the present invention has been described in detail with reference to preferred embodiments, it will be apparent to those skilled in the art that the present invention can be implemented in other specific forms without changing the technical spirit or essential features thereof. One embodiment is to be understood in all respects as illustrative and not restrictive.

In addition, the scope of the present invention is specified by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts are included in the scope of the present invention. Should be interpreted as

According to the present invention, when encoding / decoding video information to have temporal scalability, the macroblock constituting the same extended frame is determined by determining the frame to be referred to by the extended frame in units of macroblocks constituting the frame, rather than in units of frames. Each of them has an effect of further improving the compression ratio by selecting reference frames individually.

Claims (10)

a) extracting a basic frame for viewing a basic picture and extended frames for high quality viewing having a higher frame reproduction speed than the basic picture with respect to the video information; And b) generating basic frame information by encoding the basic frame in macroblock units, and generating extended frame information by encoding the extended frame in macroblock units, wherein macroblocks constituting the same extended frame are individually Generating the basic frame information and extended frame information such that one frame refers to a plurality of frames by selecting. The method of claim 1, and c) outputting the basic frame information and the extended frame information to different communication networks. The method according to claim 1 or 2, The basic frame information includes reference macroblock information, a motion vector, and prediction error information. The method according to claim 1 or 2, The extended frame information includes reference macroblock information, a motion vector, and prediction error information. Iii) receiving basic frame information for reproducing the basic frame for viewing the basic picture quality and extended frame information for reproducing the extended frame for high quality viewing having a higher frame reproduction speed than the basic picture quality; Ii) generating a basic frame by decoding the basic frame information in units of macroblocks, and generating an extended frame by decoding the extended frame information in units of macroblocks, wherein macroblocks constituting the same extended frame each separately refer to a reference frame; Selecting one frame to generate the base frame and the extended frame by referring to a plurality of frames; And And v) combining the base frame and the extended frame. A video input unit to which video information is input; A frame extracting unit for extracting a basic frame for viewing a basic quality from the video information and an extended frame for high quality viewing having a higher frame reproduction speed than the basic quality; By generating the basic frame information by encoding the basic frame in macroblock units and generating the extended frame information by encoding the extended frame in macroblock units, macroblocks constituting the same extended frame are individually selected by reference frames. A frame information generation unit generating the basic frame information and the extended frame information so that one frame refers to a plurality of frames; And And a plurality of frames including a frame information output unit configured to output the basic frame information and extended frame information. The method of claim 6, The frame information generator is divided into a basic frame information generator for generating basic frame information by encoding the basic frame in macroblock units and an extended frame information generator for generating extended frame information by encoding the extended frame in macroblock units. And a temporal scalable encoding apparatus referencing a plurality of frames. A frame information input unit configured to receive basic frame information for reproducing a basic frame for viewing a basic quality and extended frame information for reproducing an extended frame for a high definition viewing having a higher frame reproduction speed than the basic quality; By generating the basic frame by decoding the basic frame information in macroblock units, and generating an extended frame by decoding the extended frame information in macroblock units, macroblocks constituting the same extended frame are individually selected by reference frames. A frame generator configured to generate the basic frame and the extended frame with reference to a plurality of frames by one frame; And And a plurality of frames including a frame combination unit for combining the generated basic frame and the extended frame. The method of claim 8, The frame generator is divided into a basic frame generator for generating basic frames by decoding basic frame information in macroblock units and an extended frame generator for generating extended frames by decoding extended frame information in macroblock units. And a temporal scalable decoding apparatus referencing a plurality of frames. The method according to claim 8 or 9, And the frame combination unit refers to a plurality of frames, wherein only the basic frame is combined or the basic frame and the extended frame are combined together according to image quality selection information.

Images