KR100952761B1 - Apparatus for temporal scalable video coding and method thereof - Google Patents

Abstract

The present invention relates to an apparatus and method for temporal scalable video coding that is resistant to errors.

According to an aspect of the present invention, there is provided a scalability pre-processor that divides an input image differently according to a frame rate; A base layer for generating a lower layer bitstream from the first input data; One or more enhancement layers for generating a higher layer bitstream using an encoder mid-processor from the second input data. The present invention also provides a base layer for generating a first output from a lower bitstream using a decoder intermediate processor when an error occurs in a lower layer bitstream, and from an upper bitstream using an intermediate processor when an error occurs in an upper layer bitstream. One or more enhancement layers to produce a second output; And an expandable post processor for generating an output image from the first and second outputs. Therefore, according to the present invention, when an error occurs in a video service such as a wired / wireless video telephone or a VOD, the error is restored in a short time, thereby improving the quality of the image in an environment where many errors occur.

Description

Apparatus for temporal scalable video coding and method

1 is a high level block diagram of a temporal scalable video encoder.

2 is a high level block diagram of a temporal scalable video decoder.

FIG. 3 illustrates the temporal scalable video encoder of FIG. 1 in detail.

FIG. 4 is a diagram illustrating the temporal scalable video decoder of FIG. 2 in detail.

FIG. 5 is a diagram illustrating a case of having N extension layers in a high level structure of a temporal scalable video encoder. FIG.

FIG. 6 illustrates a case in which N extension layers are included in a high level structure of a temporal scalable video decoder; FIG.

7 is a flowchart illustrating a temporal scalable video coding method in an encoder of the present invention.

8 is a flowchart illustrating a temporal scalable image coding method in a decoder of the present invention.

The present invention relates to temporal scalable video coding, and more particularly, to an apparatus and method for temporal scalable video coding that is resistant to errors.

Hereinafter, the conventional and the present invention will be described.

In general, in order to improve video quality in an error-prone environment, when an error occurs in a video service such as a wired / wireless videophone or a VOD, the error should be restored in a short time.

However, in the related art, a video service using a wired / wireless communication network has a temporal scalable video coding as one of the corresponding methods since the allocated bandwidth is variable.

However, transmission errors occur in wired / wireless communication networks. The conventional temporal scalability prevents error propagation but cannot recover the error.

Therefore, in the present invention to solve the above problems, the temporal scalability of the present invention is composed of a base layer and one or more enhancement layer (enhancement layer) so that all layers can recover the error occurred in the upper or lower layer. It conveys information that can be transmitted and proposes to recover the error from the upper and lower layers if an error occurs in any layer.

The temporal scalable video coding apparatus of the present invention comprises: a scalability pre-processor for dividing an input image differently according to a frame rate; A base layer for generating a lower layer bitstream from the first input data; One or more enhancement layers for generating a higher layer bitstream using an encoder intermediate processor from the second input data.

The present invention also uses a base layer that generates a first output from a lower bitstream using a decoder intermediate processor when an error occurs in a lower layer bitstream and a mid-processor when an error occurs in an upper layer bitstream. One or more enhancement layers to generate a second output from the higher bitstream; And a scalability post-processor for producing an output image from the first and second outputs.

In addition, the temporal scalable video coding method in the encoder of the present invention comprises the steps of: configuring a base layer that is a lower layer and at least one enhancement layer that is an upper layer; Coding a synchronization frame that is the same frame at the same time as the frame of the lower layer periodically in every enhancement layer; The sync frame of the higher layer includes coding a difference between the previous frame of the same layer and the sync frame of the lower layer.

In addition, the temporal scalable video coding method in the decoder of the present invention comprises the steps of: configuring a base layer which is a lower layer and at least one enhancement layer which is an upper layer; And when an error occurs in any frame in any layer, recovering the error from the sync frame of the upper layer without errors or the frame of the lower layer without errors.

Other objects and features of the present invention will become apparent from the detailed description of the embodiments with reference to the accompanying drawings.

Hereinafter, an apparatus and method for temporal scalable video coding according to the present invention will be described with reference to the accompanying drawings.

First, the features of the present invention will be described.

Temporal scalability consists of one base layer and one or more enhancement layers.

1) The same frame is coded in all enhancement layers periodically at the same time as the frame of the lower layer. At this time, two identical frames at the same time are called sync frames .

2) A sync frame of a higher layer among a pair of sync frames codes a difference between a previous frame of the same layer and a sync frame of a lower layer.

3) If an error occurs in any frame in any layer, the error is recovered from the sync frame of the upper layer where there is no error.

4) Or, if an error occurs in any frame in any layer, the error is recovered from the frame of the lower layer without an error.

Hereinafter, a description will be given with reference to the drawings.

1 is a high level block diagram of a temporally scalable video encoder.

As shown in the figure, it shows when the output bitstream (1,0) consists of one base layer (BL) 104 and one enhancement layer (EL) 102.

When a video encoder has N extension layers, the base layer encoder of FIG. 1 indicates an extension layer n encoder, and the enhancement layer 1 encoder of FIG. 1 indicates an extension layer n + 1 encoder (n = 1 to N−). Up to 1).

Scalability pre-processor 101 divides the input image into input 0 and input 1 according to the frame rate.

That is, frames that meet the frame rate of the base layer are divided by input 0, and frames that are not otherwise divided by input 1.

The base layer encoder creates a lower layer bitstream (bitstream 0) from input zero.

The enhancement layer 1 encoder uses an encoder mid-processor 103 to create a higher layer bitstream 1 from input 1.

2 is a high level block diagram of a temporal scalable video decoder.

       As shown in the figure, this shows when the input bitstream (1,0) consists of one base layer and one extension layer.

When the video decoder has N enhancement layers, the base layer decoder 203 of FIG. 2 represents an enhancement layer n decoder, and the enhancement layer 1 decoder 201 of FIG. 2 represents an enhancement layer n + 1 decoder. (n = 1 to N-1).

The base layer decoder produces output 0 from the lower layer bitstream 0.

If an error occurs in Bitstream 0, the base layer decoder uses the decoder intermediate processor 202 to generate output 0 from the lower layer bitstream (bitstream 0).

      Meanwhile, the enhancement layer 1 decoder 201 generates an output 1 from an upper layer bitstream 1 using a decoder intermediate processor.

If an error occurs in Bitstream 1, the enhancement layer 1 decoder 201 uses the decoder intermediate intermediate processor 202 to produce output 1 from the higher layer bit stream (bitstream 1).

      Scalability post-processor 204 produces an output image from output 0 and output 1.

FIG. 3 is a detailed diagram of the temporal scalable video decoder of FIG. 1.

As shown in the figure, the operation blocks of the base layer encoder and the enhancement layer 1 encoder are basically as shown in FIG.

      The base layer encoder 303 encodes a base layer frame like a general video encoder. For example, input 0 is encoded as an I-frame or a P-frame according to the frame rate of the base layer.

     The enhancement layer 1 encoder 301 encodes input 1 using the encoder intermediate processor 302.

In this case, the encoder reference frame selector 302a determines whether to select a reference frame of the enhancement layer 1 encoder in the base layer or in the enhancement layer.

For example, when all the frames of the input 1 are encoded into P-frames, the most recently encoded frame in the base layer or the most recently encoded frame in the enhancement layer is used as the reference frame.

      The encoder sync frame selector 302b also encodes the enhancement layer frame.

      Meanwhile, since the scalability pre-processor divides the input image according to the frame rate of the base layer, the input 0 of the base layer and the input 1 of the extension layer do not have the same frame in time.

However, the encoder sync frame selector periodically encodes the frame of the enhancement layer at the same time as the frame of the base layer.

At this time, the frame of the base layer and the frame of the extension layer existing at the same time are defined as a sync frame , and this time is defined as a sync time .

This sync frame is used for error recovery in a temporally scalable video decoder. To create a sync frame, the base layer frame of sync time and the most recent extended layer frame are used. The quantized coefficients of the enhancement layer sync frame are calculated from the quantized coefficients of each frame.

Q_base layer_sync_frame = quantized coefficient of base layer sync frame

Q_extension layer_latest_frame = quantized coefficients of the most recent enhancement layer frame

Q_extended layer_sync_frame = quantized coefficients of extended layer sync frame

Q_extended_sync_frame = Q_extended_latest_frame-Q_base_sync_frame

4 is a diagram illustrating the temporal scalable video decoder of FIG. 2 in detail.

As shown in the figure, the operation blocks of the base layer decoder and enhancement layer 1 decoder are basically the same as in FIG.

The base layer decoder 403 decodes the base layer frame from bitstream 0 like a normal video decoder.

For example, the I-frame and the P-frame are decoded from the encoded bitstream 0 as described with reference to FIG. 3.

The enhancement layer 1 decoder 401 uses the decoder intermediate processor 402 to decode the enhancement layer frame from bitstream 1.

In this case, the decoder reference frame selector 402a analyzes bitstream 1 and determines whether to select a reference frame from a base layer frame or an extended layer frame.

For example, if the current enhancement layer frame is a P-frame, the reference frame is the most recent I-frame or P-frame in the base layer, or the most recent P-frame in the enhancement layer. To judge.

On the other hand, if an error occurs in the base layer bitstream 0, the next I-frame of the base layer is decoded again. This is a way to overcome the error from the frame of the base layer when an error occurs in the base layer (lower layer).

Alternatively, if an error occurs in the base layer bitstream 0, the synchronization frame is used to overcome the error. When an error occurs in the base layer (lower layer), this is a method of overcoming the error from the synchronization frame of the extended layer (higher layer).

However, if the base layer frame in which the error occurs is not a sync frame, the base layer is decoded as a reference frame based on the next sync frame of the enhancement layer.

In this case, the decoder sync frame selector transfers the enhancement layer sync frame used as the reference frame to the base layer decoder.

On the other hand, if the base layer frame in which the error occurs is a sync frame, the base layer is decoded using the current sync frame of the enhancement layer as a reference frame. In this case, the decoder sync frame selector transfers the enhancement layer sync frame used as the reference frame to the base layer decoder.

Enhancement layer 1 If an error occurs in bitstream 1, decoding is performed from the next frame of the base layer.

This is a way to overcome an error from the next frame of the base layer (lower layer) when an error occurs in the enhancement layer (higher layer).

FIG. 5 is a diagram illustrating a case of having N extension layers in a high level structure of a temporal scalable video encoder.

The encoding is done in pairs of two adjacent layers, as shown in Figs.

For example, FIG. 1 and FIG. 3 are applied in pairs of a base layer and an extension layer 1, and FIGS. 1 and 3 are applied in pairs of an extension layer 1 and an extension layer 2. FIG.

In this case, the sync frame may be matched in two or more adjacent layers.

For example, the synchronization time between the base layer and the enhancement layer 1 and the synchronization time between the enhancement layer 1 and the enhancement layer 2 may be the same.

FIG. 6 is a diagram illustrating a case of having N extension layers in a high level structure of a temporal scalable video decoder.

Decoding is performed as shown in Figs. 2 and 4 in pairs of two adjacent layers.

For example, FIGS. 2 and 4 are applied to the base layer and the extended layer 1 in pairs, and FIGS. 2 and 4 are applied to the extended layer 1 and the extended layer 2 in pairs.

If an error occurs in any layer, the error is recovered using the next I-frame in that layer as in the case of FIG.

Alternatively, if an error occurs in any layer, the error is recovered using the synchronization frame of the upper layer as in the case of FIG.

If an error occurs in any layer, the error is recovered using the next frame of the lower layer as in the case of FIG.

7 is a flowchart illustrating a temporal scalable video coding method in the encoder of the present invention.

A base layer that is a lower layer and one or more enhancement layers that are an upper layer are configured. (S 701).

In every enhancement layer, a sync frame that is the same frame is coded at the same time as the frame of the lower layer periodically. (S 702).

The sync frame of the higher layer codes the difference between the sync frame of the lower layer and the previous frame of the same layer. (S 703).

In other words, the two layers repeat the above steps S702 and S703 in pairs.

That is, the steps S702 and S703 are performed in pairs of the base layer and the extension layer 1, and the steps S702 and S703 are performed in pairs of the extension layer 1 and the extension layer 2 again, and the S702 and S703 are further paired in the extension layer 2 and the extension layer 3. To perform.

8 is a flowchart illustrating a temporal scalable video coding method in the decoder of the present invention.

A lower layer base layer and one or more enhancement layers are formed. (S 801).

When an error occurs in any frame in any layer, the error is recovered from the sync frame of the upper layer without errors or the frame of the lower layer without errors. (S 802).

Although preferred embodiments of the present invention have been described above, the present invention may use various changes, modifications, and equivalents. It is clear that the present invention can be applied in the same manner by appropriately modifying the above embodiments.

That is, in order to be compatible with the standard codec, the upper layer sync frame of the pair of sync frames is encoded using a code not used in the corresponding profile and level of the standard codec.

In this case, the standard decoder may regard the sync frame of the upper layer among the pair of sync frames as an error and ignore it.

Therefore, the above description does not limit the scope of the present invention as defined by the limitations of the following claims.

According to the present invention, when an error occurs in a video service such as a wired / wireless video telephone or a VOD, the error can be restored in a short time.

Therefore, since the error is restored in a short time, there is an advantage of improving the image quality in an environment where many errors occur.

Claims (8)

A scalability pre-processor dividing the input image differently according to the frame rate; A base layer for generating a lower layer bitstream from the first input data; One or more enhancement layers for generating an upper layer bitstream from the second input data using an encoder intermediate processor; In operation , the base layer generates a first output from the lower bitstream by using a decoder processor when an error occurs in the lower layer bitstream, and when the error occurs in the upper layer bitstream, the enhancement layer uses the decoder processor. And a scalable post processor for generating a second output from an upper bitstream by using the first and second outputs, and generating an output image from the first and second outputs . The apparatus of claim 1, wherein the encoder reference frame selector determines whether to select a reference frame of the enhancement layer 1 encoder in the base layer or in the enhancement layer. Constructing a base layer which is a lower layer and at least one enhancement layer which is an upper layer; Further comprising: periodically encoding the same frame is a sync frame in the same time as a frame of the lower layer all enhancement layer (enhancement layer) of an upper layer; The sync frame of the higher layer may include coding a difference between a previous frame of the same layer and a sync frame of the lower layer; In this case, when an error occurs in the lower layer bitstream, the base layer generates a first output from the lower bitstream using the decoder processor, and when an error occurs in the upper layer bitstream, the enhancement layer uses the decoder processor. Generating a second output from an upper bitstream and generating an output image from the first and second outputs . 4. The method of claim 3, wherein a base layer frame of synchronization time and a most recent extended layer frame are used to create a synchronization frame. 5. The method of claim 3 or 4, wherein the quantized coefficients of the enhancement layer sync frame are calculated from the quantized coefficients of each frame. A base layer for generating a first output from the lower bitstream using the decoder intermediate processor when an error occurs in the lower layer bitstream; One or more enhancement layers for generating a second output from the upper bitstream using an intermediate processor when an error occurs in the upper layer bitstream; And a scalable post processor configured to generate an output image from the first and second outputs. 7. The apparatus of claim 6, wherein the decoder reference frame selector determines whether to select a reference frame from a base layer frame or an extended layer frame after interpreting bitstream 1. Constructing a base layer that is a lower layer and one or more enhancement layers that are an upper layer; When an error occurs in any frame in any layer, restoring the error from a sync frame of the upper layer which is not in error or a frame of the lower layer which is not in error; .

Images