KR20070090494A - Inter frame error concealment method and apparatus using the average motion vector - Google Patents

Abstract

A method and an apparatus for concealing inter frame errors by using an average motion vector are provided to remove a calculation process related to a side match distortion value, ensure image quality similar to an existing method and improve its processing rate. An apparatus(300) for concealing inter frame errors by using an average motion vector comprises the following parts: a decoder(310) for restoring an inputted bit stream into a video frame; an error detector(320) for detecting an error block from the restored video frame; a peripheral block effectiveness determiner(330) for determining whether peripheral blocks of the error block are normally decoded; and an error concealer(340) which calculates the average motion vector for the effective peripheral blocks and conceals the error block by using a block of a reference frame corresponding to the calculated average motion vector.

Description

Interframe error concealment apparatus and method using average motion vector {INTER FRAME ERROR CONCEALMENT METHOD AND APPARATUS USING THE AVERAGE MOTION VECTOR}

1A is an exemplary diagram illustrating a method of interpolating an error block with values of neighboring blocks;

1B is an exemplary diagram illustrating a method of filling an error block with blocks of a previous image frame;

2 is an exemplary diagram for explaining an error concealment process using a motion vector;

3 is a schematic structural diagram showing an error concealment apparatus according to the present invention;

4 is a flowchart showing an error concealment method according to the present invention;

5A to 5C are capture screen diagrams for explaining [Experimental Example 1] of the present invention;

6a to 6c is a capture screen diagram for explaining [Experimental Example 2] of the present invention,

7A to 7C are capture screen diagrams for explaining [Experimental Example 3] of the present invention.

<Description of Symbols for Main Parts of Drawings>

300: error concealment apparatus of the present invention

310: decoding means 320: error detecting means

330: neighboring block validity determination means

340: error concealment means M: reference memory means

The present invention relates to an error concealment apparatus and a method thereof, and more particularly, to an apparatus and method for concealing inter-frame errors based on an average value calculated from motion vectors of neighboring blocks of a macroblock in which an error occurs.

In the case of H.264 or MPEG4, video data is divided and processed into macroblock (MB) units to compress and encode a video. In encoding, motion prediction and compensation, discrete cosine transform (DCT), Quantization of the DCT coefficient and variable length coding (VLC) are included, and decoding includes inverse quantization, inverse DCT, motion prediction, and compensation as the inverse of the above-described encoding.

One of the problems associated with the transmission of the compressed video (video) is that if an error occurs on the transmission channel, part or worst case of the transmitted video frame is lost, and thus the image quality of the video is greatly deteriorated. In particular, if an error occurs in one frame, the effect is applied to other frames which are restored by referring to the same.

In particular, detailed techniques (or algorithms), such as variable length coding (VLC) and motion predictive coding, included in the H.264 video compression standard, as described above, may be used in conventional compression standards such as MPEG-2 / 4. Although it guarantees a higher compression efficiency, as described above, there is a disadvantage that it is very vulnerable in a channel having a high probability of error. Specifically, in the case of variable length coding, synchronization is lost during decoding even by only one bit of error, and in the case of motion prediction coding, an error is propagated until an intra coded slice appears. error propagation).

In consideration of such a problem, in order to maintain the quality of the decoded image at a certain level or more, researches on error detection and error concealment techniques have been actively conducted. For example, when an error is detected in a system that transmits and receives a digital video signal in units of blocks (macroblocks), an error occurs when the corresponding data deviates from the syntax information by using encoded syntax information and an error occurs. The corrected blocks are corrected by using temporal or spatial concealment techniques to guarantee the final image quality.

As described above, error detection using syntax information includes a case in which a motion vector (MV) is out of a specific range, a variable length coding (VLC) table, or a DCT coefficient is lost when processing a video signal in a block unit. For example, the case may be out of a specific range, and the DCT coefficient is out of a certain number in one block.

Meanwhile, the error concealment method described above can be largely divided into an error concealment method (SEC) using temporal similarity and a temporal error concealment (TEC) method. Therefore, it is assumed that the correlation between the adjacent blocks is high and the correlation between the video frames in the adjacent time zone is high.

FIG. 1A is an exemplary diagram illustrating a method of interpolating a block in which an error occurs (hereinafter referred to as an 'error block') using values of neighboring blocks, and schematically illustrates the spatial error concealment method (SEC). . That is, the error block is concealed based on information of neighboring blocks having spatial similarity with the error block in one image frame. In this case, the neighboring block used for interpolation should be a normally decoded block.

FIG. 1B is a view illustrating a method of filling an error block with a block of a corresponding position of a previous image frame, and using the temporal similarity between an n-frame and an n-1 frame in which an error occurs. Represents a temporal error concealment (TEC) method. In this case, in order to replace an error block by taking a predetermined block from a previous image frame, a motion vector (MV) of the error block must be estimated. The motion vector is estimated using the motion vectors of the neighboring blocks, and a process of replacing a block of the previous image frame (n-1 frame) corresponding to the estimated motion vector with an error block is performed.

In other words, the above-described spatial error concealment (SEC) method is mainly applied when an error occurs in an I-frame, i.e., a picture that can be decoded using only data in a bit stream for a current video frame, and temporal error concealment (TEC). ) Is mainly applied when an error occurs in a P-frame (predictive frame, inter frame), that is, a screen decoded with reference to a previous image frame.

Among the above-described temporal error concealment (TEC) methods, when the inter-frame error concealment of H.264 is examined in detail, it is based on BMA (Boundary Matching Algorithm). Find blocks of reference picture frames that can be minimized, choose to minimize distortion among candidate motion vectors (MVs) according to BMA matching criteria, or error blocks and candidates of current picture frames based on side matching techniques (SMT). The difference between the absolute values of the blocks on the reference image frame found by the motion vector is calculated and selected.

The boundary portion distortion may be represented by d _sm as shown in Equation 1 below.

[Equation 1]

는 인접한 블록 간에 경계와 맞닿아 있는 블록의 움직임 벡터를 가지고 움직임 보상을 통해 얻은 블록의 픽셀 값이며, 'OUT'에 위치한

는 복원하려는 블록의 경계와 맞닿아 있는 이웃하는 블록들의 픽셀 값을 의미한다.Where N is the total number of pixels used to find d _sm and is located at 'IN' (see FIG. 2).

Is the pixel value of the block obtained by motion compensation with the motion vector of the block that is in contact with the boundary between adjacent blocks, and is located at 'OUT'.

Denotes pixel values of neighboring blocks that are in contact with the boundary of the block to be restored.

At this time, the left, right, up, and down blocks (①, ②, ③, ④) of the eight neighboring blocks around the block in which the error occurs as shown in FIG. 2 are divided into four again, and at least two of the eight candidates are moved. Using the vector (MV) to calculate the d _sm , [Equation 1] is to calculate the d _sm for one block of 16 × 16 size of the 'IN' block and 'OUT' block The absolute difference is calculated. Here, a total of 64 operations are performed with 16 pixels each of Top, T, Bottom, B, Left, L, and Right, and each pixel is executed. One subtraction operation, one absolute value operation, and one division operation for calculating the difference of three times are performed per pixel.

Therefore, the number of operations per block to calculate d _sm is 64 pixels × 3, which is 192 times. Since H.264 uses up to 8 candidate motion vectors, the maximum number of operations is 192 × 8 = 1,536 times.

Of course, there is an advantage of minimizing image deterioration and concealing errors by finding an optimal block using a plurality of candidate motion vectors, but the number of operations according to this is an apparatus that performs interframe error concealment, in particular, a mobile communication terminal. It can be an overload factor in terms of execution speed. Moreover, it is obvious that the environment may require anxiety that cannot be tolerated in an environment requiring real-time response.

The present invention was devised to solve the above-described problem, and does not require a calculation process for boundary distortion (d _sm ), and provides an average motion vector that can improve the processing speed while ensuring a similar image quality level compared to the conventional method. An inter frame error concealment apparatus and method are provided.

The features and advantages of the present invention will become more apparent from the following detailed description based on the accompanying drawings. Prior to this, the terms or words used in the present specification and claims are defined in the technical spirit of the present invention on the basis of the principle that the inventor can appropriately define the concept of the term in order to explain his invention in the best way. It should be interpreted to mean meanings and concepts. In addition, when it is determined that the detailed description of the known function and its configuration related to the present invention may unnecessarily obscure the subject matter of the present invention, it should be noted that the detailed description is omitted.

3 is a schematic block diagram showing an inter frame error concealment apparatus (hereinafter, 'error concealment apparatus') 300 using an average motion vector according to the present invention. As shown in FIG. Decoding means 310 for reconstructing the input image bit stream into an image frame, and error detection means 320 for detecting a macroblock (hereinafter, referred to as an "error block") in which an error occurs in the reconstructed image frame. And the neighboring block validity determining means 330 for determining whether the neighboring blocks of the error block are correctly decoded, and calculating an average motion vector for the neighboring blocks and concealing the error using the corresponding macroblock of the reference frame. And error concealment means 340 for outputting the error concealed image frame, and reference memory means M for storing a plurality of decoded image frames.

Here, neighboring blocks of the error block are used as candidate motion vectors, and a block (macroblock, MB) is set to have a size of 16 × 16 pixels. Therefore, there are four candidate motion vectors in 16x16 mode and eight in 8x8 mode. That is, four to eight candidate motion vectors are used in the present invention.

Also, the candidate motion vector is selected from the neighboring blocks of the error block detected by the error detecting means 320, and preferably from the neighboring blocks correctly decoded by the neighboring block validity determining means 330.

를 각각 매크로블록의 수직과 수평 좌표라 하고,

를 매크로블록

의 픽셀 좌표라 할 때, 상기 매크로블록

의 픽셀 좌표는

,

로 표현할 수 있다. In order to explain the function of the error concealment means 340 according to the characteristic aspect of the present invention, the above-described BMA-based interframe error concealment method will be described as follows.

Are the vertical and horizontal coordinates of the macroblock,

Macroblock

When the pixel coordinate of the macroblock,

The pixel coordinate of is

,

Can be expressed as

는 현재 영상 프레임의

좌표에 있는 픽셀 값을 나타내며,

는 참조 프레임의

좌표에 있는 픽셀 값을 나타낸다. 움직임 벡터가

일 때, 참조 프레임에서 수직, 수평 방향으로 각각

,

만큼 이동한 매크로블록의 픽셀 좌표

는

,

이다.

를 각각 위, 아래, 왼쪽, 오른쪽 경계 방향의 경계 왜곡치(boundary match distortion)이라 하면, 움직임 벡터

에 대한 경계 왜곡치들은 다음과 같다.

Of the current video frame

Represents a pixel value in coordinates.

Of the reference frame

Represents a pixel value in coordinates. Motion vector

In the vertical and horizontal directions in the reference frame, respectively.

,

Pixel coordinates of macroblock moved by

Is

,

to be.

Is a boundary match distortion in the up, down, left, and right boundary directions, respectively,

The boundary distortion values for are as follows.

의 합으로 산출되며, 그 중 가장 작은 경계 왜곡치를 갖는 움직임 벡터를 에러 은닉을 위한 움직임 벡터로 선택하게 된다. 그러나 전술한 바와 같이 후보 움직임 벡터를 최대 8개까지 사용하는 경우 최대 연산횟수는 192×8=1,536회에 달한다.Boundary distortion

The motion vector having the smallest boundary distortion value is selected as the motion vector for error concealment. However, as described above, when up to eight candidate motion vectors are used, the maximum number of operations reaches 192 × 8 = 1,536 times.

On the other hand, the error concealment means 340 of the present invention calculates an average motion vector from the candidate motion vectors (the motion vectors of the correct neighboring blocks of the error block), and extracts the macroblock of the reference frame corresponding to the calculated average motion vector. Used for error concealment.

라 하고, 후보 움직임 벡터의 개수가 n 일 때, 상기 평균 움직임 벡터(

)는 다음의 [수학식 2]로 산출될 수 있다.M-th motion vector of the candidate motion vectors

When the number of candidate motion vectors is n, the average motion vector (

) Can be calculated by the following Equation 2.

[Equation 2]

That is, the present invention performs interframe error concealment using the average motion vector calculated through Equation 2 without the above-described process of calculating the boundary distortion value. For reference, the number n of the candidate motion vectors may be easily counted by the neighboring block validity determining means 330.

Referring to FIG. 4, a series of error concealment processes described above will be described. The decoding means 310 restores an input bitstream to an image frame (S410), and the error detecting means 320 generates an error block in the restored image frame. After detecting the macroblock in which an error has occurred (S420), the neighboring block validity determining unit 330 determines whether the neighboring block of the error block detected in the step S420 is normally decoded (S430).

Subsequently, the error concealment means 340 calculates an average motion vector of valid neighboring blocks based on Equation 2 described above (S440), and references a block corresponding to the calculated average motion vector with a reference frame (preferably a reference memory). The error concealment is performed by extracting the data from the temporary block, and inserting it into the error block (S450).

On the other hand, the process (S410 ~ S450) is implemented as a program, it may be stored in a computer-readable recording medium format.

Hereinafter, the inter frame error concealment method and the H.264 inter frame error concealment method according to the present invention will be considered in detail through several experimental examples.

Experimental Example 1-'Mother and Daughter' simulation.

In this experimental example, QP is set to 22 and GOP is set to I and P, and then encoded into 100 frames, and packet loss occurs in one of two slices of the P-frame. ) Is 50 frames (applies only to P-frames).

FIG. 5A shows an image of 'Mother and Daughter' in which the above packet loss occurs, FIG. 5B shows an image after the H.264 built-in error concealment method is applied, and FIG. 5C shows an error concealment method according to the present invention. The image after application is shown.

Experimental Example 2-'Foreman' simulation.

In this Experimental Example, the simulation was performed under the same conditions as in [Experimental Example 1], FIG. 6A is an image before error concealment is performed, and FIG. 6B is an image after the H.264 built-in error concealment method is applied. 6C shows an image after the error concealment method according to the present invention is applied.

Experimental Example 3-'Bus-cif.yuv' simulation.

In this experimental example, a total of 100 frames were formed by configuring only the QP and the GOP as I and P, and packet loss was generated in one of two slices of the P-frame. The frame to which packet loss is applied is 50 frames out of 100 frames using only P-frames.

7A is an image before error concealment is performed, FIG. 7B is an image after the H.264 basic error concealment method is applied, and FIG. 7C is an image after the error concealment method according to the present invention is applied.

As shown in [Experimental Example 1] to [Experimental Example 3], the overall image quality of the image after applying the H.264 standard error concealment method and the image after the error concealment method using the average motion vector of the present invention is applied It can be seen that they are so similar that they cannot be identified.

According to the present invention, no calculation related to the boundary distortion value is required, and the processing speed can be improved while ensuring a similar image quality level as compared with the conventional method (basic error concealment method of H.264).

As described above and described with reference to a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, it is a deviation from the scope of the technical idea It will be understood by those skilled in the art that many modifications and variations can be made to the invention without departing from the scope of the invention. Accordingly, all such suitable changes and modifications and equivalents should be considered to be within the scope of the present invention.

Claims (5)

An apparatus for inter frame error concealment, Decoding means for reconstructing the input bitstream into an image frame; Error detecting means for detecting an error block in the reconstructed image frame; Neighboring block validity determining means for determining whether neighboring blocks of the error block are correctly decoded; And Error concealment means for calculating an average motion vector for valid neighboring blocks and concealing the error block using a block of a reference frame corresponding to the calculated average motion vector; Inter-frame error concealment apparatus using an average motion vector comprising a. The method of claim 1, Reference memory means (M) for storing a plurality of decoded image frames; Inter frame error concealment apparatus using the average motion vector, characterized in that further comprises. The method of claim 1, The mean motion vector (

), M-th motion vector of the candidate motion vectors

If the number of candidate motion vectors is n,

Inter frame error concealment apparatus using the average motion vector, characterized in that calculated by. An error concealment method using the error concealment apparatus according to claim 1 or 3, Restoring the input bitstream into an image frame; Detecting an error block in the reconstructed image frame; Determining validity of neighboring blocks of the detected error block; Calculating an average motion vector from motion vectors of valid neighboring blocks; And Performing error concealment by extracting a block corresponding to the calculated average motion vector from a reference frame; An interframe error concealment method using an average motion vector, comprising: a. A computer readable recording medium for performing inter frame error concealment, A decoding function for restoring an input bitstream into an image frame; An error detection function for detecting an error block in the reconstructed image frame; A neighboring block validity determination function for determining whether neighboring blocks of the error block are correctly decoded; And An error concealment function for calculating an average motion vector for valid neighboring blocks and concealing the error block using a block of a reference frame corresponding to the calculated average motion vector; Recording medium that stores a program for implementing.

Images