TWI412282B - Video decoding method and apparatus for concealment of transmission errors - Google Patents

Abstract

This invention is related to a video decoding method, based on dynamic programming (DP), and apparatus for concealment of transmission errors. The goal is targeted at the improvement of traditional boundary matching algorithm (BMA). The steps of the method include: decode the received video bit stream and perform error detection; for each detected erroneous macroblock (MB) of 16x16 pixels, perform an improved BMA to calculate the initial motion vectors; input the above-mentioned initial motion vectors to the DP process (based on 16x16-pixel blocks) to refine the motion vectors; perform an iterative DP process (based on 8x8-pixel block) for further accurately exploring the motion activity (up to 4 motion vectors for 8x8 pixel blocks) of each 16x16 pixel macroblock; decide whether to merge the motion vectors of the 8x8 pixel blocks; perform motion compensation based on the motion vectors of the merged 8x8, 8x16, 16x8, or 16x16 blocks. Our invention adds an edge cost (or, the so-called smoothness cost), and an iterative DP process, with respect to the traditional BMA, hence, a better reconstruction quality on adjacent boundaries of erroneous macroblocks can be obtained.

Description

Video decoding device and related method capable of hiding transmission errors

The present invention relates to a decoding apparatus and method for concealing (or replying) an error at a receiving end when a compressed video is transmitted over a network (wireless or wired) and an error occurs during transmission to increase the quality of the video display. .

The core technology of video compression today is to remove redundant data on the spatial domain and time domain of the video. The video compression technique in the spatial domain mainly uses the combination of Discrete Cosine Transform (DCT) and Quantization, while the compression technique in the time domain uses Motion Estimation and Motion Compensation. (Motion Compensation) A combination of the two methods. After receiving the signal, the receiving end must be decompressed to restore the original video, and decompression is the inverse of the compression, so the inverse discrete cosine transform and inverse inverse Quantization must be used to restore the spatial domain. The compression signal is used, and motion compensation is used to restore the compressed signal in the time domain.

On the ideal transmission channel, the theoretically compressed data can be effectively transmitted from the transmission end to the receiving end without error, but in reality, most of the transmission channels are not ideal channels, and the receiving end is often interfered by noise to cause the receiving end. The signal received is not a complete and correct signal. When the multimedia video data is highly compressed, the data may be lost or incorrectly generated during the signal transmission. In addition to causing the receiving end to not decode the correct image, it may also be seriously caused by the correlation of the compressed video in the spatial domain and the time domain. The error propagation phenomenon. Error propagation can be divided into two types: spatial domain and time domain. The spatial domain error spread is that for the time t image, the space coordinate ( x ₀ ( t ), y ₀ ( t )) pixel error will cause other coordinate pixels in the future. mistake. The spread is the time domain error for time t, the time t error occurred will cause a later time t video error.

The interference of the wireless transmission channel is mainly caused by the multipath Propagation effect of electromagnetic waves. The multi-path transmission effect is because when the electromagnetic wave is transmitted wirelessly, there are many obstacles in the channel, such as weather or building, etc., and refraction, diffraction and scattering occur, so the receiving end receives the direct signal. At the same time, it receives signals that are affected by the above three phenomena. The signal received by the receiving end is inconsistent, and this phenomenon is called Fading. The intensity change of the signal received by the receiving end can be measured by statistical methods. The distribution can usually be distributed by Rayleigh or Rician. In order to improve the transmission video quality on non-ideal wireless transmission channels, we must add Error Resilience in Video Communication on the transmission side or the receiving end. A mechanism for error detection and error concealment is usually designed in the receiving video decoder. The error concealment mechanism does not need to increase the amount of data transmission or channel bandwidth. As long as the image is repaired at the decoding end when the image is wrong, it can effectively prevent the error from rapidly spreading to ensure the quality of the video. The theoretical basis of its development can be divided into three categories: time domain, spatial domain, and combined time and space domain.

From the development of H.261, H.262, H.263 and video coding standards such as MPEG-1, MPEG-2, MPEG-4, H.264/AVC, they all seek to reach a high level with the lowest data volume. Quality video, especially H.264/AVC is the latest standard. In order to counter the various errors that may occur in the transmission channel, H.264 proposes many anti-missing mechanisms in the standard. For example, at the encoding end, there is a 'Flexible macroblock order (FMO), a "Parameter set structure", and a "Data partitioning"... The terminal has Error Concealment. The following is an introduction to the anti-missing mechanism of the encoding end.

FMO

FMO is a new anti-missing mechanism proposed by H.264/AVC. This mechanism breaks through the traditional method of segmenting the "slice group". It can be based on the huge blocks of different characteristics in the picture (a giant block is 16x16). Each pixel is allocated to a different slice group by a certain logic, and each slice group is independently coded (movement estimation), symbol coded, and the like. However, FMO also has shortcomings, including reducing coding efficiency or adding redundant delays.

2. Redundant slices

This mechanism allows the encoder to place redundant information of one or more identical macroblocks under the same bit stream. The main information and redundant information can be transmitted simultaneously during transmission. The main information can be used with smaller quantization parameters. Information uses higher quantization parameters because the extra information is usually lower resolution information. If the main information is successfully received, the excess information is discarded. Otherwise, if the main information is wrong, the redundant information is used to reconstruct.

3. Data segmentation

At the time of coding, some information is more important than other information. Without important information, the entire transmission data may not be used. Therefore, the coding divides the information into three categories and gives three levels of importance, as follows. Show:

1) Class A: including block types, quantization parameters, and motion vectors. This type of file is called header information.

2) Class B: Contains Intra CBPs (Coded Block Patterns) and intra-coded coefficients in the picture. It is necessary to have Class A to match.

3) Class C: Contains inter-pictured block graphics (Inter CBPs) and inter-coded coefficients. This part of the message also needs to be class A to match, but does not require class B. In addition, this part is less important because it does not have synchronization and does not affect the decoding of other information.

When using data splitting, the encoder will place different kinds of information in three different buffers. On the decoding side, the standard decoding process will start when all the information is available. However, if there is a loss of Class B or C, the remaining Class A messages can still be used to improve the error. Hidden performance. For example, since the coding class and motion vector of each block can be known, a better error concealing effect can be achieved.

When error concealment is performed on the decoding side, error detection is first performed, that is, it is detected that a slice group or a macro block cannot be correctly decoded. In H.264/AVC or the newer video standard, since encoding is performed in units of one slice group, errors are in units of one slice group. For example, if a slice group is a giant block representing a whole column, then the wrong unit is a column of giant blocks. If it is encoded in FMO mode and divided into two groups in a black and white chess phase, each error unit is a single group in the FMO code. Since most of them are currently encoded in the above two ways, the following examples are taken as examples. Regardless of the encoding of the method, when an error occurs, the decoder will refer to the message correctly decoding the macroblock around the error block to assist in the error reply. For example, when the entire column of giant blocks is a thin group, for each of the wrong macroblocks, only the boundary information of the upper and lower adjacent macroblocks can be referred to, and for the checkerboard FMO coded slices, the error can be referred to. The boundary information of the macro block adjacent to the top, bottom, left, and right of the block.

At present, the most commonly used macroblock recovery rule is called Boundary Matching Algorithm (BMA). It is based on the boundary strip pixels of the adjacent correct macroblocks of the wrong macroblock (the boundary strip pixels in each direction is 16x1 pixels), and the motion estimation is performed on the previous picture to find out The most similar place, and record the position as the motion vector that is replied, and then perform motion compensation with the motion vector of the reply, so that the reply pixel of the wrong macro block can be obtained. The BMA method is equivalent to a motion estimation program at the decoding end, similar to the motion estimation program at the encoding end. The difference is that the BMA uses the pixel data of other giant blocks (not its own, because it is not correctly decoded, Therefore, its own pixel data does not exist), and the number of pixels that can be used is small. For this reason, the motion vector obtained by BMA has limited accuracy, especially when a slice group is composed of a whole array of macroblocks (because only the pixel information of the upper and lower adjacent macroblocks can be used). In addition, since the BMA performs motion estimation and motion compensation independently for each error macroblock, and subsequent residual values can correct the distortion caused by the incorrect (the residual value is not correctly decoded and does not exist), It is easy to cause the reply result of the adjacent error giant block to be out of tune (for example, a straight line spanning two adjacent giant blocks cannot be closely stitched). These imperfect error concealments are also what the invention is intended to improve.

Non-patent academic literature:

[1] S. Wenger, "H.264/AVC over IP," IEEE Trans. on Circuits and Systems for Video Technology , Vol. 13, No. 7, pp. 645-656, July 2003.

The present invention adopts a time domain error concealment method, and our focus is on channel errors in non-FMO encoding (eg, an entire column of macroblocks in the image is in error) and an improved traditional boundary alignment algorithm (BMA) ). It is more challenging to use only the upper and lower adjacent macroblock boundary pixels of the wrong macroblock to estimate the motion vector of the wrong macroblock in the current picture to the previous picture. The feature of our method is to use the characteristics of variable block size to carry out error concealment. When the error concealment of a single macroblock is no longer limited to the size of 16×16 pixels, it is adaptive to change the single error. The size of the block, such as 8x8, 16x8, 8x16, 16x16, may be the size of a single block.

The erroneous sample of the present invention is a hypothetical independent sheet (currently each sheet contains a column of giant blocks). The core technology of the present invention is to use a dynamic programming algorithm, and the flow is roughly as follows: 1) performing an improved boundary alignment algorithm on all the macroblocks in the error slice to obtain an initial motion vector, 2) taking steps The initial motion vector obtained by 1 is used to perform the accelerated dynamic programming method, 3) the iteration is performed to achieve the error concealment of the smaller block size, and 4) the motion vector of the cell block size is integrated to form the giant block. Error hiding.

Please refer to FIG. 1 , which is a flowchart of a method for error concealment according to an embodiment of the present invention. First, decoding and error detection 101 of the received video coded bit stream, and then performing a 16×16 pixel modified boundary alignment algorithm 102 and a 16×16 pixel macro for the 16×16 pixel macroblock. The dynamic programming method 103 of the block further determines whether the motion vector (MV) estimated by the adjacent macroblock via the improved boundary alignment algorithm is small and consistent 104, and if the judgment result is yes, reconstructs 16×16 pixels. The error macroblock 108; if the judgment result is no, the motion vector integration 106 based on the dynamic plan iterative program 105 of the 8×8 pixel block and the 8×8 pixel block is first performed, and then the entire 16×16 is reconstructed. The pixel error block 107.

When the traditional boundary alignment algorithm does not see the information on the left and right sides of a giant block, only the information of the upper and lower adjacent blocks will be used for boundary comparison. However, this method is subject to insufficient information to cause comparison. The accuracy is reduced (i.e., the estimated motion vector is less accurate), so the present invention employs a modified boundary alignment algorithm. First, we calculate the gray-scale edge intensity of the upper and lower adjacent pixels of a single macroblock in a wrong slice and add it to T _diff , as shown in formula (1). Figure 2 shows the edge of the upper and lower adjacent block boundaries. A diagram of the intensity calculation, in which the error block 202 is surrounded by a plurality of pixel points 201, and each pixel point obtains an edge intensity total value through the formula (1).

Where E _up represents the edge strength of the adjacent giant block above the wrong giant block, E _low is the edge intensity of the lower giant block, and ( x , y ) represents the coordinate of the top left corner of each wrong macro block, m , n represents the length in the y- axis and x- axis directions of the edge intensity region in the upper and lower portions of Fig. 2, respectively.

Each error block can calculate a T _diff value. We can first perform boundary comparison on the error block with the highest T _diff value. The motivation is that the large block of large T _diff represents more complex features with adjacent giant blocks, and the subsequent boundary comparison will be Produce more accurate results. The boundary alignment method adopted by the present invention is as shown in the formula (2), and D _p,q (v) can be minimized in all candidate motion vectors.

among them

Representing the error giant block, respectively, using the upper macro block and the lower part of the macro block to do the boundary comparison to find the difference, It represents the difference between the pixel information temporarily reconstructed by the left and right adjacent macroblocks, and v is the candidate motion vector, and MAP ( p, q ) represents the adjacent macroblock. The weight value is defined in equation (3), and ( p , q ) represents the giant block indicator of the wrong macroblock, ( p , q -1), ( p , q +1) represent the wrong macroregion The giant block indicator of the adjacent giant block on the left and right sides of the block.

The result obtained by formula (2) will be used as the initial motion vector of the current faulty macroblock, and the pixel reconstruction will be temporarily performed for the time being. The reconstructed giant block can provide information to help the left and right neighbors. The reconstructed giant block performs a boundary alignment ( MAP ( p , q ) = 0.5) of the temporarily reconstructed giant block to find the initial motion vector of the individual giant block. In addition, since the motion vector difference of adjacent macroblocks should be small, the motion vector of the adjacent macroblocks can be used as the initial value, and the required search range can be amplified further without using each macroblock by ±16 pixels. The scope of the exercise is estimated. Algorithm relative to traditional boundary alignment (don't consider And without considering T _diff as a priority for restoration), this method can integrate more information to improve the accuracy of motion vector estimation. After an error when the estimated initial motion vector of the macroblock, and then find the subsequent macroblock has the largest T _diff giant from the remaining error blocks perform the above action, and repeated until all errors have to find the initial macroblock The motion vector is up. When each macroblock in the error slice finds the initial motion vector, it can enter the dynamic programming method for the next optimization.

Basically, dynamic programming is the process of breaking a problem into many sub-questions and finding an optimal answer for each sub-question. In essence, the dynamic programming method can be described by a multi-level topology diagram. Figure 3 is a schematic diagram of the dynamic programming method for motion vector reconstruction, which includes the starting point 301 in the dynamic programming method and the termination point 314 in the dynamic programming method. There are many stages between points. Each level has many nodes, which are the first level node 302 in the dynamic programming method, the first level node 306 in the dynamic programming method, and the first level node in the dynamic programming method. 310. The second-level node 303 in the dynamic programming method, the second-level node 307 in the dynamic programming method, the second-level node 311 in the dynamic programming method, the N-1-th node 304 in the dynamic programming method, and the Nth in the dynamic programming method - level 1 node 308, node N-1 node 312 in dynamic programming method, node N node 305 in dynamic programming method, node N node 309 in dynamic programming method, and node N node 313 in dynamic programming method There is an edge between the nodes in the adjacent level. Each node and connection has a cost, which is the node cost D ^bd (node cost) and the connection cost D ^sm (edge cost). Therefore, the dynamic programming method is to find a path from the starting point 301 of one end to the ending point 314 of the other end, so that the total cost D ^total can be minimized. Equation (4) is the total cost D ^total .

D ^total =α D ^bd +(1-α) D ^sm (4)

Where α is a weight value of 0.0 to 1.0.

The technique developed by the present invention assumes that the error unit is a slice, so each erroneous macroblock can be considered a level, and all possible candidate motion vectors are treated as nodes. Therefore, the cost of the node is defined as the cost of the boundary comparison. And the cost of connection is defined as the smoothing cost . The boundary comparison cost is shown in equations (5), (6), and (7), and the smoothing cost is shown by equations (8), (9), and (10).

among them It represents the comparison error of the pixel values of the upper and lower boundaries of the ( p -1, q ) giant block and the ( p +1, q ) giant block respectively, and FIG. 4 is a schematic diagram for calculating the boundary comparison cost, which includes The two time pictures are time t-1 picture 401 and time t picture 402 respectively, and time t-1 picture 401 includes time t-1 in the picture 16×16 pixel macro block minus the upper column pixel area 403, In the time t-1 picture, the uppermost column 405 of the 16×16 pixel macroblock and the lowermost column 406 of the 16×16 pixel macroblock in the time t-1 picture, and the time t picture 402 contains the time t in the picture 16 ×16 pixel macroblock 404, the lowest column in the adjacent macroblock above the 16×16 pixel macroblock in the time t picture, and the most in the adjacent macroblock below the 16×16 pixel macroblock in the time t picture Above column 408.

among them The pixel difference values of the 0th column and the 15th column are respectively represented, and v ₁ and v ₂ are the candidate motion vectors of the ( p , q )th and ( p , q +1)th macroblocks, respectively. FIG. 5 is a diagram for describing a smoothing cost, which includes two time pictures, which are time t-1 picture 501 and time t picture 502, and time t-1 picture 501 contains time t-1 picture 16×16 pixel macro block. 503 and the time t-1 picture 16×16 pixel macro block 504, the time t picture 502 includes a 16×16 pixel macro block 505 in the time t picture and a 16×16 pixel macro block 506 in the time t picture.

The error concealed application is instantaneous in the operation speed requirement. Basically, the computational complexity of the dynamic programming method is M ² × N , where M is the search range of the motion vector, for example, M = 32 × 32 indicates that the single vector search range is from -16~+16 pixels, N is the stage, which shows that the amount of computation is quite large and cannot be directly used in error concealment applications. The method of the present invention therefore speeds up the dynamic programming approach. In theory, dynamic programming must calculate the cost for each level and node and find the path with the lowest cost. However, in some applications, some nodes in the level can be excluded beforehand, once the node is reduced, relative to the calculation. The amount will decrease. For example, as described in FIG. 6, there are 6 nodes in each of the five levels, which are nodes 601, 602, 603, 604, 605, and 606 of the first level, and nodes 607, 608, 609, 610, and 611 of the second level, respectively. 612, nodes 613, 614, 615, 616, 617, 618 of the third level, nodes 619, 620, 621, 622, 623, 624 of the fourth level, nodes 625, 626, 627, 628, 629 of the fifth level And 630. So the original computational complexity is 5 × 6 ² , but if you can find out the very costly nodes in Figure 6, you can't be included in the best path as shown in Figure 6. White, we can only deal with The black node, at this time, the computational complexity is reduced to 5 × 3 ² , which reduces the computational complexity of dynamic programming is the acceleration concept of the invention.

In the present invention, the motion vector of each stage (ie, the error macroblock) has a search range of -16 to +16 pixels in the x- axis and y- axis directions, which is equivalent to 1024 nodes per level. However, some candidate motion vectors can be filtered out by using boundary alignments, such as backgrounds or areas where objects move slowly. When performing boundary alignment, smaller motion vectors are obtained, and larger motion vectors can be obtained. (node) delete without being placed in dynamic planning. Therefore, we can add ranges to the candidate motion vectors in equations (5) and (8), as shown in equations (11) and (12) below.

v ₁ ^initial -β ₁ < v ₁ < v ₁ ^initial +β ₁ , v ₂ ^initial -β ₁ < v ₂ < v ₂ ^initial +β ₁ (12)

Where v ^initial is the computational node cost, the ( p , q ) giant block is determined by the comparison boundary, and v ₁ ^initial , v ₂ ^initial represents the ( p , q ) when calculating the connection cost. ( p , q +1 ) The initial motion vector obtained by the boundary comparison of the giant blocks, and β ₁ is the motion vector search range.

Most of the traditional error concealment algorithms are based on giant blocks of 16×16 pixels. However, if a relatively complex area or an object movement is irregular in an image, block restoration based on 16×16 pixels is limited to the necessity of using the same motion vector, so that a more detailed result cannot be achieved. In view of this, we divide the 16×16 pixel block into 8×8 pixels to realize the dynamic programming method. However, the purpose of segmentation is to solve the complex or irregular moving area of the object, but in an image These regions do not need to be segmented (for example, a stationary background), and the result of forced segmentation will instead result in uncoupling distortion at the junction of the four 8x8 pixel blocks. Each 4×4 pixel sub-block in H.264 has a motion vector. Our method is to extract the motion vectors of all 4×4 pixel sub-blocks adjacent to the upper and lower sides of the wrong macroblock, and check these motion vectors. The intensity of the intensity, if more than 90% is below a critical value, does not perform 8×8 pixel iterative dynamic programming. This is because we judge that the error slice may belong to the background area, so it is not necessary to divide into 8×8 pixel blocks for restoration, and the 16×16 pixel method can be directly used. The iterative program using the 8×8 pixel dynamic programming method is hereinafter referred to as iterative dynamic programming.

The unit of the iterative dynamic programming is 8×8 pixels, and the wrong giant block is 16×16 pixels, so the original slice must be divided into two sub-rows, each time only For a sub-column, the other sub-column is fixed. When the execution is completed, the two sub-columns are interchanged. The overall steps are as follows:

Step 1: The motion vector (MV) of the lower sub-column block is fixed, and the upper sub-column performs the dynamic programming method.

Step 2: The upper sub-column operation ends and the MV information of each block in the upper sub-column is updated.

Step 3: The MV of the upper sub-column block is fixed, and the lower sub-column performs the dynamic programming method.

Step 4: The sub-column operation ends and the MV information of the next sub-column is updated.

Step 5: If the execution of the procedure in steps 1 to 4 above does not reach a certain number of times, return to step 1.

Iterative dynamic programming is a program that repeatedly performs operations. The motion vector information is updated every time it is executed, so each execution will make the result biased to the correct information. The actual detailed steps are as follows:

Step 1': Estimate its initial motion vector for each 16×16 pixel macroblock and designate the initial motion vector as the initial motion vector of its four 8×8 pixel blocks, as shown in FIG. The pre-cut time t picture 701 and the post-cut time t picture 702 are respectively included before and after the macro block cutting at the same time t. The pre-cut time t picture 701 includes a 16×16 pixel macro block 703 in the time t picture, a 16×16 pixel macro block 704 in the time t picture, and a 16×16 pixel macro block 705 in the time t picture. The post-cut time t picture 702 includes an 8x8 pixel block 706 within the time t picture, an 8x8 pixel block 707 within the time t picture, and an 8x8 pixel block 708 within the time t picture. The motion vector in Fig. 7(a) represents the result obtained by the 16×16 pixel giant block dynamic programming method, and Fig. 7(b) represents the 8×8 pixel block currently to be processed using the figure 7(a). The result is taken as the initial value.

Step 2': The motion vector of the 8×8 pixel block of the lower sub-column is fixed first, and the dynamic block is performed on the block of the upper sub-column, and the path cost is defined as formula (13). The difference here from the previous dynamic programming is that the 8×8 pixel block to be processed is also a wrong 8×8 pixel block, so the cost calculated with the lower block is smoothed.

C ^total = λC ^bd + ( 1- λ)(C ^smh + C ^smv ) (13)

Where C ^total is the total cost of the path, C ^bd is the node cost, C ^smh and C ^smv are the horizontal smoothing cost and the vertical smoothing cost, respectively, and λ is the number of 0.0~1.0 (weight value). The following is the introduction of each cost.

1. Node cost C ^{bd is} shown in equation (14). Figure 8 is a schematic diagram of boundary alignment of 8 × 8 pixel blocks, including time t-1 picture 801 and time t-1 8×8 pixel block in the picture. 803, time t picture 802 and time 8 picture 8 x 8 pixel block 804.

Where v ^initial is the initial motion vector and β ₂ is the search range.

2. The horizontal smoothing cost C ^smh and the vertical smoothing cost C ^smv are respectively shown in the formula (15) (16), and the figure 9 is the illustration thereof, wherein (a) the figure contains two time frames respectively for the time t-1 picture 901 and time t picture 902. The time t-1 picture 901 includes an 8×8 pixel block 903 in the time t-1 picture, and an 8×8 pixel block 904 in the time t-1 picture; the time t picture 902 includes the 8×8 pixel area in the time t picture. Block 905 and 8 x 8 pixel block 906 within time t picture. And (b) includes a time t-1 picture 907 and a time t picture 908. In the time t-1 picture 907, there are time t-1 in the picture 8×8 pixel block 909 and time t-1 in the picture 8×8 pixel block 910, time t picture 908 has time t picture 8×8 pixels Block 911 and 8 x 8 pixel block 912 within time t picture.

among them

Where v ₁ and v ₂ are the candidate motion vectors of the ( p , q ), ( p , q +1) giant blocks, respectively, and v _bm is the temporary estimate of the ( p +1, q ) giant blocks. The measured motion vector, and v ₁ ^initial and v ₂ ^initial are the initial motion vectors of the ( p , q ), ( p , q +1) giant blocks, respectively.

Step 3': The 8 x 8 tile slice of the upper sub-column in step 2' will find the best path and update its initial motion vector. At this point, the motion vector after the update of the upper sub-column is fixed, and the dynamic programming is performed by the next sub-column. The method is roughly the same as that of step 2', but the upper and lower sides are opposite, and the following are the cost values.

1. The node cost C ^{bd is} as shown in formula (21), which is described in FIG. 10, which includes time t-1 picture 1001, time t picture 1002, time t-1 picture 8×8 pixel block 1003, time t 8×8 pixel block 1004 in the picture.

Where v ^initial is the initial motion vector and β ₂ is the search range.

2. The horizontal smoothing cost C ^smh and the vertical smoothing cost C ^smv are respectively shown in formula (22) (23), and FIG. 11 is an illustration thereof, wherein (a) there is time t-1 picture 1101, time t picture 1102, 8×8 pixel block 1103 in time t-1 picture, 8×8 pixel block 1104 in time t-1 picture, 8×8 pixel block 1105 in time t picture and 8×8 pixel in time t picture Block 1106. (b) There is a time t-1 picture 1107, a time t picture 1108, a time t-1 picture 8×8 pixel block 1109, a time t-1 picture 8×8 pixel block 1110, time t picture 8 x 8 pixel block 1111 and 8 x 8 pixel block 1112 in time t picture.

among them

Where v ₃ and v ₄ are the candidate motion vectors of the ( p , q ), ( p , q +1) macroblocks respectively, and v _up is the upper ( p -1, q ) giant blocks above. The motion vector estimated in one step, and v ₃ ^initial and v ₄ ^initial are the initial motion vectors of the ( p , q ), ( p , q +1) ^macroblocks , respectively.

Step 4': The 8x8 pixel block of the lower sub-column in the above step 3' will find the best path, that is, each 8x8 block of the lower sub-column will obtain the temporally optimal motion vector instead of its initial motion vector. At this time, the motion vector after the sub-column update is fixed, and if the number of executions does not reach the set value, the process returns to step 2' to continue execution. If the number of executions has reached the predetermined number of times, skip to step 5'.

Step 5': End the iterative dynamic programming procedure, and each 16×16 pixel macroblock will have four motion vectors.

After the iterative dynamic programming process, the motion vectors of all 8×8 pixel blocks must undergo the integration test procedure. Although these motion vectors are optimized, two adjacent 8×8 pixel blocks can be integrated if they have very similar motion vectors. For example, two 8×8 blocks can be used. The pixel blocks are integrated into 8×16 pixel blocks or 16×8 pixel blocks, and four 8×8 blocks can also be integrated into one 16×16 block. The present invention integrates motion vectors of 8x8 pixel blocks using a comparison of boundary alignment costs, and the execution steps are as follows.

Step 1": Calculate the variation of the motion vector of the four 8x8 pixel blocks in a 16×16 pixel macroblock. If the variance is less than a threshold, merge the four 8x8 pixel blocks into a 16x16 pixel macroblock. Block, directly using the previously obtained 16x16 pixel giant block estimated motion vector as the corrected motion vector of the merged 16x16 giant block; if not, proceed to step 2";

Step 2": Correct the motion vector for the four 8x8 pixel blocks, and test the possibility of merging any two horizontal or vertical adjacent 8x8 pixel blocks into 16x8 or 8x16 pixel blocks; if the merge test is passed, then merge The corrected motion vector average of the two 8x8 pixel blocks is used as the modified motion vector of the merged block; if the merge test is not passed, the original 8x8 pixel block is maintained. For example, the block A in Figure 12 (16) The octave of the 8×8 block in the upper left corner of the ×16 macroblock is closer to the MV of the B block (the 8×8 block 1202 in the upper right corner of the 16×16 giant block), so A and B can be merged (16). ×8 block in the upper left 8×8 block 1205, 16×8 block in the upper right 8×8 block 1206), and C (16×16 block in the lower left 8×8 block 1203) and D (16×) The MV of the lower right 8×8 block 1204 in the 16 block is also relatively close, so C and D can be combined (the lower left 8×8 block 1207, the 16×8 block in the 16×8 block is the lower right 8× Block 8 1208. Note that the present invention does not allow (A, D) or (B, C) to be grouped together.

Step 3": Calculate the combined boundary comparison difference for the two blocks to be merged (the combined motion vector after the merged block is the average of the modified motion vectors of the original two 8x8 pixel blocks), as shown in the figure Thirteenth (a) includes a 16×8 pixel block 1307, a time t-1 picture 1308, a time t picture 1309, and a time t-1 picture 16×8 pixel block 1310 in the time t picture, It represents a boundary of the 16×8 pixel block using the modified motion vector, and FIG. 13(b) contains the time 8×8 pixel block 1301 and time t-1. 8×8 pixel block 1302 in the picture, 8×8 pixel block 1303 in time t picture, 8×8 pixel block 1304 in time t picture, time t-1 picture 1305 and time t picture 1306, and its representative 2 Each of the 8×8 pixel blocks uses the motion vector to calculate the sum of the boundary comparison differences. The two boundary comparison results are compared. If the difference in Figure 13 (a) is lower, a 16×8 is used. Pixel reconstruction, otherwise, using two 8×8 pixel blocks and v _A , v _B to reconstruct. The test procedure for merging into 8×16 blocks is similar to the above, and will not be described here.

In one embodiment, the iterative dynamic programming algorithm, the traditional boundary alignment algorithm (BMA), and the literature [2] of the present invention are respectively compared. In this experiment, the Packet Loss Rate (PLR) was 5%, 10%, and 15%, and each packet represented a thin sheet (a column of giant blocks). The experimental data are Tables 1 to 4. Tables 1-3 represent experimental results for different quantization parameters when encoding, while Table 4 represents the time taken for different methods to be performed. It can be seen that the method proposed by the present invention considers the smoothing cost and considers the optimization of the entire macroblocks in each slice, so that the result of the restoration of the adjacent blocks can be obtained (ie, the PSNR value). The highest), and its computing speed is faster than the literature [2].

Non-patent academic literature:

[2] Xueming Qian, Guizhong Liu, and Huan Wang, "Recovering connected error region based on adaptive error concealment order determination," IEEE Trans. on Multimedia, Vol.11, pp.683-695, June 2009

101. . . Received encoded bit stream decoding and error detection

102. . . Improved boundary alignment algorithm for 16×16 pixel giant blocks

103. . . Dynamic programming method based on 16×16 pixel giant block

104. . . Whether the MV of the adjacent giant block is small and consistent

105. . . Dynamic programming iterative program based on 8×8 pixel block

106. . . Motion vector integration of 8×8 pixel blocks

107. . . Rebuilding a 16×16 pixel error block

108. . . Rebuilding a 16×16 pixel error block

201. . . pixel

202. . . Wrong block

301. . . Starting point

302. . . The first level node in the dynamic programming method

303. . . Second-level node in dynamic programming

304. . . The N-1th node in the dynamic programming method

305. . . The Nth node in the dynamic programming method

306. . . The first level node in the dynamic programming method

307. . . Second-level node in dynamic programming

308. . . The N-1th node in the dynamic programming method

309. . . The Nth node in the dynamic programming method

310. . . The first level node in the dynamic programming method

311. . . Second-level node in dynamic programming

312. . . The N-1th node in the dynamic programming method

313. . . The Nth node in the dynamic programming method

314. . . Termination point

401. . . Time t-1 screen

402. . . Time t screen

403. . . The 16×16 pixel macro block in the time t-1 picture is deducted from the next column of pixels.

404. . . 16 × 16 pixel giant block in time t picture

405. . . The topmost column of the 16×16 pixel giant block in the time t-1 picture

406. . . The bottommost column of the 16×16 pixel giant block in the time t-1 picture

407. . . In the time t picture, the bottommost column in the adjacent giant block above the 16×16 pixel giant block

408. . . The topmost column in the adjacent giant block below the 16×16 pixel giant block in the time t picture

501. . . Time t-1 screen

502. . . Time t screen

503. . . 16×16 pixel giant block in time t-1 picture

504. . . 16×16 pixel giant block in time t-1 picture

505. . . 16 × 16 pixel giant block in time t picture

506. . . 16 × 16 pixel giant block in time t picture

601, 602, 603, 604, 605, 606. . . First level node

607, 608, 609, 610, 611, 612. . . Second level node

613, 614, 615, 616, 617, 618. . . Third level node

619, 620, 621, 622, 623, 624. . . Fourth level node

625, 626, 627, 628, 629, 630. . . Fifth level node

701. . . Time t screen

702. . . Time t screen

703. . . 16 × 16 pixel giant block in time t picture

704. . . 16 × 16 pixel giant block in time t picture

705. . . 16 × 16 pixel giant block in time t picture

706. . . 8 × 8 pixel block in time t picture

707. . . 8 × 8 pixel block in time t picture

708. . . 8 × 8 pixel block in time t picture

801. . . Time t-1 screen

802. . . Time t screen

803. . . 8 x 8 pixel block in time t-1 picture

804. . . 8 × 8 pixel block in time t picture

901. . . Time t-1 screen

902. . . Time t screen

903. . . 8 x 8 pixel block in time t-1 picture

904. . . 8 x 8 pixel block in time t-1 picture

905. . . 8 × 8 pixel block in time t picture

906. . . 8 × 8 pixel block in time t picture

907. . . Time t-1 screen

908. . . Time t screen

909. . . 8 x 8 pixel block in time t-1 picture

910. . . 8 x 8 pixel block in time t-1 picture

911. . . 8 × 8 pixel block in time t picture

912. . . 8 × 8 pixel block in time t picture

1001. . . Time t-1 screen

1002. . . Time t screen

1003. . . 8 x 8 pixel block in time t-1 picture

1004. . . 8 × 8 pixel block in time t picture

1101. . . Time t-1 screen

1102. . . Time t screen

1103. . . 8 x 8 pixel block in time t-1 picture

1104. . . 8 x 8 pixel block in time t-1 picture

1105. . . 8 × 8 pixel block in time t picture

1106. . . 8 × 8 pixel block in time t picture

1107. . . Time t-1 screen

1108. . . Time t screen

1109. . . 8 x 8 pixel block in time t-1 picture

1110. . . 8 x 8 pixel block in time t-1 picture

1111. . . 8 × 8 pixel block in time t picture

1112. . . 8 × 8 pixel block in time t picture

1201. . . The upper left 8×8 block in the 16×16 giant block

1202. . . The upper right 8×8 block in the 16×16 giant block

1203. . . The lower left 8×8 block in the 16×16 giant block

1204. . . The lower right 8×8 block in the 16×16 giant block

1205. . . The upper left 8×8 block in the 16×8 block

1206. . . Upper right 8×8 block in 16×8 block

1207. . . The lower left 8×8 block in the 16×8 block

1208. . . The lower right 8×8 block in the 16×8 block

1301. . . 8 x 8 pixel block in time t-1 picture

1302. . . 8 x 8 pixel block in time t-1 picture

1303. . . 8 × 8 pixel block in time t picture

1304. . . 8 × 8 pixel block in time t picture

1305. . . Time t-1 screen

1306. . . Time t screen

1307. . . 16×8 pixel block in time t picture

1308. . . Time t-1 screen

1309. . . Time t screen

1310. . . 16×8 pixel block in time t-1 picture

Figure 1 is a flow chart of the error concealment method of the present invention

Figure 2 is a schematic diagram of the calculation of the edge strength of the boundary of adjacent giant blocks.

Figure 3 is a schematic diagram of the dynamic programming method for motion vector reconstruction

Figure 4 is a schematic diagram of boundary comparison

Figure 5 is a schematic diagram of smooth comparison

Figure 6 is a schematic diagram of deleting inappropriate nodes

Figure 7: Initial motion vector designation for iterative dynamic programming

Figure 8 is the boundary comparison of 8 × 8 pixel blocks (upper sub-column)

Figure 9 (a) is the horizontal smoothing cost of the 8 × 8 pixel block (upper sub-column)

Figure 9 (b) is the vertical smoothing cost of the 8 × 8 pixel block (upper sub-column)

Figure 10 is a boundary comparison of 8 × 8 pixel blocks (lower sub-column)

Figure 11 is the 8 × 8 pixel block smoothing cost (lower sub-column)

Figure XI (a) is the horizontal smoothing cost of the 8 × 8 pixel block (lower sub-column)

Figure XI (b) is the vertical smoothing cost of the 8 × 8 pixel block (lower sub-column)

Figure 12 is a schematic diagram of grouping of motion vector similar blocks

Figure 13 (a) is a schematic diagram of 16×8 pixel block boundary alignment motion vector integration

Figure 13 (b) is a schematic diagram of 8×8 pixel block boundary alignment motion vector integration

101. . . Received encoded bit stream decoding and error detection

102. . . Improved boundary alignment algorithm for 16×16 pixel giant blocks

103. . . Dynamic programming method based on 16×16 pixel giant block

104. . . Whether the MV of the adjacent giant block is small and consistent

105. . . Dynamic programming iterative program based on 8×8 pixel block

106. . . Motion vector integration of 8×8 pixel blocks

107. . . Rebuilding a 16×16 pixel error block

108. . . Rebuilding a 16×16 pixel error block

Claims (14)

A video decoding method capable of concealing transmission errors, which can perform error concealment on a decoding end of a coded bit stream packet with a whole column of macroblocks as a slice unit, and the steps include: (a) encoding a video bit The elementary stream performs decoding and error detection; (b) if a slice error is detected, the following steps are performed; (c) an improved boundary ratio of 16×16 pixels is performed for each error macroblock in the slice. Performing an algorithm to obtain an initial estimated motion vector of each of the error macroblocks; (d) performing a 16×16 pixel dynamic programming method on the error slice to obtain an estimated motion vector of each of the macroblocks; e) performing an inspection procedure for each error macroblock: extracting motion vectors of all adjacent blocks in the correct macroblock above and below the error macroblock; if the strength of the motion vectors exceeds a threshold The percentage is below a critical value, so that the corrected motion vector of the error macroblock is equal to the estimated motion vector obtained in step (d) and jumps to step (i); if not, skips to step (f); (f) dividing each of the giant blocks into four 8×8 images The block, the estimated motion vector estimated by the macroblock of the macroblock in step (d) is used as the initial motion vector of each 8×8 pixel block; (g) the slice is divided into upper and lower 8×8 a sub-pixel sub-column, performing an iterative dynamic programming procedure on the upper sub-column and the lower sub-column to correct the initial motion vector of each 8×8 pixel block until the number of iterations reaches a preset value, and each of the 8 ×8 correction motion vector of the pixel block; (h) performing a combined test on the modified motion vector obtained in step (g) for each of the 8×8 pixel blocks to which each error macroblock belongs, if the test is successful, Combining adjacent 8×8 pixel blocks in the same error macroblock into 16×8, 8×16, or 16×16 pixel blocks, and obtaining the combined block estimation motion vector, if not, then Maintaining the original four 8×8 pixel blocks and their modified motion vectors; and (i) estimating motion vectors or 8×8 pixels according to the combined blocks of the 16×8, 8×16, or 16×16 pixel blocks described above. Corrected motion vector of the block, performing each of the error regions Motion compensation of the block. The method of claim 1, wherein the step (c) further comprises: (c1) calculating a grayscale edge intensity value of each pixel of the upper and lower adjacent pixels of the individual macroblocks in the error slice, and Adding the edge intensity values to the T _diff of the giant block: (c2) performing a boundary comparison on the wrong _{macroblock with the} highest T _diff value to the previous reconstructed picture to obtain an initial estimated motion vector; c3) the rest of the sheet has not been error ratio macroblock boundary of the selected highest value T _diff a reconstructed picture boundary priority compared to the former, in order to obtain an initial estimate of the motion vector; and (C4) cycle Step (c3) until all the macroblocks in the error slice get the initial estimated motion vector. The method of claim 2, wherein the boundary comparison step utilizes one of the upper or lower adjacent correct macroblocks or the complex column boundary pixels of the erroneous macroblock, and left/right adjacent motion compensation. One of the posterior macroblocks or a plurality of row boundary pixels, the pixel alignment of the motion vector estimation is reconstructed from the previous time; wherein the difference between the left and right adjacent motion compensated macroblocks is obtained. Multiply by a weight of 0.5, the comparison difference obtained by using the upper/lower adjacent correct macroblocks is multiplied by 1.0, and the left/right adjacent error macroblocks that have not been motion compensated are compared. The value weight is 0.0. The method of claim 1, wherein each of the 16×16 pixel dynamic programming methods in step (d) represents an error macroblock, and each node of each level represents one of the possible fault blocks. Estimating the motion vector; wherein the node cost of each node of the dynamic programming method is defined as the boundary comparison cost of the node, and the connection cost of the connection between the two nodes is defined as the motion compensation of the macroblock corresponding to the two nodes The subsequent smoothing cost; wherein the boundary comparison cost refers to the boundary comparison difference obtained by the motion vector estimation of the faulty macroblock using one or more column boundary pixels of the upper and lower adjacent correct macroblocks; the smoothing cost system It means that the motion vector represented by the node and the left or right adjacent macroblock use the motion vector represented by the other connected node in the wrong macroblock, and the two giant blocks simultaneously perform motion compensation and the result is The sum of the differences between a plurality of boundary pixel rows. The method of claim 4, wherein the dynamic programming method finds a path with a minimum path cost, wherein the path cost of each path is defined as αD ^bd +(1 - α ) D ^sm , where D ^bd For the sum of the boundary cost of the nodes passing through the path, D ^{sm is} the sum of the connection costs of the connections between the connected nodes of the path, and α is the weight value of 0.0~1.0. The method of claim 5, wherein the path through which the path having the smallest path cost defines an estimated motion vector for each of the error macroblocks. The method of claim 1, wherein the step (g) comprises: (g1) the motion vector (MV) of the sub-column block is fixed, the upper sub-column performs an 8x8 pixel dynamic programming method; (g2) the upper sub-column The operation ends and the MV information of each block in the upper sub-column is updated; (g3) the MV of the upper sub-column block is fixed, the lower sub-column performs the 8x8 pixel dynamic programming method; (g4) the sub-column operation ends and the MV of the lower sub-column is updated. Information; (g5) If the above (g1)~(g4) execution procedure does not reach a certain number of times, it returns to step (g1). The method according to claim 7, wherein the path cost of the 8x8 pixel dynamic programming method of the (g1) step is defined as λC ^bd + ( 1- λ ) (C ^smh + C ^smv ) , and λ is 0.0~1.0. Value; where C ^bd is the node cost, C ^smh , C ^smv are the horizontal smoothing cost and the vertical smoothing cost respectively; wherein the node cost is compared by using the boundary pixels of the upper adjacent ^{macroblock of} the 8x8 pixel block. The difference, the horizontal smoothing cost is the smoothing cost after the motion compensation is performed by using the motion vector of the 8x8 pixel block and the adjacent 8x8 pixel block below the vertical smoothing cost, and the 8x8 pixel block is adjacent to the left and right 8x8. The smoothing cost after motion compensation using the motion vector of the pixel between blocks. The method of claim 8, wherein the path cost of the 8x8 pixel dynamic programming method of the (g3) step is defined as λC ^bd + ( 1- λ) (C ^smh + C ^smv ) , and λ is 0.0 to 1.0. Value; where C ^bd is the node cost, C ^smh , C ^smv are the horizontal smoothing cost and the vertical smoothing cost respectively; wherein the node cost is the boundary pixel of the adjacent adjacent ^{macroblock of} the 8x8 pixel block for boundary comparison The difference, the horizontal smoothing cost is the smoothing cost after the motion compensation is performed by using the motion vector of the 8x8 pixel block and the adjacent 8x8 pixel block above it, and the vertical smoothing cost is adopted by the 8x8 pixel block and its left and right adjacent Smoothing cost after motion compensation using the motion vector of the 8x8 pixel block. The method of claim 1, wherein the step of combining the modified motion vectors of the 8x8 pixel block of the (h) step further comprises: (h1) calculating four 8x8 pixels in a 16×16 pixel macroblock. The block corrects the variance of the motion vector. If the variance is less than a critical value, the four 8x8 pixel blocks are combined into a 16x16 pixel macroblock, and the previously obtained 16x16 pixel macroblock estimation motion vector is directly used as the motion vector. The corrected motion vector of the merged 16x16 macroblock; if not, proceeds to step (h2); (h2) corrects the motion vector for the four 8x8 pixel blocks, and tests any two horizontal or vertical adjacent 8x8 pixel blocks to be merged into The possibility of a 16x8 or 8x16 pixel block; if the merge test is passed, the corrected motion vector average of the combined 8x8 pixel blocks is used as the modified motion vector of the merged block; if the merge test is not passed, Maintaining the original 8x8 pixel block; and (h3) performing motion compensation of the error macroblock with the modified motion vector of the merged block obtained by step (h1) (h2). The method of claim 10, wherein the combining test step of (h2) further comprises: calculating a boundary alignment by using any two horizontal or vertical adjacent 8x8 pixel blocks as corrected motion vectors of the combined blocks thereof. The difference value, if the boundary is compared with the difference value, each 8×8 pixel block is smaller than the sum of the difference ratios of the respective modified motion vectors, indicating that the merge test passes; if not, the merge test is performed. Fail. A video decoding device capable of concealing transmission errors can perform error concealment on a decoding end of a coded bit stream packet with a whole column of macroblocks as a slice unit, including: a video decoding unit that receives a video coding The bit stream is input and decoded; an error detecting unit performs error detection on each slice decoding result decoded by the video decoding unit, and if the detection result is correct, the video decoding of the next slice is continued. If the result of the test is an error, an error signal is sent to the error concealment processing unit described below; An error concealment processing unit that receives the error signal of the error detection unit and performs steps (c) to (i) of the method of any one of items 1 to 11. The device of claim 12, wherein the error concealing processing unit is a hardware unit. The device of claim 12, wherein the error concealment processing unit is a digital signal processor unit.