JP2009105986A - Decoder - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform error concealment processing by determining an optimum reference frame when carrying out error concealment. <P>SOLUTION: When a decode error is detected in macro block information that resides in slice information in a bit stream and error concealment processing is performed, a frame having the display time closest to a slice with the decode error detected is used as a reference frame for error concealment in the reproduction images accumulated in a reference frame buffer. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

The present invention relates to a method for determining a reference frame for error concealment and a decoder including the same.

In recent years, research and development of image compression has progressed, MPEG4 (ISO / IEC14496-2),
H. Video encoding methods such as H.264 (ISO / IEC 14496-10) are standardized. This method is used in various fields such as digital broadcasting, Internet communication, mobile phone communication and the like.

In data communication, when encoded data obtained by encoding image data by the MPEG4 encoding method or the like is sent, the receiving side can decode and output the sent encoded data by a decoder. It is common. When the encoded data is decoded by the decoder, an error such as “0” data changing to “1” data may occur due to deterioration of the wireless communication state or the like. When a decoding error occurs due to the error and the decoder detects the decoding error, a countermeasure is taken to request the transmission source to retransmit the image frame data of that portion. However, the method of requesting retransmission of image data is not very effective because it takes more time to retransmit.

As a countermeasure against decoding errors, it has been proposed to perform error image interpolation called error concealment in a decoder (see, for example, Patent Document 1).

In this error concealment process, since an image frame in which a decoding error is detected cannot be used as it is, it is necessary to create pseudo data with reference to other image frame data (hereinafter referred to as a reference frame). In the MPEG4 encoding method and the like, the reference frame is standardized as the frame decoded immediately before, and the error concealment process is performed with reference to the frame decoded immediately before or the frame with the closest display time.

JP 2001-309388 A (page 2, page 3, pages 6 to 9, page 1)

H. In the decoding of data encoded by the H.264 encoding method, when a decoding error is detected, it is necessary to perform error concealment processing as in the MPEG4 encoding method. However, H.C. In the H.264 coding standard, unlike the MPEG4 coding system, no reference frame is specified for error concealment processing. Therefore, H.H. In the H.264 encoding method, when error concealment processing is performed,
There is a problem of which frame is used for error concealment processing, and depending on the selection of the reference frame, there is a problem that the image data is decoded to completely different from the image data before encoding.

Accordingly, the present invention has been made to solve the above-described problems, and a reference frame is selected from a plurality of frames when performing error concealment processing. 26
It is an object of the present invention to provide an error concealment reference frame determination method capable of determining an optimal reference frame in a four-coding decoder and performing error concealment processing, and a decoder having the function.

  In order to achieve the above object, a decoder according to the present invention is characterized by the following.

According to the present invention, a reference frame for error concealment processing is selected from a plurality of frames. When decoding the encoded data encoded by the H.264 encoding method by the decoder, the optimum reference frame is determined and error concealment processing is performed, so that reproduced image data with better image quality can be obtained. .

In an embodiment of the present invention, H.P. The block diagram which showed the internal structure of H.264 decoder. The figure which showed the internal state of the reference frame buffer part 9 in the Example of this invention. The figure which shows the structure of the slice information in the Example of this invention. The figure which shows the reference frame determination method when the decoding error is detected in the intra macroblock in the Example of this invention. The flowchart which showed the reference frame determination method when the decoding error is detected by the intra macroblock in the Example of this invention. The figure which shows the partition division | segmentation in an intermacroblock. The figure which shows the reference frame determination method when the decoding error is detected in the intra macroblock in the Example of this invention. The flowchart which showed the reference frame determination method when a decoding error is detected in the inter macroblock which is not divided into partitions in the Example of this invention. The figure which shows the adjacent block in reference frame determination when the decoding error is detected in the inter macroblock which is not divided into the partitions in the Example of this invention. The flowchart which showed the reference frame determination method when the decoding error is detected in the inter macroblock divided into the partitions in the Example of this invention. The figure which shows the reference frame determination method when a decoding error is detected in the inter macroblock divided into the partitions in the Example of this invention. The figure explaining the "adjacent slice" in the Example of this invention.

  Hereinafter, embodiments of the present invention will be described.

The present invention is described in H.C. An embodiment applied to a decoder of the H.264 encoding method will be described with reference to FIGS. FIG. It is the block diagram which showed the internal structure of H.264 decoder.

The decoder 1 includes a decoding unit 2, an inverse quantization unit 3, an inverse DCT conversion unit 4, a decoding addition unit 5, an intra prediction unit 6, a filter 7, a motion compensation unit 8, and a reference frame buffer unit. 9 and a switch 10. And the decoding part 2, the inverse quantization part 3, the inverse DCT transformation part 4,
The decoding adder 5, the intra predictor 6, the filter 7, the motion compensator 8, the reference frame buffer 9, and the switch 10 are controlled by the controller 11, respectively.

First, the operation of each unit will be described. The decrypting unit 2 has a function of decrypting the bit stream of the data D1 sent to the decoder 1 according to the bit stream syntax and restoring the encoded data. The decoding unit 2 includes a bit stream analysis unit 12 and an error processing unit 13, and an error detection unit 14 is provided inside the bit stream analysis unit 12. The detailed operation of the decoding unit 2 will be described later.

The inverse quantization unit 3 performs inverse quantization on the data D2 from the decryption unit 2. Then, the inversely quantized data D3 is sent to the inverse DCT transform unit 4.

The inverse DCT conversion unit 4 performs inverse DCT conversion of the data D3. Then, the inverse DCT-transformed data D4 (prediction residual signal) is sent to the decoding adder 5.

The decoding and adding unit 5 is different in processing depending on the connection state of the switch 10. The switch 10 is operated by an intra (intra-frame) prediction mode and an inter (
Switch to prediction mode (between frames). In the intra prediction mode, the switch 10 is connected to the intra prediction unit 6, and the decoding and adding unit 5 receives the data D from the intra prediction unit 6.
7 and data D4 (prediction residual signal) from the inverse DCT transform unit 4 are added. In the inter prediction mode, the switch 10 is connected to the motion compensation unit 8, and the decoding and adding unit 5 receives the data D9 from the motion compensation unit 8 and the data D4 (prediction residual signal) from the inverse DCT conversion unit 4. to add.

In each mode, the data D5 added by the decoding and adding unit 5 is sent to the intra prediction unit 6 or the filter 7.

The intra prediction unit 6 operates in the intra prediction mode, and performs intra prediction from the data D5 sent from the decoding addition unit 5. The data D7 output after the completion of the intra prediction is sent to the decoding addition unit 5 via the switch 10.

The filter 7 has a function of reducing the distortion of the decoded data. Here, the data filtered by the filter 7 is output as frame data D6, and is stored in the reference frame buffer unit 9 as reference frame candidates.

The motion compensation unit 8 operates in the inter prediction mode. The motion compensation unit 8 receives the motion information D10 or the provisional motion vector data D11 from the decoding unit 2, and the motion information D10
Alternatively, it has a function of reading the reference frame data D8 from the reference frame buffer unit 9 using the provisional motion vector data D11 and performing motion compensation. Data D9 output after completion of motion compensation is sent to the decoding and adding unit 5 via the switch 10.

The reference frame buffer unit 9 stores image frame data D6 output from the filter 7.
Are sequentially accumulated. FIG. 2 is a diagram showing the state of the frame data stored in the reference frame buffer unit 9. The data D6 filtered by the filter 7 is the decoder 1
Is sent to the reference frame buffer unit 9 as frame data D6. In the reference frame buffer unit 9, headers are sequentially attached to each frame data D6 as shown in FIG. Reproduced images F1, F2,... In FIG. 2 are frame data accumulated in order.

The header includes information on a list number to be distinguished when referred to as reference frame data D8, or a long-t specified by the encoding side as a frame to be referred to for a long time at the time of decoding.
Information (not shown) such as the presence or absence of a flag called erm-picture is included. The list number is H.264. According to the H.264 standard, only a maximum of 16 numbers from 0 to 15 can be assigned, so that there is a frame having no list number (the list number is assigned to 16).
Or less).

The header can be changed every time the slice of encoded data to be decoded changes. For example, when decoding slice 1, “header 1” is “
“List number 1” and “Header 2” are “List number 0”, “Header 3” is “No list number”
“Header 4” is set as “List number 2”. When decoding slice 2,
As shown in (2) of FIG. 2, “header 1” is “list number 0”, “header 2” is “no list number”, “header 3” is “list number 1”, and “header 4” is “list number”. The list number of the header can be set every time the slice to be decoded changes, such as “2”.

As described above, when the decoder 1 receives the data D1, when the decoding unit 2 decodes the data D1 as intra prediction, the decoder 1 performs inverse quantization unit 3, inverse DCT conversion unit 4, decoding addition unit 5, intra prediction. The data is decoded by the unit 6, filtered, and output as data D 6 and stored in the reference frame buffer unit 9. Normally, the motion compensation unit 8 is used for intra prediction.
Does not work.

When the decoding result of the decoding unit 2 is inter prediction, motion information D10 from the decoding unit 2,
Based on the reference frame data D8 from the reference frame buffer 9, the motion compensation unit 8 obtains a motion compensation frame image and outputs it as data D9. Then, the difference addition data D 4 obtained by the inverse quantization unit 3 and the inverse DCT conversion unit 4 and the motion compensation data D 9 from the motion compensation unit 8 are added by the decoding addition unit 5. Thereafter, the filter 7 is applied and output as data D6 and stored in the reference frame buffer unit 9.

When the error detection unit 14 of the decoding unit 2 detects a decoding error in the slice information of the encoded data, an error concealment process is performed, and the provisional motion vector data D11 from the error processing unit 13 and the reference frame Reference frame data D from buffer 9
8, the motion compensation unit 8 obtains a motion compensation frame image and outputs it as data D9. Then, the inverse quantization unit 3 and the inverse DCT transform unit 4 are controlled by the control unit 11 to perform inverse quantization processing and inverse DC.
Control is performed so as not to perform the T conversion process, and data D2 and D3 are input, respectively, and data D4 and data D9 are added by the decoding / adding unit 5. In this case, since the difference data D4 after decoding is all 0, the data D9 is output as it is. Thereafter, the data is filtered by the filter 7 and output as data D6 and stored in the reference frame buffer unit 9. In addition,
The filter 7 may be controlled not to perform the filter process by the control unit 11.

Next, the detailed operation of the decoding unit 2 will be described with reference to FIGS. First, when the data D1 is sent to the decoding unit 2, the null unit type of the data D1 (NAL unit type)
Is sent to the bitstream analysis unit 12. The bit stream analysis unit 12 analyzes data based on the null unit type and extracts encoded data. Then, the error detection unit 14 of the bitstream analysis unit 12 detects a decoding error of slice header information and a decoding error of macroblock information included in slice information of encoded data based on a predetermined condition. Here, the NAL unit type is what kind of internal information of each NAL unit that is sent (Slice)
data, picture parameter, additional information, etc.).

FIG. 3 shows the structure of slice information. As shown in FIG. 3, the slice header information includes information such as slice specific information 1, image display order information, reference frame number information, reference frame list information, reference frame attribute information, and slice specific information 2.

In the slice specific information 1, the address of the first macroblock data 1 included in the slice information, the type of slice information (I slice or P slice), the ID of the picture parameter set referred to by the slice information, the frame number, etc. are set. Is done.

Further, the display order offset number for the decoded image is set in the image display order information. Further, in the reference frame attribute information, an attribute relating to each frame of the reference frame list information (for example, information such as long-term-picture) is set.

In addition, the slice fixed information 2 is set with an offset value of a quantization coefficient in units of slices. One slice information includes at least one macroblock data. The decoding error of the slice header information in the error detection unit 14 is detected in the following case.

(1) There is no data corresponding to the result of decoding the slice header information. This is because, for example, the maximum number of reference frames is 3 as the parameter of the bitstream to be decoded, but the result of decoding the slice header information is This means that the maximum number of reference frames is five.

(2) The result of decoding the slice header information does not exist in the encoding correspondence table. For example, the result of decoding the slice header information has a value of “10”. It means a case where only values from “0” to “9” are defined. The presence in the encoding correspondence table here means that there is a value to be returned for the decoding result, and the determination is made in the bitstream analysis unit 12.

(3) The decoder does not support the result of decoding the slice header information. This is because, for example, the slice information type includes “I slice” composed of only intra macroblocks and “P” including inter macroblocks. Although only the “slice” is supported by the decoder, the slice type based on the decoded slice header information is “B slice”.
Means the case.

  The decoding error detection is not limited to the above method, and other detection methods may be used.

Further, depending on the detection method, a decoding error may not be detected correctly. As described above, when a decoding error occurs in the slice header information, the data D2 and the inverse DCT to the inverse quantization unit 3 are prevented so that the macroblock information existing in the slice information is not erroneously decoded. Data D3 to the conversion unit 4 is output by the control unit 11 without being subjected to inverse quantization processing and inverse DCT processing. When the error detection unit 14 detects a decoding error, an error signal is sent to the error processing unit 13 of the decoding unit 2.

In addition, a macroblock information decoding error in the error detection unit 14 is detected in the following case.

(1) As a result of decoding the macroblock information, the corresponding data does not exist. For example, when the macroblock information is decoded and then reproduced with reference to a certain reference frame, the reference frame indicates “list 10”. This is the case when the reference frame list only exists from “list 0” to “list 8”.

(2) The value of the result of decoding the macroblock information does not exist in the encoding correspondence table. This is because, for example, the decoding result has a value of “40”.
This is the case when only the values from “0” to “30” are defined.

  The decoding error detection is not limited to the above method, and other detection methods may be used.

Further, depending on the detection method, a decoding error may not be detected correctly. As described above, when a decoding error occurs in the macroblock information, the data D2 to the inverse quantization unit 3 and the data to the inverse DCT conversion unit 4 are performed so that erroneous decoding is not performed on the macroblock information. D3 is output by the control unit 11 without being subjected to inverse quantization processing and inverse DCT processing. When the error detection unit 14 detects a decoding error, an error signal is sent to the error processing unit 13 of the decoding unit 2.

When a decoding error is detected by the error detection unit 14 and an error signal is sent, the error processing unit 13 performs an error concealment process. The following describes how to determine a reference frame when error concealment processing is performed in the error processing unit 13.
The detected decoding error will be described separately for (a) when a decoding error is detected with macroblock information and (b) when a decoding error is detected with slice header information.

(A) When a decoding error is detected in the macroblock information When the macroblock in which the decoding error has occurred is an intra macroblock (a-1)
And the case of the inter macroblock (a-2) will be further described.

(A-1) When a decoding error of an intra macroblock is detected When a decoding error is detected in an intra macroblock, a reference frame for error concealment processing is determined by the following processing.

4 and 7 show a reference frame for performing error concealment processing when a decoding error is detected in an intra macroblock. In addition,
H. In the H.264 standard, the reference frame is defined as follows.

(A) There is no reference frame in an intra macroblock, and processing is always performed with 16 pixels × 16 pixels as one unit as shown in FIG. 4 except for processing in the intra prediction unit.

(B) Inter macroblocks can be set as processing units in areas obtained by dividing macroblocks as shown in FIG. 6 called seven types of partitions.

(C) In an inter macro block, a reference frame can be set in units of 8 pixels × 8 pixels.

(D) Multiple reference frames cannot be set in one partition.

Next, a reference frame selection method in the error concealment process when a decoding error occurs in the intra macroblock X will be described with reference to FIGS. FIG. 5 is an operation flowchart showing a method of selecting a reference frame when performing error concealment processing.

First, whether the adjacent macroblock A adjacent to the left of the intra macroblock X in which the decoding error has occurred is an inter macroblock (unless the adjacent macroblock A is not the same slice as the intra macroblock X), or otherwise (the adjacent macroblock A case where there is no A or a case where the adjacent macroblock A is not the same slice as the intra macroblock X is determined (step 51, (1) in FIG. 4).

If the adjacent macroblock A is an inter macroblock (Yes in step 51), it is next determined whether or not the adjacent macroblock A is divided into any of the seven partitions (step 52). If the macro block A is not divided into the partitions (No in step 52), the frame a1 referred to by the macro block A is set as a reference frame (step 53a, (2) in FIG. 4). If the macroblock A is divided into any of the seven partitions (Yes in step 52), the upper right partition (the upper left pixel of the intra macroblock X is adjacent to the left among the macroblock A partitions). Frame a2 (in FIG. 7 (
1) is used as a reference frame (step 53b).

Here, another example of the reference frame determination method in step 53b will be described with reference to (2) of FIG. AP1 to AP4 in (2) of FIG. 7 are 8 pixels of inter macroblock A ×
An 8-pixel partition is shown. The reference frame is a unit of 8 pixels × 8 pixels (hereinafter,
This unit can be referred to as a sub macroblock unit) (however, as described above, a plurality of reference frames cannot be set in one partition). For example, partition AP1 assigns reference frame a3 to partition AP
Assume that 2 refers to the reference frame a4, the partition AP3 refers to the reference frame a5, and the partition AP4 refers to the reference frame a3. Furthermore, partition AP1
, Weight 1 is set for partition AP2, weight 1 is set for partition AP3, and weight 2 is set for partition AP4. Then, partitions AP1 to AP
4 occupies one-fourth of the pixels of the macroblock, so if the weight of the frame being referenced is determined by (number of occupied pixels) × (weight), the ratio is the reference frame a3: reference frame a4 : Reference frame a5 = 3: 2: 1. Therefore, intra macroblock X
The reference frame for the error concealment processing is determined to be the reference frame a3 having the highest weight.

If there are a plurality of reference frames having the highest weight, the partition closest to the upper left pixel of the intra macroblock X in which the decoding error is detected among the partitions that refer to the plurality of reference frames. The frame being referred to is set as a reference frame. For example, the partition AP1 has the reference frame a5, the partition AP2 has the reference frame a4, the partition AP3 has the reference frame a5, and the partition AP
4 refers to the reference frame a3, and the partition AP1 has a weight of 1
, Weight 2 for partition AP2, weight 1 for partition AP3, partition AP
Assume that the weight 2 is set to 4. Then, the weight ratio becomes reference frame a3: reference frame a4: reference frame a5 = 1: 1: 1. Since the partition closest to the upper left of the intra macroblock X in which the decoding error is detected is the partition AP2, the reference frame of the intra macroblock X is determined as the reference frame a4 referred to by the partition AP2.

Next, if the adjacent macro block A is other than the intra macro block (step 5).
1), whether the adjacent macroblock B adjacent to the intra macroblock X in which the decoding error has occurred is an inter macroblock or other than that (when there is no adjacent macroblock B or the adjacent macroblock B is the intra macroblock X) (Including the case where the slice is not the same slice) (step 54).

If the adjacent macroblock B is an inter macroblock (Yes in step 54), it is determined whether the macroblock B is divided into any of the seven partitions (step 55). If the macroblock B is not divided into the partitions (No in step 55), the frame b1 referred to by the macroblock B is set as a reference frame (step 56a, (3) in FIG. 4). If the macro block B is divided into any of the seven partitions (Yes in step 55), the lower left partition (the upper left pixel of the intra macro block X) is adjacent to the upper left of the macro block B partitions. The frame b2 (see (3) in FIG. 7) referred to by the current partition) is set as the reference frame (step 56b).

Here, another example of the reference frame determination method in step 56b will be described with reference to FIG. BP1 to BP4 in (4) of FIG.
An 8-pixel partition is shown. Similarly, a reference frame can refer to a different frame for each sub macroblock. For example, the partition BP1 uses the reference frame b3, the partition BP2 uses the reference frame b4, and the partition BP3.
Assume that reference frame b5 and partition BP4 refer to reference frame b3, respectively. Further, assume that weight 1 is set for partition BP1, weight 1 is set for partition BP2, weight 2 is set for partition BP3, and weight 2 is set for partition BP4. Then, since the partitions BP1 to BP4 indicate one-fourth of the pixels of the macroblock, if the weight of the frame being referred to is calculated by (number of occupied pixels) × (weight), the ratio is the reference frame b3 : Reference frame b4: Reference frame b5 = 3: 1: 2. Therefore, the reference frame when the intra macroblock X performs the error concealment process is determined as the reference frame b3 having the highest weight.

If there are a plurality of reference frames having the highest weight, the partition closest to the upper left pixel of the intra macroblock X in which the decoding error is detected among the partitions that refer to the plurality of reference frames. The frame being referred to is set as a reference frame. For example, the partition BP1 has the reference frame b5, the partition BP2 has the reference frame b4, the partition BP3 has the reference frame b6, and the partition BP.
4 refers to the reference frame b3, and further assigns a weight of 1 to the partition BP1.
, Weight 1 for partition BP2, weight 2 for partition BP3, partition BP
Assume that the weight 2 is set to 4. Then, the weight ratio is the reference frame b3: reference frame b4: reference frame b5: reference frame b6 = 2: 1: 1: 2, but the partition closest to the upper left pixel of the macroblock X in which the decoding error is detected. Is the partition BP3, the reference frame of the macroblock X is determined to be the reference frame a6 referenced by the partition BP3.

Next, if the adjacent macro block B is other than the intra macro block (step 5).
4 No), whether the adjacent macroblock C at the upper right of the intra macroblock X in which the decoding error has occurred is an inter macroblock or other than that (when there is no adjacent macroblock C or the adjacent macroblock C is the same slice as the intra macroblock X) (Including the case where it is not)) (step 57).

If the adjacent macroblock C is an inter macroblock (Yes in step 57), it is determined whether the macroblock C is divided into any of seven partitions (step 58). If the macroblock C is not divided into the partitions (No in step 58), the frame c1 referred to by the macroblock C is set as a reference frame (step 59a, (4) in FIG. 4). If the macro block C is divided into any of the seven partitions (Yes in step 58), the frame c2 ((5) in FIG. 7) referred to by the lower left partition among the macro block C partitions.
Reference) is set as a reference frame (step 59b).

Next, if the adjacent macroblock C is other than the intra macroblock (step 5)
7), whether the adjacent macroblock D in the upper right of the intra macroblock X in which the decoding error has occurred is an inter macroblock or other than that (if there is no adjacent macroblock D or the adjacent macroblock D is the same slice as the intra macroblock X (Including the case where it is not) (step 60).

If the adjacent macroblock D is an inter macroblock (Yes in step 60), it is determined whether the macroblock D is divided into any of the seven partitions (step 61). If the macroblock D is not divided into the partitions (No in step 61), the frame d1 referenced by the macroblock D is set as a reference frame (step 62a, (5) in FIG. 4). Further, if the macroblock D is divided into any of the seven partitions (Yes in step 61), the frame d2 ((6 in FIG. 7) referred to by the lower right partition among the macroblock D partitions). )
Reference) is set as a reference frame (step 62b).

If the adjacent block D is other than the inter macroblock (No in step 60),
A frame decoded immediately before the frame including the intra macroblock X is referred to (step 63). The step 63 is not limited to this, and the frame displayed immediately before the frame including the intra macroblock X may be used as the reference frame.

In step 51, step 54, step 57, and step 60, if the adjacent block of the intra macroblock X is not the same slice as the intra macroblock X, the flow proceeds to step 54, step 57, step 60, and step 63, respectively. However, if there is an adjacent macroblock even in a slice different from the intra macroblock X and the macroblock is an inter macroblock, the process may proceed to step 52, step 55, step 68, and step 61, respectively.

When importance is attached to the processing speed of the decoder 1, at least step 51 to step 5
4 may be executed, and the subsequent processing may be omitted at the stage where No is determined in Step 54 and Step 57, and Step 63 may be executed. For example, if it is determined No in step 54, step 63 may be executed immediately (dotted line portion in FIG. 5).

When the reference frame is determined as described above, the error processing unit 13 performs error concealment processing using the macroblock in the reference frame located at the same address as the intra macroblock X (for example, ( 2) to (5) X1 to X4) All the temporary motion vector data D11 are sent to the motion compensation unit 8 as 0. The error concealment process after the determination of the reference frame is not limited to this method, and any method may be used as long as the error concealment process is performed using the determined reference frame.

(A-2) When a decoding error of an inter macro block is detected When a decoding error is detected in an inter macro block, the error processing unit 13
Determines the reference frame for the error concealment process by the following process.

First, an operation when a decoding error occurs in an inter macro block (hereinafter referred to as partition Y1) that is not divided into partitions will be described with reference to FIGS.

FIG. 8 shows that a decoding error is detected in the partition Y1 of the inter macroblock,
It is the flowchart which showed the selection method of the reference frame at the time of carrying out error concealment processing of it. FIG. 9 illustrates an adjacent block of the partition Y1 in which a decoding error has occurred. In the following description, the adjacent block means an adjacent macroblock in principle, and if the adjacent macroblock is divided into partitions, it means an adjacent partition.

First, an adjacent block E left adjacent to the upper left pixel of the partition Y1 in which a decoding error has occurred is
(A) Intra macroblock or
(I) If it does not exist,
(C) If it is another slice,
(Step 81, see FIG. 9). When the condition of step 81 is not satisfied, the frame e1 referred to by the adjacent block E is set as a reference frame (step 82).

Next, if the above conditions (a) to (c) in step 81 are satisfied, the adjacent block F (upper adjacent to the upper left pixel of the partition Y1 where the decoding error has occurred is described.
It may be determined whether any of the above (a) to (c) is applicable (step 83, see FIG. 9). If the condition of step 83 is not met,
The frame f1 referred to by the adjacent block F is set as a reference frame (step 84).

Next, if the above conditions (a) to (c) in step 83 are satisfied, the adjacent block G adjacent to the upper right side of the upper right pixel of the partition Y1 in which the decoding error has occurred.
It is determined whether (which may be divided into partitions) corresponds to any one of the above (a) to (c) (step 85, see FIG. 9). When the condition of step 85 is not satisfied, the frame g1 referred to by the adjacent block G is set as a reference frame (step 86).

Next, if the above conditions (a) to (c) in step 85 are satisfied, the adjacent block H adjacent to the upper left of the upper left pixel of the partition Y1 in which the decoding error has occurred.
It is determined whether (which may be divided into partitions) corresponds to any one of the above (a) to (c) (step 87, see FIG. 9). When the condition of step 87 is not satisfied, the frame h1 referred to by the adjacent block H is set as a reference frame (step 88).

Next, if the above conditions (a) to (c) are satisfied in step 87, the frame decoded immediately before the frame including the inter macroblock Y1 is referred to (step 89). Note that step 89 is not limited to this, and the frame displayed immediately before the frame including the inter macroblock Y1 may be used as the reference frame.

In Step 81, Step 83, Step 85, and Step 87, (A) to (A) to
If the condition of (c) is not met, Step 82, Step 84, Step 86, respectively.
, Proceed to step 88, but step 81, step 83, step 85,
The determination condition in step 87 may be limited to the above (a) and (b).

When the processing speed of the decoder 1 is important, at least step 81 to step 8
3 may be executed, and the subsequent processing may be omitted at the stage where it is determined to be applicable in Step 83 and Step 85, and Step 89 may be executed. For example, if it is determined in step 83 that this is the case, step 89 may be immediately executed (dotted line portion in FIG. 8).

Next, FIG. 10 and FIG. 11 are used when a decoding error occurs in an inter macro block (hereinafter referred to as partition Y2) divided into one of seven partitions as shown in FIG. I will explain.

FIG. 10 is a flowchart showing a method of selecting a reference frame when a decoding error is detected in the partition Y2 of the inter macroblock and error concealment processing is performed on the decoding error. FIG. 11 is a diagram illustrating a reference frame selection method when a decoding error is detected in the partition Y2 of the inter macroblock and error concealment processing is performed.

First, the partition Y2 where the decoding error is detected is less than 8 pixels × 8 pixels, or 8
It is determined whether or not the pixel × 8 pixels or more (step 101). If it is determined that the partition Y2 is less than 8 pixels × 8 pixels (less than step 101), it is determined whether the reference frame is correctly decoded (step 102). If the reference frame of the sub-macroblock including the partition Y2 (a block of 8 pixels × 8 pixels, which is a quarter of the macroblock, the same applies hereinafter) is decoded correctly (Yes in step 102), the partition Y2 is included. The frame k referred to by the sub macroblock is set as the reference frame (step 103a, see (1) in FIG. 11). Further, when it is determined that the partition Y2 is 8 pixels × 8 pixels or more (from step 101) and when the reference frame of the sub macroblock including the partition Y2 is not correctly decoded (No in step 102).
, All the frames (one to three) that are referred to by sub-macroblocks that do not include partition Y2 in the inter-macroblock with partition Y2 are listed. As shown in (2) of FIG. The ratio of the frames referred to in the block is obtained (step 103b).

For example, in (2) of FIG. 11, if the partition Y2 is 8 pixels × 8 pixels, the sub macroblocks SMB1, SMB2, and SMB3 are inter macroblocks that include the partition Y2 and do not include the partition Y2. is there. If the sub macroblock SMB1 refers to the frame k1, the sub macroblock SMB2 refers to the frame k1, and the sub macroblock SMB3 refers to the frame k2, the sub macroblock SMB1 refers to the frame k1. The ratio of the frames being framed is 1/2 for the frame k1 and 1/4 for the frame k2.

Next, it is determined whether the number of frames referred to in the inter macroblock obtained in step 103b is the highest or one or more (step 104). If there is one frame with the highest ratio (Yes in step 104), that frame is set as a reference frame (step 105). For example, in the case of (2) in FIG. 11, the frame k1 becomes the reference frame of the partition Y2.

If there are two or more frames having the largest ratio (No in step 104), the value of the motion vector of the partition Y2 and the sub macroblock S in the same macroblock are used.
The values of motion vectors MB1 to SMB3 are received from the bitstream analysis unit 12 (step 106). Then, partition Y2, sub macroblocks SMB1, SMB2
, SMB3, the motion vectors UY2 (uy1, uy2), the motion vector US1 (u11, u12), the motion vector US2 (u21, u22), the motion vector US3 (u31, u32), and the motion vectors US1 to US32 A motion vector that is most similar to the motion vector UY2 is searched for in US3 (step 107). For example, each difference vector is obtained and the smallest one is set as a candidate (see (3) in FIG. 11). Then, the frame referenced by the sub macroblock having the motion vector most similar to the motion vector UY2 obtained in step 107 is set as the reference frame of the partition Y2 (step 108).

The process for determining the reference frame using the value of the motion vector is not limited to this method, and any process may be used as long as it is a process for determining the reference frame using the motion vector.

When the reference frame is determined as described above, error concealment processing is performed using the macroblock in the reference frame located at the same address as the macroblock including the partition Y1 or the partition Y2 (for example, the frame in FIG. 9). e1 to the hatched portion of the frame h1), the provisional motion vector data D11 are all sent to the motion compensation unit 8 as 0.

(B) When a decoding error is detected in the slice header information When a decoding error is detected in the slice header information and the error processing unit 13 receives the error signal from the error detection unit 14, any of the methods described below is used. Determine the reference frame.

For example, the error processing unit 13 sets a frame specified by “list 0” as a reference frame.

This is because the frame specified in “list 0” is originally set on the encoding side, but the frame that becomes “list 0” in that setting is often the most referenced frame. It is possible.

Further, for example, the error processing unit 13 obtains reproduction time information T (information such as STC) of the reproduction image stored in the reference frame buffer unit 9 from the control unit 11. Further, the reproduction time information T1 (information such as PTS) of the frame in which the decoding error is detected is obtained from the decoder 1.
Or a control unit having a function of controlling time information of the entire data receiver such as DEMUX (not shown). Then, the error processing unit 13 among the obtained reproduction time information T, the reproduction time information T (before or after the time T1) whose display time is closest to the time T1 of the reproduction time information of the frame where the decoding error is detected. A reference image is a reproduced image having (or may be). This is because an image having the reproduction time information T closest to the reproduction time information T1 of the reproduction time information of the frame in which the decoding error is detected is considered to be close to the image in which the decoding error is detected.

Alternatively, for example, a method may be used in which a frame decoded by the decoder 1 immediately before a frame in which a decoding error is detected is used as a reference frame. This is because the frame decoded immediately before is considered to be close to an image in which a decoding error is detected.

Further, for example, the frame types of all reproduced images stored in the reference frame buffer unit 9 are searched, whether or not there is an I picture, and the display time (ST of the frame)
C) (information such as C). If there is an I picture, a method may be used in which a frame having the display time closest to the display time of the frame in which the decoding error is detected is used as a reference frame.

Also, for example, for all the playback images stored in the reference frame buffer unit 9, it is searched whether or not a long-term-picture attached to a frame to be referred to for a long time is attached in the header, If there is a reproduced image to which long-term-picture is attached, a method may be used in which a frame having the closest display time to the frame in which the decoding time decoding error is detected is the reference frame.

For example, the following method may be used. In the following description, “Slices adjacent to the left”
Refers to a slice that touches the left side of the slice (slice X in FIG. 12A) (refers to slice A in the example of FIG. 12A), and “slice adjacent to the top” Refers to the slice that touches the upper side of the slice (in the example of FIG. 12A, slice B1, slice B2,
“Slice adjacent to the upper right” refers to a slice adjacent to the upper right corner of the slice (in the example of FIG. 12A, refers to the slice B3), and “slice adjacent to the upper left”. Indicates the slice that is adjacent to the upper left vertex of the slice (see FIG. 12 (
In the example of A), it refers to slice D).

When a slice adjacent to the left is decoded, a frame that is referenced at the highest rate in the slice is set as a reference frame. If there are a plurality of frames that are referenced at the highest rate in the slice, the frame that is referenced by the macroblock closest to the slice in which the decoding error is detected is used as the reference frame. .

If the left adjacent slice has not been decoded, it is determined whether the upper adjacent slice has been decoded. If the upper adjacent slice has been decoded, the largest percentage of that slice is decoded. The frame referred to in (1) is set as a reference frame. If there are a plurality of frames that are referenced at the highest rate in the slice, the frame that is referenced by the macroblock closest to the slice in which the decoding error is detected is used as the reference frame. . Here, when there are a plurality of adjacent slices above, it is determined whether all the slices are decoded, and then the reference frame is determined as described above. If it exists, it may be determined whether it is decoded in order from the upper left, and a reference frame may be determined when one decoded slice is detected.

If the upper adjacent slice has not been decoded, it is determined whether the upper right adjacent slice has been decoded. If the upper right adjacent slice has been decoded, the highest percentage of the slice is referenced. The frame that has been set as the reference frame. If there are a plurality of frames that are referenced at the highest rate in the slice, the frame that is referenced by the macroblock closest to the slice in which the decoding error is detected is used as the reference frame. .

If the slice adjacent to the upper right is not decoded, it is determined whether the slice adjacent to the upper left is decoded. If the slice adjacent to the upper left is decoded, it is referenced at the highest rate in the slice. The frame that has been set as the reference frame. If there are a plurality of frames that are referenced at the highest rate in the slice, the frame that is referenced by the macroblock closest to the slice in which the decoding error is detected is used as the reference frame. The method is fine.

Note that the “adjacent slices” defined here are “slices adjacent to the left”, respectively, and slices that are in contact with the left side of the slice (slice Y in FIG. 12B) (FIG. 12B)
In the example of slice a), “slice adjacent on top” is a slice in contact with the upper side of the first macroblock belonging to the slice (slice b in the example of FIG. 12B), “slice adjacent on top right” "Is the upper right adjacent slice to the upper right vertex of the first macroblock belonging to the slice (slice c in the example of FIG. 12B), and" the upper left adjacent slice "is the upper left of the first macroblock belonging to the slice It may be defined as a slice that is adjacent to the top of the top left corner (slice b in the example of FIG. 12B).

It should be noted that when determining whether or not decoding has been performed, it may be set so that it is not determined twice.

When the reference frame is determined as described above, the error processing unit 13 performs error concealment processing using the macroblock in the reference frame located at the same address as the macroblock of the slice in which the decoding error is detected, and provisional motion The vector data D11 is all set to 0 and sent to the motion compensation unit 8.

c According to an embodiment of the present invention, an optimum reference frame can be determined when a decoding error is detected and error concealment is performed. Note that after the reference frame is determined, error concealment processing is performed, but this processing is not limited to that described in the present embodiment, and other error concealment processing may be performed.

DESCRIPTION OF SYMBOLS 1 Decoder 2 Decoding part 3 Inverse quantization part 4 Inverse DCT conversion part 5 Decoding addition part 6 Intra prediction part 7 Filter 8 Motion compensation part 9 Reference frame buffer part 10 Switch 11 Control part 12 Bit stream analysis part 13 Error processing part 14 Error detector

Claims (4)

A decoding means for decoding the encoded information,
Error detection means for detecting a decoding error from the bitstream decoded by the decoding means;
A reference frame buffer for sequentially storing decoded reproduced images;
When the decoding error is detected in the macroblock information existing in the slice information in the bitstream by the error detection means and error concealment processing is performed, the decoded reproduction image stored in the reference frame buffer Then, a reference frame determination means that uses a frame having the closest display time to the slice where the decoding error is detected as a reference frame for error concealment,
And an error processing means for performing an error concealment process using the reference frame determined by the reference frame determining means. A decoding means for decoding the encoded information,
Error detection means for detecting a decoding error from the bitstream decoded by the decoding means;
A reference frame buffer for sequentially storing decoded reproduced images;
When the decoding error is detected in the macroblock information existing in the slice information in the bitstream by the error detection means and error concealment processing is performed, the decoded reproduction image stored in the reference frame buffer A frame including an I picture from the frame, and a reference frame determining unit using a frame having a decoding time closest to a slice in which the decoding error is detected as a reference frame of an error concealment among the frames including the I picture;
And an error processing means for performing an error concealment process using the reference frame determined by the reference frame determining means. A decoding means for decoding the encoded information,
Error detection means for detecting a decoding error from the bitstream decoded by the decoding means;
A reference frame buffer for sequentially storing decoded reproduced images;
When the decoding error is detected in the macroblock information existing in the slice information in the bitstream by the error detection means and error concealment processing is performed, the decoded reproduction image stored in the reference frame buffer To search for frames with long-term reference information in each header,
A reference frame determination unit that uses a frame having a display time closest to a slice in which the decoding error is detected among frames in which the long-term reference information exists, as a reference frame for error concealment;
And an error processing means for performing an error concealment process using the reference frame determined by the reference frame determining means. A decoding means for decoding the encoded information,
Error detection means for detecting a decoding error from the bitstream decoded by the decoding means;
A reference frame buffer for sequentially storing decoded reproduced images;
When a decoding error is detected in the macroblock information existing in the slice information in the bitstream by the error detection means, and the adjacent first slice is decoded when performing error concealment processing, the first Among the plurality of macroblocks in one slice, the first frame that is referred to in the largest proportion is set as a reference frame for error concealment, and the adjacent first slice is not decoded and is adjacent. When the second slice is decoded, the second frame that is referenced in the largest proportion among the plurality of macroblocks in the second slice is set as a reference frame for error concealment, and 2 slices are not decoded and the adjacent 3rd slice is decoded The third frame referenced in the highest proportion among the plurality of macroblocks in the third slice is used as the error concealment reference frame, and the third slice is decoded. If the fourth slice adjacent to the fourth slice is not decoded, the fourth frame that is referenced in the largest proportion among the plurality of macroblocks in the fourth slice is selected as the error concealment. A reference frame determining means as a reference frame, a decoding means for decoding the encoded information,
Error detection means for detecting a decoding error from the bitstream decoded by the decoding means;
A reference frame buffer for sequentially storing decoded reproduced images;
When the decoding error is detected in the macroblock information existing in the slice information in the bitstream by the error detection means and error concealment processing is performed, the decoded reproduction image stored in the reference frame buffer To search for frames with long-term reference information in each header,
A reference frame determination unit that uses a frame having a display time closest to a slice in which the decoding error is detected among frames in which the long-term reference information exists, as a reference frame for error concealment;
And an error processing means for performing an error concealment process using the reference frame determined by the reference frame determining means.

Images