WO2008079503A2 - Method and apparatus for adaptive error resilience for video decoders - Google Patents

Method and apparatus for adaptive error resilience for video decoders Download PDF

Info

Publication number
WO2008079503A2
WO2008079503A2 PCT/US2007/082594 US2007082594W WO2008079503A2 WO 2008079503 A2 WO2008079503 A2 WO 2008079503A2 US 2007082594 W US2007082594 W US 2007082594W WO 2008079503 A2 WO2008079503 A2 WO 2008079503A2
Authority
WO
WIPO (PCT)
Prior art keywords
macro
error
video
bit
block
Prior art date
Application number
PCT/US2007/082594
Other languages
French (fr)
Other versions
WO2008079503A3 (en
Inventor
Raghavan Subramaniyan
Original Assignee
Motorola, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Motorola, Inc. filed Critical Motorola, Inc.
Publication of WO2008079503A2 publication Critical patent/WO2008079503A2/en
Publication of WO2008079503A3 publication Critical patent/WO2008079503A3/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/89Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder
    • H04N19/895Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder in combination with error concealment
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
    • H04N19/89Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving methods or arrangements for detection of transmission errors at the decoder

Definitions

  • the invention relates generally to the field of video decoding and in particular to error resiliency capability of video decoders operating in an error prone environment.
  • video decoders have to contend with impairments in the input data due to channel conditions.
  • An example is when a video data signal is transmitted over a wireless channel.
  • the signal is transmitted over a channel that is subject to impairments such as noise, interference and fading.
  • Mechanisms for detecting errors in the data are known, but since the data throughput is typically high for video data, video packets cannot be retransmitted without reducing overall throughput. The frame rate cannot be reduced without adversely affecting the Quality of Service (QOS). Further, certain types of errors may not be detectable.
  • QOS Quality of Service
  • QOS Quality of Service
  • certain types of errors may not be detectable.
  • Error Resilience A video decoder with good error resilience can become an important differentiator in a video communication system.
  • Video coding standards such as H.264 (MPEG-4 Part 10), H.263 , MPEG-2 , and MPEG-4, defined by the Moving Picture Expert Group (MPEG), have mechanisms that allow implementers to build error resilience into their video systems. These mechanisms include resynchronization markers, splitting the data into packets, and INTRA macro-block refresh techniques. Video communication systems can take advantage of these mechanisms to improve video quality in error prone environments.
  • MPEG Moving Picture Expert Group
  • errors can be categorized into two types: packet loss and bit errors. Packet loss occurs in packet switched networks when either the transmitted video packet is lost in the network, or when the receiver detects an error in the video packet and chooses not to pass on the video packet to the decoder.
  • the video decoder does not have prior knowledge of the type of error expected.
  • the same video decoder library may be used with several applications, each of which may have a different error characteristic.
  • a video decoder to be able to detect and dynamically adapt it error mitigation scheme to the type of error encountered.
  • Prior decoder error resiliency algorithms are either (a) designed and optimized for bit errors in the channel or (b) designed to handle only packet losses. In either case, the expected type of error is assumed, and the error concealment strategy is optimized for that type of error. It is possible that the decoder can still handle both types of errors, but the obtained video quality would be optimal for one type of error. For example, a decoder designed to work optimally with bit errors may achieve reasonable performance when there are only packet losses.
  • bit error detection is difficult because of variable length codes used in encoding video data. Packet losses are relatively easier to detect. For example, the receiving terminal can determine a loss of a packet based on the packet number. There may or may not be a mechanism for communicating this information to the decoder. None of the prior video decoders enable optimal resilience to both bit errors and packet errors. Brief Description of the Figures
  • FIG. 1 is a block diagram of an exemplary video decoder in accordance with some embodiments of the invention.
  • FIG. 2 is a state chart an exemplary video decoder in accordance with some embodiments of the invention.
  • FIG. 3 is a flow chart of an exemplary method for video decoding in accordance with some embodiments of the invention.
  • embodiments of the invention described herein may be comprised of one or more conventional processors and unique stored program instructions that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of error resilience in video decoders, described herein.
  • the non-processor circuits may include, but are not limited to, a radio receiver, a radio transmitter, signal drivers, clock circuits, power source circuits, and user input devices. As such, these functions may be interpreted as a method to provide an error resilient video decoder.
  • a video decoder does not have prior knowledge of the type of error expected.
  • the same video decoder library may be used with several applications, each of which may have a different error characteristic.
  • a video decoder to be able to detect the type of error and dynamically adapt its error mitigation scheme to the type of error encountered.
  • One embodiment of the present invention relates to a method for decoding a video bit-stream in a video decoding system.
  • the video bit-stream comprises a number of video frames each containing numbered macro-blocks of video data.
  • the video bit-stream is received as a plurality of packets and the video decoding system is operated in a packet loss state, in which the video bit-stream is decoded by a first video decoder configured to be resilient to loss of one or more packets of the plurality of packets. If a bit error is detected the video decoding system is operated in a bit error state, in which the video bit-stream is decoded by a second video decoder configured to be resilient to bit errors.
  • FIG. 1 is a block diagram of an exemplary video decoding system in accordance with some embodiments of the invention.
  • the video decoding system 100 facilitates adaptive concealment of channel errors. Referring to FIG. 1, a digital video signal (a bit stream) 102 is input to an error type detector 104.
  • the error type detector 104 is operable to analyze the error patterns observed in the received video signal and to automatically determine the nature of error in the signal.
  • the error type detector 104 produces a switching signal 106 that is used to control a selector switch 108. If the error type detector 104 detects a bit error, the video signal 102 is passed, by switch 108, to a bit-error resilient decoder 110 to be decoded. If the error type detector 104 detects a packet loss, the video signal 102 is passed to a packet-loss resilient decoder 112 to be decoded. In either case, the decoded signal is output for further processing or display. In this way, both bit errors and packet loss are compensated for.
  • a signal 114 related to the occurrence of bit errors is supplied to the error type detector 104. This enables the error type detector 104 to determine whether the period of time since the last bit error was detected has exceeded a certain threshold.
  • the signal 114 may, for example, indicate when bit errors occur, when the time period has been exceeded or the time since the last bit error.
  • the signal 114 may be produced from data recovery process operated in a receiver/error detector 116, for example. When a bit error occurs in a bit-stream, it is often detected much later in the bit-stream.
  • a data recovery process is a method to determine the actual location of the first bit error that occurred in the bit-stream is required so that the "correct" data can be recovered.
  • the receiver/error detector 116 may also detect inconsistencies in the macro-block numbers embedded in the video bit-stream.
  • the video decoder in FIG. 1 may be in one of two states: a "Packet Loss state” (PL) and a "Bit Error state” (BE).
  • FIG. 2 is an exemplary state chart of the video decoding system.
  • the first state is the packet loss state 202, in which the packet-loss resilient decoder is active, and is used to decode the video signal.
  • the packet-loss resilient decoder is optimized for handling packet losses.
  • the second state is a bit error state 204, in which the bit-error resilient decoder is active and is used to decode the video signal.
  • the bit-error resilient decoder is optimized for handling bit errors.
  • the initial state is the packet loss state 202.
  • the system moves to the bit error state 204, as depicted by the state transition arc 206.
  • the system remains in the bit error state 204 unless no further bit errors have been received in a given time period.
  • the system moves back to the packet loss state 202 if no new bit errors are detected for five seconds, as depicted by the state transition arc, 208.
  • the bit error resilient video decoder which is optimized for handling bit errors, is used in the bit error state.
  • Known mechanisms may be employed in the decoder for error detection.
  • the bit error resilient decoder expects only bit errors, and has stringent and sophisticated methods to detect, localize and conceal errors.
  • the video decoding system remains in the state until a significant amount of time has elapsed without any errors being detected. It then moves to the packet loss state. If no error is detected for a specified period (5 seconds for example) the system moves to the packet loss state. In the packet loss state, the video decoder error detection mechanism is modified to distinguish between bit errors and packet loss. Bit errors are detected using techniques known to those of ordinary skill in the art.
  • the video decoder If only packet loss is present in the video signal, the video decoder remains in the packet loss state. In the packet loss state, no data recovery is performed, and the lost packet(s) are concealed. In the packet loss state, if a bit error is detected, the decoder immediately transits to the bit error state.
  • error concealment strategies can be applied optimally. Based on the adaptively determined error type, error concealment is performed such that it is optimal for the particular error type. This results in better overall error resilience performance. This method can be used to enhance the performance of prior error mitigation schemes.
  • the specific region to be concealed or mitigated can be identified.
  • the decoder is able to distinguish between bit errors and packet loss without any additional information about the channel.
  • the video decoder relies on the fact that packet loss does not result in any syntax errors. Rather, only semantic errors would be encountered.
  • the types of semantic errors encountered in an MPEG-4 bit stream due to packet loss are:
  • the starting macro-block number of a video packet is inconsistent with the last decoded macro-block. However, the decoded macro-block number is valid, i.e., it is a legal value within the prescribed range. This happens when a video packet or group of video packets is lost.
  • VOP start code The Picture Start Code (VOP start code) for a new frame is not found. This can happen when the first video packet or a group of video packets at the beginning of a frame is lost.
  • bit error and packet loss errors may not be distinguished based on semantic errors alone.
  • FIG. 3 is a flow chart of an exemplary method for video decoding in accordance with some embodiments of the invention.
  • FIG. 3 depicts a method for distinguishing between the packet loss and bit errors with a high probability.
  • the method begins at start block 302 when there is an inconsistency in macro-block numbers (MBN 's) when decoding a new video packet. For example, if the new macro-block number (MBN) is not one greater than the previous macro-block number or if the MBN is outside of the expected range.
  • MBN macro-block number
  • the expected macro-block number is the first macro-block (number 0) of a frame, as depicted by the positive branch from decision block 304, a check is made at decision block 36 to determine if the block number is greater than one quarter of the total number N of macro-blocks in a video frame.
  • the value MBN DIFF is set, at block 308, to be the difference between the current macro block number (MBN) and the expected block (i.e. the block number required for consistency). If the current block is not expected to be the first block of a sequence, as depicted by the negative branch from decision block 304, a check is made at decision block 310 to determine if the block number is greater than or equal to the expected block number.
  • block number is within the expected range, as depicted by the positive branch from decision block 310, flow continues to block 308 where the value MBN DIFF. It expected that the MBN is greater than or equal to the expected MBN and less than the total number of blocks N in the frame.
  • a check is made to determine if the value MBN DIFF is greater than N/2 (half the number of macro-block in the frame). If it is, as depicted by the positive branch from decision block 312, then the error is judged to be a bit error and at block 314 the bit error is signaled and the process ends at block 316.
  • the bit error may be signaled by setting a flag, for example.
  • a bit error is also signaled if the block number is less than the expected block number or outside of the expected range, as depicted by the negative branch from decision block 310. If the value MBN_DIFF is not greater than N/2, as depicted by the negative branches from decision block 312, the error is judged to be a packet loss and is signaled at block 318 and the process terminates at block 316.
  • the threshold values (N/2 and N/4) can be set to be other fractions of the frame size, N.
  • N is total number of macro-blocks in a frame
  • LAST MBN is the previously decoded macro-block number
  • EXPECTED MBN is the expected value of the next macro-block number (equals 0 for a new frame, and equals LAST MBN+1 otherwise)
  • MBN is the decoded macro-block number from the new video packet.
  • MBN EXPECTED MBN
  • the values Tl and T2 are threshold values and can be set to be any fraction of the frame size, N.
  • Tl is set to one half of the frame size and T2 is set to one quarter of the frame size.
  • the BIT-ERROR flag is set to one if the error is judged to be a bit error and is set to zero if the error is judged to be a packet loss.
  • Decoderl A version that is optimized for handling bit errors.
  • Decoder2 A version that is optimized for handling packet errors.
  • Decoder3 A version corresponding to the embodiment described above. The performance was tested for various Quarter Common Intermediate Format (QCIF) encoded bit-streams. The bit-streams were encoded using various encoding parameters as given in TABLE 1 below.
  • QCIF Quarter Common Intermediate Format
  • PSNR objective quality metrics
  • bit error bursts were simulated with various error rates.
  • the average error burst length was chosen to be 2 milliseconds.
  • the results for QCIF bit-streams listed above are shown in TABLE 3 below.
  • Decoder3 which is in accordance the present invention, performs very close to Decoder2 when there are errors due to lost packets and very close to Decoder 1 when there are bit errors. Thus, it is successfully able to adapt to the type of error and give optimal performance in both scenarios.
  • the gain in terms of PSNR is fairly significant, especially at 1% and 2% packet loss.
  • the error resilient video decoding system described above provides enhanced video quality in error prone environments.
  • the system has application in mobile and handheld devices such as mobile phones and computer, and other client-server applications, such as video surveillance, that operate in error prone environments.

Abstract

A method and apparatus for decoding a video bit-stream received as a plurality of data packets. An error detector (116) detects error in the video bit-stream and an error type detector (104) determines if a detected error is due to a bit error or a packet loss. A bit-error resilient video decoder (110) decodes the video bit-stream if the error is determined to be due to a bit error and a packet-loss resilient video decoder (112) decodes the video bit-stream if the error is determined to be due to a packet loss. In one embodiment, errors in a video bit-stream are detected using inconsistencies in macro-block numbers.

Description

METHOD AND APPARATUS FOR ADAPTIVE ERROR RESILIENCE FOR VIDEO
DECODERS
Field of the Invention
[0001] The invention relates generally to the field of video decoding and in particular to error resiliency capability of video decoders operating in an error prone environment.
Background
[0002] In some situations, video decoders have to contend with impairments in the input data due to channel conditions. An example is when a video data signal is transmitted over a wireless channel. Here, the signal is transmitted over a channel that is subject to impairments such as noise, interference and fading. This results in errors introduced in the received video data. Mechanisms for detecting errors in the data are known, but since the data throughput is typically high for video data, video packets cannot be retransmitted without reducing overall throughput. The frame rate cannot be reduced without adversely affecting the Quality of Service (QOS). Further, certain types of errors may not be detectable. As a result there is a need for a decoder that can cope with the errors in the video data stream and still deliver reasonable video quality. This is broadly called "Error Resilience". A video decoder with good error resilience can become an important differentiator in a video communication system.
[0003] Video coding standards such as H.264 (MPEG-4 Part 10), H.263 , MPEG-2 , and MPEG-4, defined by the Moving Picture Expert Group (MPEG), have mechanisms that allow implementers to build error resilience into their video systems. These mechanisms include resynchronization markers, splitting the data into packets, and INTRA macro-block refresh techniques. Video communication systems can take advantage of these mechanisms to improve video quality in error prone environments. [0004] From a video decoder's point of view, errors can be categorized into two types: packet loss and bit errors. Packet loss occurs in packet switched networks when either the transmitted video packet is lost in the network, or when the receiver detects an error in the video packet and chooses not to pass on the video packet to the decoder. Bit errors occur is several situations. In circuit switched networks, channel impairments can manifest as bit errors in the input to the decoder. Even in packet switched networks, bit errors can appear in the decoder input either when the packet error cannot be detected by the receiver, or when the receiver detects the error and still chooses to pass the video data to the video decoder. The last mentioned case allows for the video decoder to recover partial good data and deliver improved video quality.
[0005] In most practical systems, the video decoder does not have prior knowledge of the type of error expected. In fact the same video decoder library may be used with several applications, each of which may have a different error characteristic. Thus, there is need for a video decoder to be able to detect and dynamically adapt it error mitigation scheme to the type of error encountered.
[0006] Prior decoder error resiliency algorithms are either (a) designed and optimized for bit errors in the channel or (b) designed to handle only packet losses. In either case, the expected type of error is assumed, and the error concealment strategy is optimized for that type of error. It is possible that the decoder can still handle both types of errors, but the obtained video quality would be optimal for one type of error. For example, a decoder designed to work optimally with bit errors may achieve reasonable performance when there are only packet losses.
[0007] In terms of error detection, bit error detection is difficult because of variable length codes used in encoding video data. Packet losses are relatively easier to detect. For example, the receiving terminal can determine a loss of a packet based on the packet number. There may or may not be a mechanism for communicating this information to the decoder. None of the prior video decoders enable optimal resilience to both bit errors and packet errors. Brief Description of the Figures
[0008] The accompanying figures, in which like reference numerals refer to identical or functionally similar elements throughout the separate views and which together with the detailed description below are incorporated in and form part of the specification, serve to further illustrate various embodiments and to explain various principles and advantages all in accordance with the present invention.
[0009] FIG. 1 is a block diagram of an exemplary video decoder in accordance with some embodiments of the invention.
[0010] FIG. 2 is a state chart an exemplary video decoder in accordance with some embodiments of the invention.
[0011] FIG. 3 is a flow chart of an exemplary method for video decoding in accordance with some embodiments of the invention.
[0012] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the present invention.
Detailed Description
[0013] Before describing in detail embodiments that are in accordance with the present invention, it should be observed that the embodiments reside primarily in combinations of method steps and apparatus components related to error resilience in video decoders. Accordingly, the apparatus components and method steps have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present invention so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
[0014] In this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element that is preceded by "comprises ...a" does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.
[0015] It will be appreciated that embodiments of the invention described herein may be comprised of one or more conventional processors and unique stored program instructions that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all of the functions of error resilience in video decoders, described herein. The non-processor circuits may include, but are not limited to, a radio receiver, a radio transmitter, signal drivers, clock circuits, power source circuits, and user input devices. As such, these functions may be interpreted as a method to provide an error resilient video decoder. Alternatively, some or all functions could be implemented by a state machine that has no stored program instructions, or in one or more application specific integrated circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic. Of course, a combination of the two approaches could be used. Thus, methods and means for these functions have been described herein. Further, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating such software instructions and programs and ICs with minimal experimentation.
[0016] In most practical systems, a video decoder does not have prior knowledge of the type of error expected. In fact, the same video decoder library may be used with several applications, each of which may have a different error characteristic. Thus, there exists a need for a video decoder to be able to detect the type of error and dynamically adapt its error mitigation scheme to the type of error encountered.
[0017] One embodiment of the present invention relates to a method for decoding a video bit-stream in a video decoding system. The video bit-stream comprises a number of video frames each containing numbered macro-blocks of video data. The video bit-stream is received as a plurality of packets and the video decoding system is operated in a packet loss state, in which the video bit-stream is decoded by a first video decoder configured to be resilient to loss of one or more packets of the plurality of packets. If a bit error is detected the video decoding system is operated in a bit error state, in which the video bit-stream is decoded by a second video decoder configured to be resilient to bit errors.
[0018] In a further embodiment, an inconsistency in the numbering of macro-blocks is detected and a bit error is signaled if a current macro-block number is less than a previous macro-block number. A bit error is also signaled if a difference between a current macro-block number and an expected macro-block number exceeds a threshold value. [0019] FIG. 1 is a block diagram of an exemplary video decoding system in accordance with some embodiments of the invention. The video decoding system 100 facilitates adaptive concealment of channel errors. Referring to FIG. 1, a digital video signal (a bit stream) 102 is input to an error type detector 104. The error type detector 104 is operable to analyze the error patterns observed in the received video signal and to automatically determine the nature of error in the signal. The error type detector 104 produces a switching signal 106 that is used to control a selector switch 108. If the error type detector 104 detects a bit error, the video signal 102 is passed, by switch 108, to a bit-error resilient decoder 110 to be decoded. If the error type detector 104 detects a packet loss, the video signal 102 is passed to a packet-loss resilient decoder 112 to be decoded. In either case, the decoded signal is output for further processing or display. In this way, both bit errors and packet loss are compensated for. A signal 114 related to the occurrence of bit errors is supplied to the error type detector 104. This enables the error type detector 104 to determine whether the period of time since the last bit error was detected has exceeded a certain threshold. The signal 114 may, for example, indicate when bit errors occur, when the time period has been exceeded or the time since the last bit error. The signal 114 may be produced from data recovery process operated in a receiver/error detector 116, for example. When a bit error occurs in a bit-stream, it is often detected much later in the bit-stream. A data recovery process is a method to determine the actual location of the first bit error that occurred in the bit-stream is required so that the "correct" data can be recovered. The receiver/error detector 116 may also detect inconsistencies in the macro-block numbers embedded in the video bit-stream.
[0020] The video decoder in FIG. 1 may be in one of two states: a "Packet Loss state" (PL) and a "Bit Error state" (BE). FIG. 2 is an exemplary state chart of the video decoding system. The first state is the packet loss state 202, in which the packet-loss resilient decoder is active, and is used to decode the video signal. The packet-loss resilient decoder is optimized for handling packet losses. The second state is a bit error state 204, in which the bit-error resilient decoder is active and is used to decode the video signal. The bit-error resilient decoder is optimized for handling bit errors. In the embodiment shown in FIG. 2, the initial state is the packet loss state 202. When a bit error is detected by the error type detector, the system moves to the bit error state 204, as depicted by the state transition arc 206. The system remains in the bit error state 204 unless no further bit errors have been received in a given time period. In example, the system moves back to the packet loss state 202 if no new bit errors are detected for five seconds, as depicted by the state transition arc, 208.
[0021] The bit error resilient video decoder, which is optimized for handling bit errors, is used in the bit error state. Known mechanisms may be employed in the decoder for error detection. The bit error resilient decoder expects only bit errors, and has stringent and sophisticated methods to detect, localize and conceal errors. Once in the bit error state, the video decoding system remains in the state until a significant amount of time has elapsed without any errors being detected. It then moves to the packet loss state. If no error is detected for a specified period (5 seconds for example) the system moves to the packet loss state. In the packet loss state, the video decoder error detection mechanism is modified to distinguish between bit errors and packet loss. Bit errors are detected using techniques known to those of ordinary skill in the art. If only packet loss is present in the video signal, the video decoder remains in the packet loss state. In the packet loss state, no data recovery is performed, and the lost packet(s) are concealed. In the packet loss state, if a bit error is detected, the decoder immediately transits to the bit error state.
[0022] Once the error is determined, known error concealment strategies can be applied optimally. Based on the adaptively determined error type, error concealment is performed such that it is optimal for the particular error type. This results in better overall error resilience performance. This method can be used to enhance the performance of prior error mitigation schemes.
[0023] In addition to identifying the type of error, the specific region to be concealed or mitigated can be identified.
[0024] The detailed embodiment presented below is described with respect to an MPEG-4 bit stream and decoder. However, the essential concepts described are also applicable to other video coding standards such as H.263, MPEG-2 and H.264, for example.
[0025] In some embodiments, the decoder is able to distinguish between bit errors and packet loss without any additional information about the channel. The video decoder relies on the fact that packet loss does not result in any syntax errors. Rather, only semantic errors would be encountered. The types of semantic errors encountered in an MPEG-4 bit stream due to packet loss are:
[0026] (1) The starting macro-block number of a video packet is inconsistent with the last decoded macro-block. However, the decoded macro-block number is valid, i.e., it is a legal value within the prescribed range. This happens when a video packet or group of video packets is lost.
[0027] (2) The Picture Start Code (VOP start code) for a new frame is not found. This can happen when the first video packet or a group of video packets at the beginning of a frame is lost.
[0028] The above semantic errors can also occur as a result of bit errors. Not all bit errors result in syntax errors, and may only cause semantic errors that are described above. Accordingly, bit error and packet loss errors may not be distinguished based on semantic errors alone.
[0029] FIG. 3 is a flow chart of an exemplary method for video decoding in accordance with some embodiments of the invention. In particular, FIG. 3 depicts a method for distinguishing between the packet loss and bit errors with a high probability. The method begins at start block 302 when there is an inconsistency in macro-block numbers (MBN 's) when decoding a new video packet. For example, if the new macro-block number (MBN) is not one greater than the previous macro-block number or if the MBN is outside of the expected range.
[0030] If the expected macro-block number is the first macro-block (number 0) of a frame, as depicted by the positive branch from decision block 304, a check is made at decision block 36 to determine if the block number is greater than one quarter of the total number N of macro-blocks in a video frame. The value MBN DIFF is set, at block 308, to be the difference between the current macro block number (MBN) and the expected block (i.e. the block number required for consistency). If the current block is not expected to be the first block of a sequence, as depicted by the negative branch from decision block 304, a check is made at decision block 310 to determine if the block number is greater than or equal to the expected block number. If the block number is within the expected range, as depicted by the positive branch from decision block 310, flow continues to block 308 where the value MBN DIFF. It expected that the MBN is greater than or equal to the expected MBN and less than the total number of blocks N in the frame. At decision block 312 a check is made to determine if the value MBN DIFF is greater than N/2 (half the number of macro-block in the frame). If it is, as depicted by the positive branch from decision block 312, then the error is judged to be a bit error and at block 314 the bit error is signaled and the process ends at block 316. The bit error may be signaled by setting a flag, for example. A bit error is also signaled if the block number is less than the expected block number or outside of the expected range, as depicted by the negative branch from decision block 310. If the value MBN_DIFF is not greater than N/2, as depicted by the negative branches from decision block 312, the error is judged to be a packet loss and is signaled at block 318 and the process terminates at block 316.
[0031] The threshold values (N/2 and N/4) can be set to be other fractions of the frame size, N.
[0032] An exemplary listing of a C-code routine to implement the method of FIG. 3 is given below. On entry to the routine, N is total number of macro-blocks in a frame, LAST MBN is the previously decoded macro-block number, EXPECTED MBN is the expected value of the next macro-block number (equals 0 for a new frame, and equals LAST MBN+1 otherwise), and MBN is the decoded macro-block number from the new video packet.
[0033] BIT ERROR = 0;
Tl = N / 2; T2 = N / 4; if ( (MBN >= EXPECTED MBN) ||
( (EXPECTED MBN == 0) && (MBN < T2) ) ) {
MBN DIFF = MBN - EXPECTED MBN; If ( MBN DIFF > Tl ) BIT ERROR = 1;
} else
BIT ERROR = I;
[0034] The values Tl and T2 are threshold values and can be set to be any fraction of the frame size, N. In this example Tl is set to one half of the frame size and T2 is set to one quarter of the frame size.
[0035] On exit from the routine, the BIT-ERROR flag is set to one if the error is judged to be a bit error and is set to zero if the error is judged to be a packet loss.
[0036] If a bit error is detected, the decoder immediately goes to the bit error state.
[0037] In the packet loss state, concealment is performed in a manner optimal for packet loss and error concealment is performed from the macro-block after the last error free macro-block to the macro-block numbered "MBN-I". The number of the last error free macro-block is denoted as LAST GOOD MBN. However, if a bit error is detected, concealment is performed from blocks "LAST MBN+1" to "MBN- 1". In prior systems, the former is always performed irrespective of whether a bit error or a packet loss occurs. The concealment performance gain is due to the fact that the number of the last error free macro-block often lower than LAST MBN, and hence more good macro-blocks are salvaged.
[0038] The embodiment described above has been implemented and integrated into an MPEG-4 Simple Profile decoder and tested by subjecting test video bit- streams with both bit errors and packet loss. The results were obtained using random error patterns and averaging over 1000 cases. Three versions of the decoder were created:
[0039] Decoderl : A version that is optimized for handling bit errors. [0040] Decoder2: A version that is optimized for handling packet errors.
[0041] Decoder3: A version corresponding to the embodiment described above. The performance was tested for various Quarter Common Intermediate Format (QCIF) encoded bit-streams. The bit-streams were encoded using various encoding parameters as given in TABLE 1 below.
[0042] TABLE 1.
Figure imgf000013_0001
[0043] The objective quality metrics (PSNR) is detailed in the following tables.
[0044] To measure performance for Packet Loss, a random packet loss was simulated with various packet loss rates. The results are shown in TABLE 2 below. The columns for each bit stream indicate the measured peak signal-to-noise ratio (PSNR) between images.
[0045] TABLE 2.
Figure imgf000013_0002
[0046] To measure performance for Bit Errors, bit error bursts were simulated with various error rates. The average error burst length was chosen to be 2 milliseconds. The results for QCIF bit-streams listed above are shown in TABLE 3 below.
[0047] TABLE 3.
Figure imgf000014_0001
[0048] From the results tabulated above, it is evident that the Decoder3, which is in accordance the present invention, performs very close to Decoder2 when there are errors due to lost packets and very close to Decoder 1 when there are bit errors. Thus, it is successfully able to adapt to the type of error and give optimal performance in both scenarios. The gain in terms of PSNR is fairly significant, especially at 1% and 2% packet loss.
[0049] Memory and computation requirements for implementation of the state selector are small compared to the requirement for the video decoder.
[0050] The error resilient video decoding system described above provides enhanced video quality in error prone environments. The system has application in mobile and handheld devices such as mobile phones and computer, and other client-server applications, such as video surveillance, that operate in error prone environments.
[0051] In the foregoing specification, specific embodiments of the present invention have been described. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present invention as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention. The benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential features or elements of any or all the claims. The invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of those claims as issued.

Claims

What is claimed is:
1. A method for decoding a video bit-stream in a video decoding system, the video bit- stream comprising a plurality of video frames each containing a plurality of numbered macro-blocks of video data and the method comprising: receiving the video bit- stream as a plurality of packets; operating the video decoding system in a packet loss state, in which the video bit- stream is decoded by a first video decoder configured to be resilient to loss of one or more packets of the plurality of packets; detecting a bit error; and operating the video decoding system in a bit error state, in which the video bit- stream is decoded by a second video decoder configured to be resilient to bit errors, if a bit error is detected.
2. A method in accordance with claim 1, further comprising operating the video decoding system in a packet loss state if no bit error is detected over a selected time period.
3. A method in accordance with claim 1, further comprising: detecting an inconsistency in the numbering of macro-blocks; and if an inconsistency in the numbering of macro-blocks is detected: determining if the inconsistency in the numbering of macro-blocks is due to a packet loss or a bit error; operating the video decoding system in a bit error state, in which the video bit-stream is decoded by a second video decoder configured to be resilient to bit errors, if the inconsistency is determined to be due to a bit error; and operating the video decoding system in a packet loss state if the inconsistency is determined to be due to a packet loss.
4. A method in accordance with claim 3, wherein detecting an inconsistency in the numbering of macro-blocks comprises comparing a macro-block number to a threshold value to determine if a start code for new frame is missing from the video bit stream.
5. A method in accordance with claim 3, wherein the inconsistency in the numbering of macro-blocks is determined to be due to a bit error if a current macro-block number is less than a previous macro-block number.
6. A method in accordance with claim 3, wherein the inconsistency in the numbering of macro-blocks is determined to be due to a bit error if a difference between a current macro-block number and an expected macro-block number exceeds a threshold value.
7. A method in accordance with claim 3, wherein determining if the inconsistency in the numbering of macro-blocks is due to a packet loss or a bit error comprises: if the macro-block number (MBN) of a current macro-block is greater than or equal to an expected MBN, or if the expected MBN is the first macro-block of a frame and the MBN is greater than a first fraction of the number of macro-blocks in a frame: setting a first value (MBN DIFF) as the difference between the macro- block number (MBN) of a current macro-block and a last error- free decoded macro-block; determining the inconsistency to be due to a bit error if the first value (MBN DIFF) is greater than a second fraction of the number of macro-blocks in a frame; determining the inconsistency to be due to a packet loss if the first value (MBN DIFF) is less than or equal to the second fraction of the number of macro- blocks in a frame; otherwise: determining the inconsistency to be due to a bit error.
8. A method in accordance with claim 3, wherein: operating the video decoding system in the bit error state comprises performing error concealment from the macro-block numbered LAST GOOD MBN+1 to the macro-block numbered MBN-I, where LAST GOOD MBN is the macro-block number of the last error free macro-block and MBN is the macro-block number of a current macro-block, and operating the video decoding system in the packet loss state comprises performing error concealment from block numbered LAST MBN+1 to block numbered MBN-I, where LAST MBN is the number of the last macro-block decoded.
9. A method in accordance with claim 8, wherein the last error-free macro-block is determined by a data recovery process.
10. A video decoding system for decoding a video bit-stream received as a plurality of data packets, the system comprising: an error detector for detecting an error in the video bit-stream; an error type detector operable to determine if the error detected by the error detector is due to a bit error or a packet loss; a bit error resilient video decoder, operable to decode the video bit-stream if the error is determined to be due to a bit error; and a packet-loss resilient video decoder, operable to decode the video bit-stream if the error is determined to be due to a packet loss.
11. A video decoding system in accordance with claim 10, wherein the packet loss resilient video decoder is further operable to decode the video bit- stream if no bit error has been detected within a specified time period.
12. A video decoding system in accordance with claim 10, wherein the video bit-stream comprises a plurality of video frames each containing a plurality of numbered macro- blocks of video data, and wherein the error detector is operable to detect an inconsistency in macro-block numbering.
13. A video decoding system in accordance with claim 10, wherein the video bit-stream comprises a plurality of video frames each containing a plurality of numbered macro- blocks of video data, and wherein the error type detector is operable to determine an error to be due to a bit error if a current macro-block number is less than a previous macro-block number of a frame.
14. A video decoding system in accordance with claim 10, wherein the video bit-stream comprises a plurality of video frames each containing a plurality of numbered macro- blocks of video data, and wherein the error type detector is operable to determine an error to be due to a bit error if a difference between a current macro-block number and an expected macro-block number exceeds a threshold value.
15. A video decoding system in accordance with claim 10, wherein the video bit-stream comprises a plurality of video frames each containing a plurality of numbered macro- blocks of video data, and wherein the error type detector is operable to determine an error to be due to a bit error if a first macro-block of frame is expected and a current macro-block number exceeds a threshold value.
16. A video decoding system in accordance with claim 10, wherein the video bit-stream comprises a plurality of video frames each containing a plurality of numbered macro- blocks of video data, and wherein the error type detector is operable to determine an error to be due to a bit error if: a current macro-block number is outside of an expected range, or if a difference between a current macro-block number and an expected macro- block number exceeds a threshold value, or if a first macro-block of a frame is expected and a current macro-block number exceeds a threshold value, and is operable to determine an error to be due to a packet loss error otherwise.
17. A video decoding system in accordance with claim 10, wherein the video bit-stream comprises a plurality of video frames each containing a plurality of numbered macro- blocks of video data and wherein the bit error resilient video decoder is operable to perform error concealment from the macro-block numbered LAST GOOD MBN+1 to the macro-block numbered MBN-I, where LAST GOOD MBN is the macro-block number of the last error free macro-block and MBN is the macro-block number of a current macro-block, and the packet loss resilient video decoder is operable to perform error concealment from block numbered LAST MBN+1 to block numbered MBN-I, where LAST MBN is the number of the last macro-block decoded.
18. A method for detecting errors in a video bit- stream, the video bit- stream comprising a plurality of video frames each containing a plurality of numbered macro-blocks of video data and the method comprising: receiving the video bit- stream as a plurality of packets; and detecting an inconsistency in the numbering of macro-blocks; signaling a bit error if a current macro-block number is less than a previous macro-block number; and signaling a bit error if a difference between a current macro-block number and an expected macro-block number exceeds a threshold value.
19. A method in accordance with claim 18, further comprising: if the macro-block number (MBN) of a current macro-block is greater than or equal to an expected MBN, or if the expected MBN is the first macro-block of a frame and the MBN is greater than a first fraction of the number of macro-blocks in a frame: setting a first value (MBN DIFF) as the difference between the macro- block number (MBN) of a current macro-block and a last error- free decoded macro-block; determining the inconsistency to be due to a bit error if the first value (MBN DIFF) is greater than a second fraction of the number of macro-blocks in a frame; determining the inconsistency to be due to a packet loss if the first value (MBN DIFF) is less than or equal to the second fraction of the number of macro- blocks in a frame; otherwise: determining the inconsistency to be due to a bit error.
20. A method in accordance with claim 19, further comprising: decoding the video-bit stream using a bit-error resilient video decoder if the inconsistency is determined to be due to a bit-error; and decoding the video-bit stream using a packet-loss resilient video decoder if the inconsistency is determined to be due to a packet loss.
PCT/US2007/082594 2006-12-19 2007-10-26 Method and apparatus for adaptive error resilience for video decoders WO2008079503A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IN2717DE2006 2006-12-19
IN2717/DEL/2006 2006-12-19

Publications (2)

Publication Number Publication Date
WO2008079503A2 true WO2008079503A2 (en) 2008-07-03
WO2008079503A3 WO2008079503A3 (en) 2008-08-21

Family

ID=39563145

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2007/082594 WO2008079503A2 (en) 2006-12-19 2007-10-26 Method and apparatus for adaptive error resilience for video decoders

Country Status (1)

Country Link
WO (1) WO2008079503A2 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050254584A1 (en) * 2001-03-05 2005-11-17 Chang-Su Kim Systems and methods for enhanced error concealment in a video decoder

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050254584A1 (en) * 2001-03-05 2005-11-17 Chang-Su Kim Systems and methods for enhanced error concealment in a video decoder

Also Published As

Publication number Publication date
WO2008079503A3 (en) 2008-08-21

Similar Documents

Publication Publication Date Title
EP1303917B1 (en) Unequal error protection for packets with variable length
KR100416866B1 (en) Receiver device and signal processing method
KR100493770B1 (en) Picture coder, picture decoder, and picture transmission method
EP2268042B1 (en) A decoder architecture for optimized error management in streaming multimedia
US8379716B2 (en) Method, apparatus, and system for enhancing robustness of predictive video codecs using a side-channel based on distributed source coding techniques
EP1842380B1 (en) Video error detection technique using a crc parity code
US8743970B2 (en) Video decoding with error detection and concealment
WO2003007536A1 (en) Method for detection and recovery of errors in the frame
JP2008546230A (en) Apparatus and method for error recovery using in-band error patterns
Khan et al. Iterative error detection and correction of H. 263 coded video for wireless networks
US20050195903A1 (en) Method and apparatus to check for wrongly decoded macroblocks in streaming multimedia applications
US6983016B2 (en) Method for detecting errors in video information
US5912907A (en) Satellite receiver code rate switching apparatus
US20040145677A1 (en) System, method, and apparatus for error recovery in coded video signals
KR20020064778A (en) Coding of a data stream
US20140119445A1 (en) Method of concealing picture header errors in digital video decoding
US8379729B2 (en) Content adaptive error localization to improve quality of corrupted video data streams upon decode
US11601216B1 (en) Alignment detection by full and partial FEC decoding
WO2008079503A2 (en) Method and apparatus for adaptive error resilience for video decoders
RU2374787C2 (en) Structure of decoder for optimised control of error processing in multimedia data streaming
EP1555788A1 (en) Method for improving the quality of an encoded video bit stream transmitted over a wireless link, and corresponding receiver
KR100761542B1 (en) Error concealment device using motion vector of adjacent macro block and method thereof
CN101176353B (en) Decoder architecture for optimized error management in streaming multimedia
JP2002077922A (en) Picture control system and picture decoder
Hannuksela H. 263 picture header recovery in H. 324 videophone

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 07871252

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase in:

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 07871252

Country of ref document: EP

Kind code of ref document: A2