RU2374787C2 - Structure of decoder for optimised control of error processing in multimedia data streaming - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: method and device are suggested for multilevel integration used for elimination of errors. Error is detected in multimedia data on the basis of the first level protocol, and then error detected in multimedia data is masked on the basis of the second level protocol. In one aspect error in multimedia data is eliminated on the basis of communication level protocol, and it is controlled on the basis of transport level protocol. Further distribution of controlled error is determined on the basis of synchronisation level protocol, then error detected in multimedia data is masked on the basis of applied level protocol. Further stage of error elimination and scaling stage are provided.

EFFECT: increased efficiency of multimedia data stream processing by reception of multiple streams of coded multimedia data, eliminating errors in erroneous part of stream and recovering multimedia data from multiple streams.

40 cl, 10 dwg

Description

Description

This patent application claims priority in accordance with provisional patent application No. 60 / 660.681, entitled "Method and apparatus for error recovery in video communications", filed March 10, 2005, provisional patent application No. 60 / 660.923, called "Method and apparatus for video decoding ”, filed March 10, 2005, and provisional patent application No. 60 / 660.867, entitled“ Method of error recovery for a decoder ”, filed March 10, 2005, each of which belongs to the applicant of this application and is included in the present description by reference.

FIELD OF THE INVENTION

The present invention relates to methods and apparatus for decoding streaming multimedia data in real time on portable devices.

State of the art

Due to the high growth and great success of the Internet and wireless communications, as well as the increased demand for multimedia services, the streaming of multimedia data over the Internet and mobile / wireless channels is attracting great attention. In a heterogeneous IP network, video is provided by the server, but can also be provided by one or more clients. Wired connections include telephone line, digital network with integrated services (ISDN), cable lines, digital subscriber line protocols (collectively referred to as xDSL), fiber optic cables, local area networks (LANs), and global networks. networks (WAN), etc. The transmission mode can be either unicast or multicast (multicast).

Mobile / wireless is similar to heterogeneous IP networking. Transporting multimedia content (content) via mobile / wireless channels is rather difficult due to the fact that the quality of these channels is often significantly deteriorated due to fading due to multipath propagation, shadowing, intersymbol interference and noise interference (interfering noise). Some other reasons, such as mobility and competing traffic, also lead to bandwidth changes and losses. Channel noise and the number of users served determine the time-varying property of the channel media.

The demand for higher data rates and higher quality of service in both heterogeneous IP networks and mobile communication systems is growing rapidly. However, factors such as limited delay time (delay), limited transmit power, limited bandwidth, and multipath fading continue to limit the data rate of a practical system. When transmitting multimedia data, in particular in error prone environments, the error tolerance of transmitted multimedia data is critical in ensuring the desired quality of service, due to the fact that errors, even in a separate decoded value, can lead to decoding of artifacts (image distortion) propagating spatially and temporally. To minimize errors, various encoding parameters are used that support the necessary data rate, however, all these methods suffer from problems associated with errors reaching the decoder side.

Data is compressed using a source encoder that transmits the maximum amount of information with a minimum bit rate, following the channel encoder, which tends to maximize channel throughput for a given error probability when receiving these bits.

Channel coding, such as Reed-Solomon coding, is used to increase the reliability of data encoded by the source encoder. Joint channel coding techniques are used to provide varying levels of error protection to data encoded by a source encoder, with varying levels of importance, or to enable adaptation of the encoded video data rate to the available network bandwidth by dividing and rejecting packets. This is due to the fact that conventional transport protocols do not deliver corrupted data to the source decoder.

Source coding methods, such as reverse coding with variable-length codes (for example, in the MPEG-4 standard), were used to eliminate errors by decoding a packet in the reverse order when actually receiving damaged packets. There is a trade-off in coding efficiency using source coding methods that preserves the quality of decoded video data at a given data rate.

Hybrid coding standards such as MPEG-1, MPEG-2, MPEG-4 (collectively named MPEG-x), H.261, H.262, H.263 and H.264 (collectively named H.26x) use dots resynchronization in the bitstream as the main way to handle errors in the decoder.

Another reason that can cause data loss in excess of the initial distortion is due to emulation of an invalid keyword. Identification of the initial position of the bit error is not a trivial task and, as a rule, is impossible without a special structure that supports the identification of the positions of bit errors at the MAC level or component of the physical layer. Therefore, after detecting damage to the bitstream, the decoder will probably have to stop decoding and advance through the bitstream to search for the next resynchronization point, during which a large amount of potentially useful data is inevitably skipped. Despite emulating another keyword that has a length similar to the original, that is, true, the keyword may seem to be a lesser problem compared to the sequence of events described above, in fact it is not an example. There are many ways in which this type of error can lead to errors in decoding the correct decoder bitstream. For example, in the most modern codecs (encoder-decoders) in the bitstream there are objects (parameters related to compression), the values of which affect the order of the next part of the bitstream. Therefore, an incorrect value of such an object can lead to decryption of the incorrect bit stream.

Due to the fact that conventional transport protocols do not deliver damaged data to the decoder (for example, the use of a video or audio decoder), the decoder has a limited ability to process bit errors using packet rejection and re-synchronization, which is the most common solution. An improved way to handle bit errors that lead to the propagation of errors, as well as data loss due to problems such as loss of synchronization and emulation of the wrong keyword, is needed.

SUMMARY OF THE INVENTION

In one aspect, a method and apparatus for multilevel integration used to eliminate errors, comprise a method or means for detecting an error in multimedia data based on a first level protocol and masking an error in multimedia data based on a second level protocol. In another aspect, the device for multi-level integration used to troubleshoot comprises a detection means for detecting errors in multimedia data based on a first level protocol and masking means for masking an error in multimedia data based on a second level protocol. In a method and apparatus for multi-level integration, the first level may include a communication level. The communication layer may include one element from a set containing a physical layer, a MAC layer and a transport layer, or a combination of these elements. In addition, the method and device may further comprise a method or means for monitoring a detected error based on a transport layer protocol. The step of monitoring the detected error may include the step of limiting the distribution of the detected error. The method and device may also further include a method or means for determining the distribution of the detected error based on the synchronization level protocol. The second layer may include an application layer.

In another aspect, a method and apparatus for multilevel integration used to eliminate errors includes a method or means for detecting errors in multimedia data based on a communication layer protocol, monitoring a detected error based on a transport layer protocol, determining a distribution of a controlled error based on a level protocol synchronization and masking of errors detected in multimedia data based on the application layer protocol. In another aspect, the method and apparatus used for processing multimedia data includes a method or means for performing error correction of encoded multimedia data and supporting scalability of encoded multimedia data. In yet another aspect, a device used for processing multimedia data includes an error correction component for performing error correction on encoded multimedia data and a scaling component for supporting scalability of encoded multimedia data. In the method and device used in processing multimedia data, scalability can be either spatial or temporal, or both. The error correction step may comprise either a temporary error concealment step, or a spatial error concealment step, or a frame rate conversion step, or a combination of these steps.

In another aspect, the method and apparatus used to process a multimedia stream includes a method or means for receiving multiple streams of encoded multimedia data, performing error correction on an erroneous part of the stream, and recovering multimedia data from the multiple streams. In another aspect, a device used to process a multimedia stream includes a receiver for receiving multiple streams of encoded multimedia data, an error correction component for performing error correction on an erroneous part of the stream, and a recovery unit for recovering multimedia data from the multiple streams. In the method and apparatus used to process the multimedia stream, the error correction step may include either a temporary error concealment step, a spatial error concealment step, or a frame frequency conversion step, or a combination of these steps.

It should be noted that the aforementioned method and device can be implemented using a computer-readable medium and / or processor configured to perform a method or operation of the device.

Brief Description of the Drawings

1A is a block diagram of an example communication system for delivering streaming multimedia data.

FIG. 1B is a block diagram of an example multilevel communication system for delivering streaming multimedia data.

1C is a block diagram of another example of a multilevel communication system for delivering streaming multimedia data.

Fig. 2A depicts a block diagram of an example structure of a decoding device for decoding streaming media.

FIG. 2B is a protocol stack diagram of an integrated multi-level control system including a transmitter, and another representation of the structure of a decoding apparatus shown in FIG. 2A.

Figure 3 depicts an example of multimedia symbols ordered for cascading coding with erasing Reed-Solomon errors and turbo coding.

4 is a flowchart of an example method for decoding streaming multimedia data.

5 depicts the structure of an information turbopackage for video data.

FIG. 6 is a block diagram of an illustrative component system that may be part of the multimedia receiver 24 of FIG. 1.

7 depicts a flowchart of an error correction process.

Detailed description

A method and apparatus for providing advanced error correction capabilities in a multimedia decoder are described. Integrated error correction capabilities are provided, such as detecting errors in the multimedia data stream at the upper level (for example, at the communication level) and performing error correction with respect to the detected errors at the application level (for example, video or audio decoder). In one example, a decoder structure is provided that provides information marking damaged bits for application layer components used to create informed decisions when performing various types of error correction methods. Error correction methods are used to replace damaged characters with prospective characters obtained from information available for the application layer component, such as pre-decoded video data, audio data, text and graphic information. In order to provide a thorough understanding of the embodiments, characteristic elements are provided in the following description. However, it will be understood by those skilled in the art that embodiments may be practiced without these characteristic elements. For example, electrical components may be shown in block diagrams so as not to obscure embodiments with unnecessary elements. In other cases, similar components, other structures and methods can be depicted in detail to further clarify embodiments. It is also understood by those skilled in the art that the electrical components depicted as separate units can be rearranged and / or combined into a single component.

It should also be noted that some embodiments may be described as a process depicted as a flowchart, a flowchart, a block diagram, or a circuit diagram. Although the flowchart may describe operations as a multi-stage process, many of the operations can be performed in parallel or simultaneously, and the process can be repeated. In addition, the order of operations can be rearranged. The process ends if its operations are completed. A process may correspond to a method, function, procedure, subprogram, part of a program, etc. If the process corresponds to a function, then its termination corresponds to the return of the function to the calling function or to the main function.

1A is a block diagram of an example communication system for delivering streaming multimedia data. System 20 includes a transmitter 22 and a multimedia receiver decoder 24. The transmitter 22 comprises compressed multimedia data of various forms, including but not limited to video data, audio data, graphic information, text information and images. The data may be compressed video and audio data, as in the MPEG-x and H.26x standards, compressed audio data, as in the speech or video compression standards MPEG-4 AAC, MP3, AMR and G.723, or any other type of digital data.

The transmitter 22 receives data from various sources, including external memory, the Internet and the operational supply of audio and / or video data. The transmitter 22 also transmits (Tx) the received data over the network. The network may be a wired network 28, such as a telephone, cable or fiber optic, or wireless network 26. With respect to a wireless communication system, network 26 may comprise, for example, part of a code division multiple access (CDMA or CDMA2000) communication system or alternately a system may be a frequency division multiple access (FDMA) system, orthogonal frequency division multiple access (OFDMA) system, time division multiple access (TDMA) system, as a technology mobile communications for the service industry GSM / GPRS (general packet radio system) / EDGE (advanced GSM for data transmission) or TETRA (terrestrial radio trunk communication), code division multiple access (WCDMA), high-speed data transmission system (1xEV- DO or 1xEV-DO Gold Multicast) or any wireless communication system using a combination of methods.

Decoder 24 comprises means, such as a radio frequency antenna or network connection, for receiving data over a wireless network 26 and / or a wired network 28. Decoder 24 may include multiple processors containing a different combination of pre-processors (for example, any type of central processing unit (CPU) ), for example, ARM), a digital signal processor (DSP), software, firmware, and hardware such as the VideoCore multimedia processor for distributing demodulation and decoding h related to received data. Decoder 24 also contains memory components for storing received data and intermediate data at various stages of the demodulation / decoding process. In some embodiments, the ARM preprocessor performs less complex tasks, including decompressing (removing side information such as headers and messages) and demultiplexing a plurality of bit streams including audio, video, and more. The ARM pre-processor also performs bitstream parsing, error detection, masking and decoding of variable length entropy. In some such embodiments, the digital signal processor (DSP) performs VLC keyword expansion (with variable length code), reverse zigzag scan of video data to spatially determine pixel parameters, reverse AC / DC pixel parameter prediction for MPEG-4 video data (not is a feature of the H.264 standard due to context-sensitive adaptive encoding of entropy) and decoding of audio data (for example, MPEG-4 AAC, MP3, AMR or G.723). The VideoCore multimedia processor can perform more computationally complex video decoding tasks, including dequantization, inverse transform, motion compensation prediction and unlocking (a form of filtering to reduce edge distortion of an image (artifacts) between the edges of a pixel block). In communication system 20, one or more elements may be added, rearranged, or combined. With respect to a wired communication system, the network 26 may include, for example, a portion of a communication system based on the Internet Protocol (IP), with transport protocols such as Real-time Transport Protocol (RTP) or Universal Datagram Transmission Protocol (UDP).

FIG. 1B depicts a block diagram of a multi-level protocol stack used for division tasks performed by transmitter 22 and decoder 24. Top-level components 205 and 210 located in transmitter 22 and decoder 24, respectively, may include a variety of applications, such as, for example, video or audio encoders and / or decoders. Some embodiments may include multiple streams of information that are intended for simultaneous decoding. In these cases, synchronization tasks of multiple threads can also be performed on top-level components 205 and 210. Top-level component 205 can provide encoded synchronization information in a bitstream transmitted over wireless network 26 and / or wired network 28. Top-level component 210 can parse multiple information streams so that related applications decode them at approximately the same time .

The lower level components 215 located in the transmitter 22 may include various circuits to provide error tolerance. Error-prone channels, such as wireless network 26 and / or wired network 28, can introduce errors into the bitstream received by decoder 24. Such error tolerance schemes provided in lower level components 215 may include one or more error-correcting coding schemes ( with error control), alternating circuits, and other circuits known to those skilled in the art. The lower level components 220 located in the decoder 22 may include corresponding error decoding components that allow for error detection and correction. Some errors introduced through wireless network 26 and / or wired network 28 may not be correctable using lower level components 220. For uncorrectable errors, decisions, such as lower-level components 220, requesting the retransmission of damaged components by lower-level components 215 of transmitter 22, may not be feasible in some situations, for example, when transmitting multimedia data in real time, for example, when streaming applications. In some embodiments, lower layer components 215 and 220 comprise communication layer components. One or more elements may be added, rearranged, or combined at transmitter 22 or decoder 24 shown in FIG. 1B.

FIG. 1C is a block diagram of a more detailed example of a multi-level protocol stack used for dividing tasks in transmitter 22 and decoder 24. Top-level components 205 located in transmitter 22 are allocated to either one level from a plurality containing application layer 206 and level 207 synchronization, or in several levels. The lower level components 215 located in the transmitter 22 are allocated either to one level from a plurality containing a transport layer 216, a medium access control (MAC) layer / stream 217 / a stream layer 217, and a physical layer 218, or several levels. Similarly, the top-level components 210 located in the decoder 24 are distributed either to one level from a plurality containing the application layer 211 and the synchronization level 207, or to several levels. The lower level components 220 located in the decoder 24 are allocated either to one level from a plurality containing the transport layer 221, the medium access control (MAC) layer 222 / stream layer 222 and the physical layer 223, or into several levels. Those skilled in the art are familiar with these levels, and they are also familiar with the distribution of various tasks among them. The following discusses an example of a structure combining the various layers of the decoding device 24, as discussed above, to take advantage of the error tolerance provided by the transmitter 22. One or more elements can be added, rearranged, or combined in the transmitter 22 or decoder 24 shown in Figs.

Fig. 2A depicts a block diagram of an example structure of a decoding device for decoding streaming media. FIG. 2B is a diagram of a protocol stack of an example integrated multi-level control system comprising a transmitter 22 and a representation of a protocol stack of the structure of a decoding apparatus shown in FIG. 2A. As shown in FIGS. 2A and 2B, the multimedia decoder 30 comprises a physical layer component 32, a MAC layer component 34, a transport layer (TSP) and synchronization layer parsing unit 39, and an application layer component 50. The multimedia decoder 30 receives an input bitstream (B) comprising a cascade error correction circuit, for example, a cascade coding scheme for turbo / Reed-Solomon. The component 32 of the physical layer can perform demodulation tasks, including but not limited to, including receiving, noise-free decoding, for example, decoding using a turbo code, and interacting with the MAC layer (media access control layer). The MAC layer component 34 may perform error-correcting decoding, for example, detection of Reed-Solomon errors, error correction, and marking of uncorrectable damaged data, for example, a group of one or more bits. The MAC layer component 34 interacts with the transport layer and synchronization layer (TSP) component 39.

The TSP component 39 may further comprise a transport layer demultiplexing component 36 and a synchronization level parsing component 38. The transport layer demultiplexing component 36 may receive a bitstream transmitted from the MAC layer component 34, containing both correct and damaged bits, and information marking the damaged groups of bits. The damaged bit groups and the corresponding marking information contain information corresponding to errors 33 in the turbo decoder checksum (CRC) and Reed-Solomon errors 35. (In some protocol stacks, the transport layer demultiplexing component 36 is also known as the stream layer sublayer, where the MAC layer sublayer and the stream layer sublayer are transport layer sublayer.) The transport layer demultiplexing component 36 can demultiplex (de-mux) or parse the received bitstream into many bit streams. The parsed bitstreams may comprise bitstreams for various applications, such as a video decoder, audio decoder, and various combinations of text, graphics, and display applications. The transport layer demultiplexing component can also parse a single bitstream intended for a separate application, such as a video bitstream, for example, into two or more separate layers (for example, using scalable encoding), for example, to a main layer and an enhancement layer. These levels can then be used to provide scalability, such as temporal and / or SNR scalability. One example of scalable coding divides internally encoded frames (such as I-frames) and other inter-coded frames (e.g., P-frames or B-frames obtained using, for example, motion-compensated prediction) into different levels in the bitstream . I-frames can be encoded at the main level, and P-frames and / or B-frames can be encoded at the enhancement level. Scalable coding is useful in dynamic channels where scalable bitstreams can be adapted to match fluctuations in network bandwidth. In error prone channels, scalable coding can increase reliability by protecting against unequal errors at the core and enhancement levels. Better error protection can be applied to a more important level.

The synchronization level parsing component 38 further parses the bitstream into bit substreams associated with each other based on time synchronization. The multimedia video bitstream can be parsed into a video bitstream, an audio bitstream, and a bitstream with corresponding closed caption text information. The synchronization level parsing component 38 transmits the parsed bitstreams to the associated decoder application along with time synchronization information. This provides the associated audio, video and text information with the ability to display and play at the appropriate time.

In addition to the parsing discussed above, the transport layer demultiplexing component 36 can parse and forward information indicating damage (e.g., CRC error information 33 and Reed-Solomon error information 35) that it received from the MAC layer component 34 and the physical component 32 level, component 38 parsing the synchronization level and / or the corresponding process of the application level (for example, a video decoder or audio decoder). The synchronization level parsing component 38 may transmit information marking the damage in the form of error distribution information 37. The synchronization level parsing component 38 may also transmit recommended information 41 on the error control strategy. Then, the application layer process can use information 37, marking damage, and information 41 about the control strategy for error handling.

The application layer component 50 may comprise one or more components, such as, for example, error correction component 40, scaling component 42, frame rate conversion (FRUC) component 44, base application decoding component 46, and post-processing component 48. The application layer component 50 uses the transmitted damage marking information 37 and the control strategy information 41 to make a decision regarding how to use the error correction, scaling, and frame rate conversion (FRUC) components to manage the damaged data, thereby offering higher quality decoding with using the decoding component 46 of the base application. For example, with regard to temporal scalability, where some multimedia data is received at one level containing important information, and the rest of the multimedia data at another level, the frame rate conversion component (FRUC) can be used to recover missing multimedia data in case the second level was not accepted, lost or damaged. After decoding, the subsequent processing component 48 performs any necessary specific hardware modifications to enable the display, playback, or rendering of the video and audio output on the display device or speakers, respectively. The post-processing component 48 may also perform enhancement or restoration operations before the multimedia data is reproduced or presented.

The decoding component 46 of the base application may comprise video decoder (s), audio decoder (s), as well as text and graphics applications. By performing error correction, scaling, and frame rate conversion (FRUC) processing of bit streams of various applications before or during decoding using the decoding component 46 of the base application, improvements can be made to improve the quality of the low quality bitstream (encoded with low quality or received with poor quality due to errors). For example, components 40, 42, and 44 can offer improvements to the standard source H.264-compliant video bitstream (the base profile is a very simple profile that was designed for a low-power device), and provide some elements of other profiles of the H.264 standard such as B-frames and separation of layered data, which are required to perform scaling, detection and error tolerance for the video stream. Elements of processes using the components of multimedia decoder 30 are presented below. One or more elements may be added, rearranged, or combined at transmitter 22 or decoder 30 shown in FIGS. 2A and 2B.

The following is a brief discussion of the error detection and error correction process. One example of an error detection and correction scheme uses a cascade code containing both an internal (channel) code and an external (channel) code. Cascading channel codes consist of a turbo (internal) code at the physical level and a correcting code of (external) error erasing of Reed-Solomon installed at the MAC level. Figure 3 depicts an example of multimedia symbols ordered for cascading coding with erasing Reed-Solomon errors and turbo coding. As for the coding side, the output of characters from an information source, the output of double keywords from an encoder is glued into bytes 102. Each byte 102 is considered a character in a final field, known as a “Galois field (256),” for external (N, K) purposes Reed-Solomon code (RS) for the Galois field (256) (GF). N and K respectively denote the sizes of the entire Reed-Solomon (RS) keyword 104 and its source data 106, containing a systematic part in the number of characters. Thus, N minus K represents the number of parity symbols 108 included in each keyword 104. The Reed-Solomon code (RS) (N, K) can be corrected for N minus Erase errors.

The top rows 106 essentially contain characters derived from an information source, and these characters can be scanned from K rows or initially scanned rows or columns. Alternation is achieved by scanning the initial columns and leads to significantly smaller groups of damaged bits in the event that this line 112 information turbopackage is damaged. The length of each group of damaged bits due to erasing errors of the information turbo packet can be equal to 1 byte for the initially scanned columns, in contrast to the length equal to (L-1) bytes for the initially scanned row. At the decoder, marking of these damaged bit groups, as discussed below, may be required to identify the size and position (in the bit stream) of these bit groups. After this initial stage of source data placement, each of the L columns 104 (K bytes) is encoded using RS in N bytes by adding NK parity bytes and, therefore, K + 1, ..., N lines 108 shown in FIG. 3 are formed. To the top lines consist of source data 106 referred to as an RS information block, and a whole set of N lines is referred to as RS encoded block, or simply encoded block 110.

A CRC and some trailing bits are required for each turbo encoder to work correctly on each line 112. Due to the addition of a checksum to each line 112, these lines will not be able to satisfy their respective checksums after the turbo decoding can be declared erased. Each code block 110 enters one line 112 into a turbo encoder at a time and, therefore, each line is referred to as an information turbo packet.

The turbo decoding process provides clear data to the Reed-Solomon decoding process, which then reduces the residual error rate. The ability to successfully correct for erasure depends on the total number of erasures within the code block and the number (N minus K) of parity symbols used in the RS keyword.

In the channel coding structure for the multimedia decoder 30 shown in FIG. 2, if the Reed-Solomon (RS) code block is erased outside of the correcting ability, the corresponding RS information block (FIG. 3) can be transmitted to the components at the transport level, synchronization level, or at the application level with a notification marking which of the K information turbopackages 112 (Figure 3) are damaged. The systematic structure of the external (N, K) Reed-Solomon (RS) code over the Galois field (256) allows the direct use of correctly accepted information turbopackages (not damaged).

4 is a flowchart of an example method for decoding streaming multimedia data. Modulated error data (IN) is received and input to the decoding process 60 and demodulated (step 62). Data may be received over wired or wireless networks, for example, over wireless network 26 and wired network 28 shown in FIG. At step 62, the physical layer component 32 depicted in FIG. 2 demodulates received erroneous data. Then, the demodulated data is transmitted to step 64, where errors can be detected and corrected. In step 64, the physical layer component 32 shown in FIG. 2 can perform turbo decoding, while the MAC layer component 34 shown in FIG. 2 can perform Reed-Solomon error correction.

After at 64, the Reed-Solomon turbo decoding and decoding detected and corrected all correctable errors, information turbo packets and / or damaged bytes are identified at 66, for example, by marking. Before transmitting the bitstream for parsing at step 68, the CRCs, trailing bits in each turbo information packet, and parity lines 108 (FIG. 3) are rejected. Faultlessly received, received with errors, but corrected and marked corrupted data along with marking information identifying the corrupted data are jointly transmitted to step 68 to parse the bitstream. At step 66, the physical layer component 32 and / or the MAC layer component 34 (FIG. 2) can identify (eg, by marking) the damaged data.

As discussed above, the transport layer demultiplexing component 36 and the synchronization layer parsing component 38 parse the bitstream into a plurality of bit streams intended for a plurality of application layer processes. In the example shown in FIG. 4, the bitstream is parsed (at 68) into a video bitstream 70, an audio bitstream 72 and a text and / or graphic information bitstream 74. Each bit stream 70, 72 and 74 may contain data identifying damaged groups of bits in individual bit streams. In addition, if the individual bit streams must be synchronous in time, then the bit streams may contain synchronization information.

After parsing the individual bit streams, error processing is performed at the application level, where the damaged bits are replaced in steps 78, 82 and 86 using one of several error correction or masking methods. The error correction component 40, the scaling component 42, and the frame rate conversion (FRUC) component 44 of the application layer component 50 (FIG. 2) can replace at steps 78, 82, and 86. After replacing the damaged characters, separate streams of video, audio, and text / graphics information can be decoded at steps 80, 84 and 88, respectively. The decoding component 46 of the base application of the application layer component 50 (FIG. 2) can perform decoding in steps 80, 84 and 88. The decoding component 46 of the base application of the application layer component 50 (FIG. 2) can also replace damaged bits in steps 78, 82 and / or 86, which may be labeled with a lower level component. One or more elements may be added, rearranged, or combined for the processing process 60 of FIG. 4.

Next, an example of identifying corrupt bits (step 66) by marking corrupt data as discussed in FIG. 4 is discussed in detail. 5 depicts the structure of an information turbopackage for video data. Block 140 represents a set of information turbopackages, for example, K lines of information turbopackages 106 constituting the RS information block shown in FIG. A video data frame may require coding of several data turbo information packets. For example, the first frame (F1) originates in line 142A of information block 148A. The data remaining in frame F1 is located in lines 142B, 142C, 142-D, as well as in the first part of line 142E. Line 142A also contains a synchronization level (SH) header 146A, which contains information such as flow identification, synchronization time, frame identification (number of frames, number of available levels, if a base level and enhancement level are present) and other information. The synchronization level header 146A is used to identify the synchronization level parsing unit (reference number 38 in FIG. 2), which (whose application) sends data contained in the following information blocks representing the F1 frame. At the end of each line is a CRC, as discussed above, which is used in conjunction with Reed-Solomon decoding to identify erased (damaged) packets.

At the beginning of each line of the information turbopackage, a transport layer (TH) header 144 is present. Each transport layer (TH) header 144 contains the Last_Flag and Offset_Pointer fields. The marking of the damaged information turbopackage performed in step 66 shown in FIG. 4 can be performed directly using the value of the “Offset_Pointer” field. Setting the wrong value in the “Offset_Pointer” field may indicate a packet as damaged. Alternatively, the Error_Flag field may be used in the transport layer (TH) header. Marking can be achieved by setting the value of the "Error_Flag" field to one (Error_Flag = 1) in the corresponding transport layer header (TH). For example, if line 142C is damaged, then the transport layer (TH) header value of line 142C can be set to one. The Last_Flag field is used to indicate that the current line is the last line of the frame (for example, by setting its value to one), and if it is the last line, the Offset_Pointer field is used to indicate where in the line the information turbopackage is the beginning of the next frame (in bytes). For example, on line 142E, the value of the transport layer (TH) header 144B may be one (the value of the Last_Flag field is one), and the value of the Offset_Pointer field may be the number of bytes contained in the line before the transport layer header 144C (TH ) (beginning of frame F3). The header 146B synchronization level may contain data indicating the field "Frame_ID" and the bit stream, as discussed above. Information block 148C contains video data representing frame F3. If line 142E was determined to be damaged, then the decoder may not determine where frame F1 ends and where frame F3 begins.

In addition to (or instead of) including the “Error_Flag” field in the transport layer header, as discussed above, a data table can be created and transmitted to the application layer with the information listed in Table 1 for each video frame.

Selected information contained in the video bitstream error table

Table 1 Selected information contained in the video bitstream error table - Number of frames - an integer limited by the set of frames, for example, 30 frames, where after the largest number the numbering resumes from one.
- B / E - indicates the number of base levels or frame enhancement levels in the presence of multiple zoom levels.
- Frame_Length - the length of the video frame in bytes.
- RAP_Flag - an indicator indicating whether the frame is a random access point, for example, a whole I-frame with internal encoding. Such an indicator can also serve as a re-synchronization point in the event of an error.
- Display time stamp (PTS) - time in the sequence of frames during which the frame should be displayed.
- Frames per second (FPS)
The number of information turbopackages occupied by the personnel (both fully and partially)
- Error_Bit_Pattern - a variable indicating which information turbopackages are damaged (five bits “00100” will indicate that the third of the five information turbopackages is damaged).
- Error_Ratio - a variable indicating the coefficient of damaged information turbopackages for packet correction (1/5 will indicate that one of the five packets is damaged).

Other bitstream error tables, such as Table 1, may also be generated. Error marking information contained in transport layer headers and / or error tables, such as Table 1, may be used by application layer components such as components 40, 42 and 44 (FIG. .2), to identify damaged characters in the parsed bit streams 70, 72 and 74, as well as to replace them at steps 78, 82 and 86 (Figure 4), respectively. As discussed above, erase packets from Reed-Solomon (RS) coding are not discarded, they are transmitted to the application layer, for example, to a video decoder. This protects entire packets, potentially 122 bytes or more in length for video packets, from loss.

Replacing damaged characters in steps 78, 82, and 86 (FIG. 4) can take two main forms, namely, error correction and error concealment. Error correction and concealment operations at the bitstream level are performed on erasures and packet errors. Decoding errors (due to errors in bytes) are corrected by using the MAP criterion (maximum posterior probability) to the extent possible. Errors that cannot be corrected are masked by using spatial and / or temporal information from neighboring macroblocks (a macroblock is a 16 × 16 pixel area, which, as a rule, operates with video compression standards, and smaller ones can also be used the size of sub-macroblocks, for example 8 × 8, 8 × 16 and others). Temporary masking can be used, for example, if the image is static over one frame. If the damaged masked macroblock is in an area that has remained relatively constant in previous frames, then it will probably be present in the damaged frame, and the areas of the past frame can be used as an estimate for the damaged area. Spatial masking can take advantage of boundaries or objects that exist in neighboring macroblocks of a single frame.

As discussed above, multiple layers of encoded data representing one bit stream can be received. Levels may include a base level and one or more enhancement levels, where the enhancement level (s) may provide additional frames that are not available at the basic level (for example, predicted bidirectional B-frames), or it may provide higher quality differential enhancements for basic level pixel parameters. As for the differential enhancements for the parameters of the basic level, if the basic level is damaged, then the level of improvement can be useless, since it was obtained based on the pixel values of the basic level. In this way, the enhancement layer can be freed from decoding if the base layer data is corrupted. This process is called selective decoding, and it can also be used under low power scenarios. For example, if the device containing the decoder is in low power mode or is running on battery power, only the base level can be decoded, omitting the improvement level, thereby preserving the calculation cycles and, in turn, the power consumption. Spatial and / or temporal masking methods can be used to replace the data of the combined levels (base level and level of improvement).

The improvement level (sometimes interpreted as the low priority level) can be transmitted at low power, in contrast to the base level (or the high priority level). This increases the likelihood of error at the baseline improvement level. Therefore, if the adopted baseline contains a high percentage of errors, then the improvement level could potentially be skipped.

Predicted bidirectional frames (B-frames) are predicted from both the previous frame and the subsequent frame using motion-compensated prediction. B-frames offer very high compression rates and are highly desirable for maintaining bandwidth. B frames are also desirable for their temporal scalability characteristics. Standards-compatible B-frames are not used to predict any other frames, and therefore, they can be skipped without affecting other frames. B-frames are also most susceptible to error propagation due to errors in the frames on which they depend. In this regard, B-frames are most likely to be at the level of improvement (or low priority). If the improvement level is transmitted at low power, then the B-frames are even more susceptible to errors.

If the B-frame (or any other type of frame) is completely damaged or has a percentage of error content above a threshold that makes temporal or spatial masking of errors impossible, then frame rate conversion (FRUC component 44 shown in FIG. 2) can be used. Frame Rate Conversion (FRUC) is used to recover a lost frame. Frame rate conversion (FRUC) can also be used to restore a frame that has been freed from decoding for purposes of, for example, saving energy. Frame rate conversion (FRUC) methods use the motion vector and pixel information from the previous and subsequent frames to interpolate values for motion vectors and pixels. Appropriate frame rate conversion methods (FRUC) are not included in the context of this discussion. Frame Rate Conversion Methods (FRUC) include the “encoder only” and “decoder only” forms. The frame rate conversion method (FRUC) “decoder only” pseudo-interpolates frame data without other information. The encoder-only frame rate conversion (FRUC) method uses third-party information transmitted in H.264 standard enhancement information (SEI) messages or in an MPEG-2 user data message.

FIG. 6 depicts a block diagram of an example component system that may be part of a multimedia receiver, such as receiver 24 of FIG. 1. System 600 is a tiered integration system used to eliminate errors. 7 depicts a flowchart of an error correction process that can be used by system 600. With respect to FIGS. 6 and 7, process 700, at step 705, receives one or more media streams. Data streams may include video data, audio data and / or closed caption text information, etc. Received one or more streams may contain encoded data. The encoded data may be transformed data, quantized data, compressed data, or combinations thereof. A reception means, such as the receiver 24 shown in FIG. 1A, may receive one or more streams in step 705. In addition to receiving multiple media streams, each received stream may comprise multiple layers, such as a base layer and an enhancement layer. Receiver 24 may be a wired or wireless receiver, or a combination thereof.

Errors remaining in the bitstream are detected at block 710. Errors detected may include errors remaining after lower-level error correction and detection protocols containing some of the corrected errors that were introduced by transmission over a channel, such as wireless channel 26 or wired channel 28 (Fig. 1A). As discussed above, not all errors are correctable, and any of the lower layer protocols can flag damaged data and / or data groups that contain errors. The lower layer protocols used in step 710 for error detection can be implemented at the communication layer, as discussed above. Communication layers can be one element from a set containing a physical layer, a MAC layer (or a streaming layer) and a transport layer, or a combination of these elements. A detection means, such as the error detection unit 605 shown in FIG. 6, may perform error detection in step 710. The error detection unit 605 may use various detection schemes known to those skilled in the art, for example, Reed-Solomon and / or turbo code schemes, as discussed above. Error detection may occur due to damage to the CRC in the turbo decoder. Error detection may occur due to a failure of the Reed-Solomon decoder.

Errors detected can be monitored (at 715) using several methods. The inspection step 715 may include the step of marking (or tagging) the corrupted data, as discussed above. Control step 715 may include the step of limiting error propagation by identifying data groups, for example, packets, blocks, parts, frames, macroblocks, submacroblocks, which contain detected errors. Control step 715 may be based on a transport layer protocol. Such a protocol may flag errors remaining after encoding one or more bits in the header of the transport layer for use by higher layers (Figs. 1B and 1C). The upper layers may use transport-level header error indicators to further identify and / or limit the propagation of upper-level packets consisting of one or more corrupted transport-layer packets, thereby further limiting the spread of errors in upper-level bitstreams. A control means, such as the error control unit 610 shown in FIG. 6, can perform error control tasks.

At 720, an error distribution is determined. In one aspect, the error distribution is determined based on a synchronization level protocol. One or more bitstreams that were received at block 705 may be parsed at the synchronization level. If the synchronization level receives information marking damaged data from one of the lower levels, for example, from the communication level, then it can identify damaged parts of the bit streams. The availability of such information can provide the synchronization layer protocol with the ability to plan for masking errors at the upper level (for example, the application level) and / or error correction strategies. Other strategies may be applied depending on the size of the corrupted data. Packets of the transport layer, which may be marked as damaged, can be combined into packets of the synchronization layer, which will be transmitted to various components of the application layer, depending on which part of the bitstream they are. Packets of the transport layer can have an unchanged length, and packets of the synchronization layer can have a variable. The synchronization layer can identify the distribution of errors by inserting data to identify which part of the variable length synchronization packet contains a damaged transport layer packet. In addition to using error control information (block 715), the synchronization level protocol may include additional error detection methods. These error detection methods may include a CRC check of the synchronization layer packets. A specific error distribution can be further transmitted to application layer components by inserting error markings into parsed data packets. A determination means, such as an error distribution determination unit 615, may determine an error distribution (block 720).

At block 725, error correction can be performed on parts of one or more encoded media streams. Error recovery can be based on application layer protocols. Application layer components can perform troubleshooting. Application layer components can determine which type of error recovery to use based on information received from the synchronization layer, as discussed above. Error correction may include one element from a set comprising temporary error concealment, spatial error concealment, frame conversion frequency (FRUC), or a combination of these elements, as discussed above, as well as other methods. The masking of detected errors in one or more streams of multimedia data may be based on an application layer protocol. Troubleshooting can also support the scalability of one or more data streams. Scalability can include either spatial or temporal scalability, or both, as discussed above. An error resolver, such as the error resolver component 40 shown in FIG. 2A, can perform error correction at block 725. An error concealer, such as an error concealer 620, can mask errors. A scaling means, such as the scaling components 42 and 620 shown in FIGS. 2A and 6, can support scalability when performing error correction in step 725.

Process 700 may include a step 730 for recovering one or more bitstreams. Recovery step 730 may include the step of combining the correctly received data with the masked data. The recovery step may include the step of reducing the frame rate (a form of temporary scalability), the number of which exceeds the error threshold in the frame. The recovery step may include a decision not to decode, mask, or adjust the level of improvement (a form of SNR scalability). Application layer protocols may be the basis for recovery step 730. A recovery tool, such as a data stream recovery unit 630, may perform a recovery in step 725. One or more items may be added, rebuilt, or combined in the system 600. One or more items may be added, rebuilt, or combined for a process 700.

Examples of the above methods and devices include the following.

A method for decoding multimedia data, including the steps of receiving a bit stream, performing decoding with error control of the received bit stream, the decoding step with error control comprising the step of identifying damaged bits that have not been corrected, transmitting the bit stream decoded with error control containing identified damaged bits to the decoder, replacing at least one of the identified bits, and decoding the bitstream decoded with error control approx containing the replaced bits. In one aspect, the method further includes the step of parsing the bitstream decoded with error control into one or more characters, as well as the step of parsing information identifying the damaged bits so that any characters containing damaged bits are identified as damaged . In another aspect, the method further includes the step of constructing an error table containing parsed information identifying the damaged bits, where the error table contains information marking the identified bits for positions in the sequence of video frames. In another aspect, the method further includes the step of parsing the error-encoded bitstream into a first parsed bitstream and a second parsed bitstream, where the first parsed bitstream is a base layer bitstream and the second parsed bitstream is an enhancement layer bitstream .

A method for decoding multimedia data, including the steps of receiving a bit stream, performing decoding with error control of a received bit stream, where the step of decoding with error control comprises the step of identifying damaged bits that have not been corrected, transmitting a bit stream decoded with error control containing identified damaged bits, to the decoder, replacing at least one of the identified bits by performing frame rate conversion between the first adrom and second frame, and decoding the bit stream decoded with error control.

Specialists in the art should understand that information and signals can be represented using any variety of different technologies and methods. For example, data, instructions, commands, information, signals, bits, symbols and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any their combination.

Specialists in the art should also understand that various illustrative logical blocks, modules, and algorithm steps described with reference to the examples disclosed herein can be implemented as electronic hardware, firmware, software, firmware, microprograms or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above, in general, based on their functionality. Whether such functionality is implemented as hardware or software depends on the specific purpose and structure constraints imposed on the entire system. Skilled artisans may implement the described functionality in varying ways for each particular purpose, but such implementation decisions should not be interpreted as a departure from the scope of the disclosed methods.

The various illustrative logical blocks, components, modules, and circuits described with reference to the disclosed examples that can be implemented or performed using a universal processor, digital signal processor (DSP), specialized integrated circuit (ASIC), gate array with operational programming (FPGA) or other programmable logic device, logic element on discrete components or transistor logic, discrete hardware components or any oh combination thereof designed to perform the functions herein. A universal processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, for example, a combination of a digital signal processor (DSP) and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor (DSP) core, or any other such configuration.

The steps of a method or algorithm described with reference to the examples disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination thereof. The program module may reside in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable memory (EPROM), electrically erasable programmable memory (EEPROM), registers, on the hard disk , on a removable disk, a compact disk storage device (CD-ROM), or in any other form known in the art of storage media. An exemplary storage medium is connected to the processor so that the processor can read and write information to the storage medium. Alternatively, the storage medium may be an integral part of the processor. The processor and the storage medium may reside in a dedicated integrated circuit (ASIC). Specialized Integrated Circuit (ASIC) can reside in a wireless modem. Alternatively, the processor and the storage medium may reside as discrete components in a wireless modem.

The foregoing description of the disclosed examples is provided to provide any person skilled in the art with the ability to create or use the disclosed methods and devices. Various modifications to these examples will be apparent to those skilled in the art, in addition, the principles defined herein may be applied to other examples, and additional elements may be added.

Thus, methods and apparatus for decoding streaming multimedia data in real time using information marking damaged bits and damaged data have been described in a decoder application for performing intelligent error concealment and error correction of damaged data.

Claims (40)

1. A method for eliminating errors in multimedia data using multi-level integration, comprising the steps of detecting errors in multimedia data through a first level based on a protocol for the first level; determining the distribution of the detected error by a second level based on a protocol for the second level; and mask the errors detected in the multimedia data by a third level based on a protocol for the third level and a specific error distribution. 2. The method according to claim 1, in which the first level is the communication level. 3. The method of claim 2, wherein the communication layer is one of or a combination of a physical layer, a MAC layer (or a streaming layer), and a transport layer. 4. The method according to claim 1, further comprising monitoring the detected error at a transport layer based on a protocol for the transport layer. 5. The method according to claim 4, in which the step of monitoring the detected error comprises the step of restricting the spread of the detected error. 6. The method according to claim 1, in which the second level contains a synchronization level, and in which the error distribution identifies which part of which bitstream is damaged and provides one or more strategies for masking the error and correcting the error for the third level. 7. The method according to claim 1, in which the third level contains the application level. 8. The method according to claim 1, in which the first level contains the communication level and the transport level, the second level contains the synchronization level, the third level contains the application level, the method also includes controlling the detected error by means of the transport layer based on the protocol for transport level. 9. A device for eliminating errors in multimedia data using multi-level integration, comprising: means for detecting errors in multimedia data based on a protocol for a first level, means for determining a detected error based on a protocol for a second level; and means for masking an error detected in the multimedia data based on a protocol for a third level and a specific error distribution. 10. The device according to claim 9, in which the first level is the communication level. 11. The device of claim 10, wherein the communication layer is one of or a combination of a physical layer, a MAC layer (or a streaming layer), and a transport layer. 12. The device according to claim 9, which also contains means for monitoring the detected errors based on the protocol for the transport layer. 13. The device according to item 12, in which the control means comprises means for limiting the distribution of the detected error. 14. The device according to claim 9, in which the second level contains a synchronization level and in which the error distribution identifies which part of which bitstream is damaged and provides one or more strategies for masking errors and correcting errors for the third level. 15. The device according to claim 9, in which the third level contains the application level. 16. The device according to claim 9, in which the first level contains the communication level and the transport level, the second level contains the synchronization level, the third level contains the application level, and the device also contains means for monitoring the detected error through the transport layer based on the protocol for the transport layer . 17. A device for eliminating errors in multimedia data using multi-level integration, comprising a sensor in a first level that detects an error in multimedia data based on a protocol for a first level; an error distribution determinant in the second level, which determines the distribution of the detected error by the second level based on the protocol for the second level; and a masking unit in the third level, which masks the error detected in the multimedia data based on the protocol for the third level and a specific error distribution. 18. The device according to 17, in which the first level is the communication level. 19. The device of claim 18, wherein the communication layer is one of or a combination of a physical layer, a MAC layer (or a streaming layer), and a transport layer. 20. The device according to 17, which also contains a control unit in the transport layer, which controls the detected error based on the protocol for the transport layer. 21. The device according to claim 20, in which the control unit limits the spread of the detected error. 22. The device according to 17, in which the second level contains a synchronization level and in which the error distribution identifies which part of which bitstream is damaged and provides one or more strategies for masking errors and correcting errors for the third level. 23. The device according to 17, in which the third level contains the application level. 24. The device according to 17, in which the first level contains the communication level and the transport level, the second level contains the synchronization level, the third level contains the application level, and the device also contains a control unit in the transport level to control the detected error through the transport level protocol for the transport layer. 25. A processor for eliminating errors in multimedia data using multi-level integration, configured to detect errors in multimedia data through a first layer based on a protocol for the first level; and determining a distribution of the detected error by a second layer based on a protocol for the second layer; and masking an error detected in the multimedia data by a third level based on a protocol for the third level and a specific error distribution. 26. The processor of claim 25, wherein the first layer is a communication layer. 27. The processor of claim 26, wherein the communication layer is one of or a combination of a physical layer, a MAC layer (or a streaming layer), and a transport layer. 28. The processor according A.25, also configured to control the detected errors through the transport layer based on the protocol for the transport layer. 29. The processor of claim 28, wherein the control unit restricts the propagation of the detected error. 30. The processor according A.25, in which the second level contains the synchronization level and in which the error distribution identifies which part of which bitstream is damaged and provides one or more strategies for masking errors and error correction for the third level. 31. The processor of claim 25, wherein the third layer comprises an application layer. 32. The processor of claim 25, wherein the first layer comprises a communication layer and a transport layer, the second layer contains a synchronization layer, the third layer contains an application layer, and the processor is also configured to monitor a detected error based on a protocol for the transport layer. 33. A computer-readable medium for implementing a method for eliminating an error in multimedia data using multi-level integration, comprising the steps of detecting an error in multimedia data by means of a first layer based on a protocol for the first level; determining the distribution of the detected error by a second level based on a protocol for the second level; and mask the errors detected in the multimedia data by a third level based on a protocol for the third level and a specific error distribution. 34. The medium of claim 33, wherein the first level is a communication level. 35. The medium of claim 34, wherein the communication layer is one of or a combination of a physical layer, a MAC layer (or a streaming layer) and a transport layer. 36. The medium of claim 33, wherein the method further comprises monitoring a detected error by a transport layer based on a protocol for the transport layer. 37. The medium of claim 36, wherein the step of monitoring the detected error comprises the step of restricting the spread of the detected error. 38. The medium of claim 33, wherein the second level comprises a synchronization level and in which the error distribution identifies which part of which bitstream is damaged and provides one or more strategies for masking the error and correcting the error for the third level. 39. The medium of claim 33, wherein the third level comprises an application layer. 40. The medium of claim 33, wherein the first level comprises a communication layer and a transport layer, the second layer contains a synchronization layer, the third layer contains an application layer, and the medium also contains instructions for monitoring a detected error based on a protocol for the transport layer.

Images