WO2007045141A1

WO2007045141A1 - A method for supporting multimedia data transmission with error resilience

Info

Publication number: WO2007045141A1
Application number: PCT/CN2006/001846
Authority: WO
Inventors: Zhong Luo; Bin Song
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2005-10-17
Filing date: 2006-07-25
Publication date: 2007-04-26
Also published as: CN100450187C; CN1859580A

Abstract

A method for supporting multimedia data transmission with error resilience is disclosed, thereby the error resilience mechanism of multimedia data real-time transmission can be realized on the transmission protocol layer. In present invention, at first, it provides multimedia data real-time transmission by providing the ERRTP protocol which carrying information about the forward error correction(FEC) coding manner to the exist RTP protocol, so that it can mark the information about the corresponding FEC coding manner for multimedia data at the same time it is transmitted on ERRTP, thereby the error resilience mechanism can be introduced in transmission layer. Next, each standby FEC coding manner can be selected according to present network condition and the level of importance about the multimedia data at transmitting end, so that it can achieve the purpose of protect based on the level, and can realize the equilibrium about the protect ability and the transmission efficiency.

Description

Multimedia data transmission method supporting fault tolerance elasticity

Technical field

The present invention relates to the field of multimedia communication technologies, and in particular, to a multimedia data transmission method supporting fault tolerance and flexibility. Background technique

With the rapid development of computer Internet (Internet) and mobile communication networks, streaming media technology is becoming more and more widely used, from streaming media, movie playback to distance learning and online news sites. Currently, there are two ways to download video and audio on the Internet, including downloading and streaming. Streaming is the continuous transmission of video/audio signals, and the rest of the video continues to be downloaded in the background while the streaming media is playing. Streaming has two methods: Progressive Streaming and Realtime Streaming. Real-time streaming is a real-time transmission, especially for live events. Real-time streaming must match the connection bandwidth, which means that the image shield will be degraded due to the reduced network speed to reduce the need for transmission bandwidth. The concept of "real time" means that the delivery of data in an application must be kept in precise time relationship with the generation of the data.

Especially with the advent of third-generation mobile communication systems (3G, 3rd Generation) and the rapid development of networks based on Internet Protocol (IP), video communication is gradually becoming one of the main services of communication. Two-way or multi-party video communication services, such as video telephony, video conferencing, and mobile terminal multimedia services, impose strict requirements on the transmission of multimedia data streams and the quality of services. Not only does network transmission require better real-time performance, but equivalently requires video data compression coding to be more efficient.

In view of the current demand for media communication, the ITU-T Telecommunication Standardization Sector officially released Η.264 in 2003 after the development of video compression standards such as Η·261, Η·263, Η.263+. standard. This is an efficient compression coding standard jointly developed by ITU-T and the Moving Picture Experts Group (MPEG) of the International Standardization Organization (ISO) to adapt to the new phase of network media transmission and communication requirements. It is also the main content of Part 10 of the MPEG-4 standard. The purpose of the H.264 standard is to improve video coding efficiency and its adaptability to the network more effectively. In fact, due to its superiority, the H.264 video compression coding standard has gradually become the mainstream standard in multimedia communication. A large number of H.264 multimedia real-time communication products (such as conference TV, videophone, 3G mobile communication terminal) and network streaming products have been published. Whether to support H.264 has become the key to determining product competitiveness in this market segment. factor. It can be predicted that with the official promulgation and widespread use of H.264, multimedia communication based on IP networks and 3G and 3G wireless networks will inevitably enter a new stage of rapid development.

As mentioned above, multimedia communication not only requires high efficiency of media compression coding, but also requires real-time transmission of the network. At present, multimedia streaming basically adopts Real-time Transport Protocol (RTP) and Real-time Transport Control Protocol (RTCP). RTP is a transport protocol for multimedia data streams over the Internet, published by the Internet Engineering Task Force (IETF). RTP is defined to work in one-to-one or one-to-many transmissions with the goal of providing time information and stream synchronization. The typical application of RTP is based on the User Datagram Protocol (UDP), but it can also work on other protocols such as TCP (Transport Control Protocol) or Asynchronous Transfer Mode (ATM). .

RTP itself only guarantees the transmission of real-time data, and does not provide a reliable transmission mechanism for transmitting packets in sequence, nor does it provide flow control or congestion control. It relies on RTCP to provide these services. RTCP is responsible for managing the transmission quality to exchange control information between current application processes. During the RTP session, each participant periodically transmits RTCP packets, which contain statistics such as the number of transmitted packets and the number of lost packets. Therefore, the server can use this information to dynamically change the transmission rate, even Change the payload type. RTP and RTCP work together to optimize transmission efficiency with effective feedback and minimal overhead, making it suitable for delivering real-time data on the network.

H.264 multimedia data is transmitted over the IP network, also based on UDP and its upper layer RTP protocol. RTP itself is structurally applicable to different media data types, but different high-level protocols or media compression coding standards in multimedia communication (eg H.261, H.263, MPEG-1/-2/-4, MP3) Etc), the IETF will develop an RTP net for the agreement. The specification file of the Payload packaging method, which specifies the method of encapsulating large packets of RTP, is optimized for this specific protocol. Similarly, the corresponding IETF standard for H.264 is RFC 3984: RTP Payload Format for H.264 Video. This standard is currently the main standard for H.264 video stream transmission over IP networks, and is widely used. In the field of video communication, the products of major manufacturers are based on RFC 3984, and it is currently the only H.264/RTP transmission method.

In fact, the key difference between H.264 and other video compression coding protocols is that H.264 defines a new layer, called Network Abstract Layer (NAL), which is a standard that makes it standard. The interface opens up the underlying business capabilities and shields the underlying network from the differences and abstracts the business capability layer. In order to increase the separation and independence of its video coding layer (VCL, Video Coding Layer) and the following specific network transport protocol layer, H.264 brings greater application flexibility and defines a new layer of NAL. The early ITU-T video compression coding protocols such as H.261, H.263/H.263+/H.263++ were not available. However, how to design a more efficient and better solution for the advantages of H.264 in the NAL and RTP protocol bearer cooperation makes RTP better for H.264, practical, and worthy of study.

The method of RTP carrying H.264 NAL layer data proposed by RFC3984 is the current mainstream transmission method. Based on RTP protocol (RFC 3550), the scheme encapsulates NAL layer data in RTP payload for bearer. The NAL layer is located between the VCL and the RTP, and specifies that the video stream is divided into a series of network abstraction layer data units (NALUs, NAL Units) according to defined rules and structures. The encapsulation format of the RTP payload for NALU is defined in RFC3984. The following is a brief introduction to the RTP frame format and the NALU packaging method in the prior art.

The main objectives of the RTP design are real-time multimedia conferencing and continuous data storage, interactive distributed simulation, control and measurement applications. RTP is typically carried over the UDP protocol to take advantage of its multiplexing and parity functions. If the underlying provides multipoint distribution, RTP supports multi-address delivery. Features provided by RTP include: payload type identification, sequence numbering, timestamp, and transmission monitoring.

In the case of carrying H.264 video, RTP packages the NA. package of H.264 into RTP. Packet flow. The NALU is mainly defined in the RFC 3984 file, and based on this, the encapsulation and packing format of the H.264 layer NAL data in the RTP is given. The RTP encapsulation format of this NALU is shown in Figure 2. '

Figure 1 shows the encapsulation structure of a NALU in the payload of the RTP. The first byte in the previous byte is the NALU header information, followed by the data content of the NALU. The multiple NALUs are filled end-to-end into the payload of the RTP packet. Finally, there is optional RTP padding, which is specified in the RTP packet format. In order to make the length of the RTP packet meet certain requirements (such as reaching a fixed length), the optional RTP padding data is generally filled with zeros.

The NALU header information is the first byte, also known as the octet (Octet), which has three fields. The meaning and full name are respectively described as follows:

The F field is defined as a forbidden bit (forbidden-zero-bit), which is 1 bit, used to identify grammatical errors, etc., and is set to 1 if there is a syntax conflict. When the network recognizes that there is a bit error in this unit, it can be set. Is 1, for the receiver to drop the unit, mainly used to adapt to different kinds of network environments (such as wired and wireless combined environment);

The I field is defined as the NAL reference identifier (nal_ref_idc), which is 2 bits, used to indicate

The importance of NALU data, whose value is 00 means that the content of the NALU is not used to reconstruct the inter-predicted reference picture, while the non-00 indicates that the current NALU is a slice or sequence parameter set belonging to the reference frame (SPS, Sequence Parameter Set), image parameter set (PPS, Picture Parameter Set) and other important data. The larger the value, the more important the current NAL is;

The Type field is defined as NALU type (Nal_unitjype), a total of 5 bits, which can have

The types of 32 NALUs, the correspondence between their values and specific types are given in detail in Table 1.

Table 1 Relationship between Type and Type of Type Fields in NALU Header Information

Type value Type of NALU content

0 not specified

1 encoding of non-IDR images

2 encoding slice data division A

3 encoding slice data division B

4 encoding slice data division C

5 Coded slice in IDR image 6 SEI (Supplemental Enhancement Information)

7 SPS (sequence parameter set)

8 PPS (image parameter set)

9 access unit delimiter

10 end of sequence

11 code stream ends

12 Fill data

13-23 Reserved '

24-31 Unspecified It can be seen that the information given in one byte of the NALU header information mainly contains the validity and importance level of the NALU. Based on this information, the importance of the data carried by the RTP can be determined.

After understanding the transport structure of H.264/RTP, closely related to the content of the present invention is a fault tolerant resilient mechanism for multimedia network transmission. The following is a brief introduction to the fault-tolerant resiliency and related technical background of video network transmission.

H.264 video is the main protocol for multimedia communication in the future. The network of future multimedia communication applications is mainly the packet switching network and wireless network represented by IP. Neither of these two types of networks can provide good quality of service (QoS) guarantees. Therefore, video transmission on the network is bound to be affected by various transmission errors and packet loss, resulting in lower communication quality. Since the IP network implements "best effort" transmission, it does not guarantee the QoS of the transmitted video signal. Especially for H.264 code streams that are efficiently compressed and encoded. The best-effort transmission on the IP network does not guarantee the QoS of real-time video communication, which is manifested in three aspects: packet loss, delay, and delay jitter. Among them, packet loss has the greatest impact on the quality of recovered video. Because H.264 compression coding algorithm uses motion estimation and motion compensation technology, once there is packet loss, it not only affects the current decoded image, but also affects the subsequent decoded image. Error spread. The effect of error spread on recovering video quality is very large. Only when the combination of the encoding end and the decoding end is combined with error resistance can the error spread be completely avoided.

Error Resilience refers to the ability of the transmission mechanism to prevent errors from occurring or to be corrected with certain ability after the error occurs. (The error strength can be completely corrected within a certain range; if it exceeds a certain range, it can only be partially corrected). Extensive in the future (can be said to be omnipresent) In a multimedia communication environment, it is critical that a video delivery mechanism is resilient to fault.

There are a variety of fault-tolerant resilience mechanisms, such as Forward Error Correction (FEC), Automatic Retransmission Request (ARQ), Error Concealment, and Source Channel Joint Coding (JSCC, Joint Source- Channel Coding), Interleaving, and elimination of bit error spread. For H.264 video to be transmitted over a packet network, FEC is a very practical technique that works well. This method mainly uses a variety of error correction coding to encode the data to be protected, which essentially forms data redundancy, thereby increasing the ability to resist errors.

The main error of the packet on the network is the packet loss error, which is called Erasure Error in the error correction coding theory. Error correction codes for deletion errors are a large class called Erasure Codes. The so-called erasure code is to divide the data stream sequence into segments of the same size (Unit), also called data nodes (Data Nodes). For convenience of presentation, it is assumed that there are n data nodes. Then, according to certain mathematical operation rules, these data nodes are calculated to generate a check node (? 1:

In order to enhance the protection capability, the check nodes may continue to generate the second layer check node according to the same or different mathematical operation rules, and so on, and the third layer, the fourth layer, and the Nth layer check may be generated. node.

In general, if multiple layers of check nodes are involved, the number of nodes on each layer is decremented according to a certain rule (the most common is the law of proportionality), so that it becomes a layer-by-layer shrinkage. Layer node structure. It can be visually represented as a pyramid that turns 90 degrees to the right. The leftmost side is the data node layer, and the right side is the first layer check node, the second layer check node, ..., the Nth layer check node.

One type of erasure code has a very important property, that is, the time complexity required for processing is linear with the number n of data nodes, so it is called linear time characteristic (linear-time I and many other erasure codes such as famous The Reed-Solomon code requires much more time complexity and is on the order of n*log2n*log(logn). Therefore, linear time-based erasure codes are much better used in real-time communication.

Tornado erasure code (hereinafter referred to as Tornado code) is a kind of one that appeared around 1998. New erasure code. Tornado code is simple in structure and efficient in operation because it has linear time and strong protection. In practical applications, good results have been obtained. It has been widely used. 1" The latest ITU-T dynamics, where SG16 is currently considering the possibility of standardizing Error Control Codes technology, mainly for video and audio network transmission protection. Tornado code and its many variants are very May be an important technology among them.

In the Tornado code, multiple check node layers are generated layer by layer from the data node. Both the check node and the data node are sent by the sender to the receiver over the network. If some nodes are lost during the network transmission process, because the upper node participates in the generation of the lower node, the information of the upper node is already included in the lower node and the lower node, so the information of the lost node can pass the lower level of sufficient majority. The node or lower node is fully recovered. If each node is a packet, the lost packet can be fully recovered by other packets that are correctly received. Let the number of data nodes be n, and the number of generated check nodes is L. The code rate and redundancy rate of the erasure code are defined as: r=n/(n+l), lr=l/(n+l) Under the same conditions (protection ability, delay caused, etc.), the higher the code rate (inevitably, the lower the redundancy rate), the higher the efficiency of the erasure code.

Figure 3 shows a typical Tornado code data node and the relationship between the check nodes of each layer. The connection between the nodes in the figure is called the edge, and the node on the left side of the edge participates in the calculation of the right node. It can be seen that there is a many-to-many logical relationship between the two nodes before and after. The most commonly used calculation method in the Tornado code generation process is the XOR operation, because the XOR operation has a convenient recovery function, and any node can be recovered by all the remaining nodes after it is lost. Since the scaling factor of the last layer of check nodes is different, it is generally calculated using a conventional error correction coding scheme, such as a Reed-Solomon code.

In fact, the range of erasure codes is very large. Tornado codes are only one of them. In addition, there are RS (Reed-Solomon) codes and Low Density Parity Codes (LDPC).

An important performance indicator of the erasure code is its error correction capability (or protection capability), which is directly reflected in the maximum number of lost packets allowed under the packet loss error (under the total number of precursors of a certain packet), or when The packet loss is higher than this maximum allowable number, and the percentage of the packet can be corrected correctly. In general, the protection is higher and the redundancy is the same under other conditions. The higher the rate.

The protection capability is not only applicable to erasure codes, but on a larger scale, all FEC codes can be measured by protection capabilities. In video data, some data are relatively important, such as structural parameters of video sequences, structural parameters of images, header information, etc. Other data are relatively less important, such as image content data. When using FEC for protection, a code with stronger protection is used for relatively important data, and a code with weak protection for relatively unimportant data. This balances protection and efficiency. The protection capability cannot be adjusted blindly because it leads to high redundancy and the P-bar is inefficient. This method of FEC protection based on the relative importance of data for different protection capabilities is called Unequal Protection (UEP), and QoS guarantee for video communication services is easily realized by unequal protection.

Currently, the RTP protocol for transmitting video multimedia data does not support fault-tolerant flexibility and is provided by a higher application layer. In the prior art, for the transmission of video data networks such as H.264, the erasure code protection is generally used to achieve elastic fault tolerance. Taking H.264 as an example, the measures taken by the prior art scheme are: - The sender is at the NALU level of H.264, and directly uses some type of erasure code for the NALU data unit, and then the result (including the data node and the checksum) The node) is directly encapsulated in the RTP packet and then transmitted.

After receiving the RTP data packet, the receiving end performs decapsulation to extract the data node and the check node. If packet loss occurs, that is, some or some RTP data packets are lost, then according to which data nodes are encapsulated in the lost packets. Or verifying the node, it can be judged whether the correctly received data node and the check node can be used to completely recover or partially recover the lost node, and the recovery operation is performed.

Of course, other fault-tolerant mechanisms other than erasure codes are used, but the protection of H.264 data can provide the most efficient fault-tolerant elastic mechanism.

It can be seen that the prior art performs erasure coding on the multimedia data such as NALU at the upper layer and then transmits the data in the RTP, and performs corresponding erasure decoding on the receiving end. It should be noted that the transmitting and receiving parties generally negotiate and decide what forward error correction coding scheme to use and the parameter settings adopted by the scheme, such as H.323/H.245 and other protocol channels. The two sides negotiated. In practical applications, the fault-tolerant and flexible mechanisms in the prior art solutions are implemented at the upper layer of the RTP. The two parties negotiate or inform the type of the erasure code to be used and its parameter settings need to be implemented through other logical channels, which seriously affects the multimedia transmission efficiency. The network bandwidth resource is consumed. For the RTP transport layer, the fault-tolerant resiliency mechanism is transparent. Therefore, the RTP layer cannot know the structure of the encoded multimedia data generated by the FEC codec scheme, and thus cannot perform targeted encapsulation and encapsulation. , unable to reorganize the transport hierarchy, lengthen network transmission delays, and the transmission equipment becomes complicated;

After the transmitting and receiving ends negotiate the FEC encoding scheme, the multimedia data is always transmitted according to the scheme. For different importance data and network transmission states at different times, the unequal protection mechanism cannot be implemented, and the fault-tolerant elastic mechanism cannot be implemented. To achieve QoS guarantee.

The prior art implements a fault-tolerant elastic mechanism such as FEC at a high level, and does not utilize the RTP protocol and its encapsulation. Therefore, the transmitting and receiving parties need to establish another logical channel or use a specific application layer protocol, such as some in the H.323 protocol system. Protocol H.245, to negotiate or inform the FEC encoding type, structural parameters and other information used; no fault-tolerant resiliency related details are involved in the RTP layer, and no RTP data packet is encapsulated to encapsulate the data nodes and check nodes generated by FEC protection; There is also no choice of FEC codec scheme according to the network condition and the importance of multimedia data, and there is no mechanism for providing FEC protection for different protection capabilities with different relative importance data, that is, unequal protection cannot be achieved. Summary of the invention

In view of this, the main purpose of the present invention is to provide a real-time transmission method for a multimedia data network that supports fault-tolerant resilience, so that a fault-tolerant elastic mechanism for real-time transmission of multimedia data can be implemented at a transmission protocol level. Further U of the present invention is implemented for Unequal protection mechanisms and hierarchical protection mechanisms for different data and network conditions.

A real-time transmission method for a multimedia data network supporting fault tolerance resilience according to the present invention includes:

The transmitting end selects a forward error correction coding mode to perform forward error correction coding on the multimedia data;

The transmitting end encapsulates the encoded multimedia data by using a fault-tolerant elastic real-time transmission protocol, and And carrying the forward error correction coding mode related information in the header information of the fault tolerant elastic real-time transmission protocol data packet, and sending the information to the receiving end;

The receiving end decapsulates the received fault-tolerant elastic real-time transport protocol data packet, and extracts the forward error correction coding mode related information from the header information of the fault-tolerant elastic real-time transport protocol data packet;

When the fault-tolerant elastic real-time transport protocol packet corresponding to the data node is lost during the transmission, the receiving end selects the forward error correction decoding mode to perform forward error correction decoding according to the forward error correction coding mode related information, Restoring or partially recovering the lost multimedia data.

The forward error correction encoded multimedia data includes a data node and a check node.

The transmitting end selects a forward error correction coding mode according to a current network transmission condition or/and a service quality level of the multimedia data to be transmitted, wherein the service volume level is determined according to the relative importance of the data.

The packet fault information of the fault tolerant elastic real-time transport protocol includes:

a forward error correction coding type field, configured to indicate a forward error correction code type used; a forward error correction coding subtype field, configured to indicate a related parameter setting of the forward error correction coding mode;

a packet length field, configured to indicate a length of a node obtained after correcting the forward error correction code for the multimedia data;

A packet number field, used to indicate the number of the data nodes carried by the fault tolerant elastic real-time transport protocol data packet.

Preferably, when the multimedia data is an H.264 network abstraction layer unit, the transmitting end divides at least one of the H.264 network abstraction layer units into at least one data node of equal length, and then performs the foregoing. Encoding to the error correction to obtain at least one calibration node; the transmitting end encapsulates the data node and the verification node packet in at least one of the fault tolerant elastic real-time transmission protocol packets for transmission;

After receiving the fault tolerant elastic real-time transport protocol packet, the receiving end decapsulates the data node and the check node;

If a data node loss occurs during transmission, the receiving end is according to the school The node performs forward error correction decoding on the data node, and divides and obtains the H.264 network abstraction layer unit.

More suitably, before starting the transfer, include:

For each of the types of the forward error correction code, the transmitting end and the receiving end negotiate to determine the value of the fault tolerant forward error correction code subtype field and the related parameter setting of the forward error correction code indicated. Correspondence relationship.

And the sending end and the receiving end both establish a correspondence table according to the indication correspondence relationship of the forward error correction coding subtype field, and configured to perform, according to the forward error correction coding type field and the forward error correction coding The forward type error correction coding or forward error correction decoding processing module corresponding to the subtype field query;

The transmitting end invokes a corresponding forward error correction coding processing module to perform forward error correction coding; the receiving end invokes a corresponding forward error correction decoding processing module to perform forward error correction decoding. Determining, by the sending end, the relative importance of the corresponding data according to the network abstraction layer reference identifier field or/and the network abstraction layer unit type field in the header information of the H.264 network abstraction layer unit, determining the quality of service level, selecting Corresponding forward error correction coding mode determines the forward error correction coding type field and the forward error correction coding subtype field.

The transmitting end evaluates the network transmission status according to the transmission report fed back by the receiving end, and further selects the forward error correction coding mode, and determines the forward error correction coding type field and the forward error correction coding subtype field. .

Preferably, the forward error correction coding type field is located after the contribution source identifier list; the forward error correction coding subtype field is located after the forward error correction coding type field;

The data packet length field is located after the forward error correction coding subtype field; the data packet number field is located after the data packet length field.

Preferably, the forward error correction coding mode uses an improved "Tornado" erasure code; the improved "Tornado" erasure code generates only one layer of the check node for a set of said data nodes.

The main difference from the prior art is that, according to the technical solution of the present invention, first, an ER TP transmission that can carry information related to the forward error correction coding scheme is provided on the basis of the existing RTP. Sending a layer encapsulation format, so that the multimedia data is transmitted on the ERRTP while marking its corresponding forward error correction coding scheme information, thereby integrating the error resilience mechanism into the transport layer;

Secondly, at the transmitting end, various alternate forward error correction coding schemes can be selected according to factors such as current network conditions and multimedia data importance levels, thereby achieving the purpose of unequal protection and hierarchical protection, achieving protection capability and transmission. Balance of efficiency;

Finally, for the NALU erasure code protection scheme of Η.264, the methods of generating, transmitting, encapsulating and decapsulating data nodes and check nodes are given.

The fault-tolerant elastic mechanism in the transport layer greatly simplifies the fault-tolerant elastic transmission structure, which saves the network transmission bandwidth. The realization of the unequal protection achieves the balance between protection capability and transmission efficiency, facilitating the realization of QoS guarantee for multimedia transmission; H.264 data The implementation of the specific transmission scheme can greatly improve the performance and user satisfaction of H.264-based multimedia communication products such as conference television, videophone application on IP networks. DRAWINGS

1 is a schematic diagram of a package format of an RTP packet payload to NALU data;

2 is a schematic diagram showing the structure of a header information of an RTP data packet;

Figure 3 is a schematic diagram of the Tornado erasure code principle;

4 is a schematic structural diagram of an EERTP packet header according to a first embodiment of the present invention; FIG. 5 is a flow chart of a H.264 multimedia data transmission method according to a second embodiment of the present invention;

6 is a schematic diagram of a H.264 NALU partitioning codec process according to a second embodiment of the present invention. detailed description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings.

In view of the problems existing in the prior art, the present invention proposes an improved RTP protocol supporting fault tolerance resilience, which aims to integrate a fault-tolerant elastic mechanism into a transport layer protocol, which not only simplifies the transmission structure and reduces complexity, but also improves the fault-tolerant elastic mechanism. Flexibility enhances transmission reliability. Due to its fault-tolerant flexibility, this improved RTP protocol is called a fault-tolerant elastic real-time transport protocol. ( ERRTP /ER2TP, Error Resilience Real-time Transport Protocol ). The main difference between ERRTP and RTP is that the ERTP protocol packet header information extension can carry information about the forward error correction codec scheme, such as FEC type, protection capability, and coding parameters.

On the basis of ERRTP, the present invention conveniently realizes unequal protection. Firstly, various protection measures with different protection capabilities are available for selection, and then the sender can collect information such as network status and importance of multimedia data. These factors are used to select appropriate protection measures to achieve the goal of unequal protection and to achieve a balance between protection capability and transmission efficiency. Since the FEC related information is carried on each ERRTP data packet, the transmitting end only needs to fill in the information of the selected scheme into the ERRTP header information, and the receiving end can correctly recover or correct the error according to it. .

Finally, for the NALU data transmission application of H.264, the specific implementation method based on erasure code protection is given, including the steps of dividing, generating, encapsulating and decapsulating the data node and the check node. A series of NALUs are equally divided into several data nodes, and then the check nodes are generated by Tornado codes. All of these nodes are distributed in several ERRTP packets, and the receiving end performs this inverse process.

In order to facilitate the understanding of the technical solution of the present invention, here, the format of the RTP packet is briefly introduced: The basic option of the RTP header information occupies 12 bytes (minimum case), and the header information of the IP protocol and the UDP protocol respectively occupy 20 bytes and 8 words. Therefore, the RTP packet is encapsulated in the UDP packet and then encapsulated in the IP packet. The total number of bytes occupied by the header information is 12+8+20=40 bytes. The detailed structure of the header information of the RTP packet is shown in Figure 2.

The front-to-back RTP header information shown in Figure 2 is: The first byte (byte 0) is some field about the header information structure itself, the second byte (byte 1) is the defined payload type, the third 4 bytes (bytes 2, 3) are the sequence number (Sequence Number), the 5th-8th byte is the timestamp (timestamp), and the 9th-12th byte is the synchronous contribution source identifier (SSRC ID, Synchronous Source) Identifier ) , and finally the list of contributing source identifiers ( CSRC Ids , Contributing Source Identifiers ), the number of which is uncertain. Note that the first byte in the description in this article is the byte 0 of the label, and so on.

The first 12 bytes appear in all different types of RTP packets, while other data in the header information, such as the contribution source identifier, is only available when the mixer is inserted. therefore CSRC is generally used when there is media mixing. For example, in multi-party conferences, audio needs to be mixed, and video can also provide multi-screen functions in this way. The synchronization source identifier SSRC is actually the identifier of the carried media stream.

The specific meanings and full names of the above fields are described as follows:

The V field is version (Version) information, which occupies 2 bits. Currently, the version used is 2, so V=2 is set, and other values such as V=l indicate the earlier RTP version, and V=0 indicates the original. The RTP predecessor, which was adopted in the voice over IP (VOIP) communication system used on the early Mbone network, later evolved into RTP, and V=3 has not yet been defined, so the present invention can be used. ;

The P field is a padding flag (Padding), which occupies 1 bit. If P is set, it indicates that the packet contains one or more padding bytes (Padding) at the end, and the padding is not part of the payload;

The X field is an extension identification bit (Extension), which occupies 1 bit. If X is set, the RTP header must be followed by a variable-length header extension (if there is a CSRC list, the header extension is followed), mainly Retaining the case where the header information field is not sufficient for some application environments, the header extension includes a 16-bit length field to count how many 32-bit words in the extension, and the first 16 bits of the header extension are left-open. In order to distinguish between identifiers and parameters, the 16-bit format is defined by a specific level specification, which is described in detail in section 5.3.1 of RFC 3550, which is not given here;

The CC field is the number of contributing sources (CSRC Count), which is 4 bits s, indicating the number of CSRC identifiers at the end of the header information, and the receiving CC field can determine the length of the CSRC IDs list following the header information;

The M field is a marker bit (Marker), which occupies 1 bit. The interpretation of the identifier bit is defined in a specific profile, which allows identification of important events in the packet stream. One layer can define additional identification bits or regulations. There is no identification bit. The so-called level here refers to the specific application environment setting, which is specifically agreed by the communication parties and is not limited by the agreement;

The PT field is the payload type (PT, Payload Type), a total of 7 bits s, identifies the format of the RTP payload and determines his interpretation in the application; the flag bit and the payload type share a layer of specified information, this byte may It will be redefined by specific levels to suit different needs. In a specific application, a so-called profile can be defined, which is actually a set of static (ie communication). The two parties agree on the corresponding relationship in advance, and the different values of the FT bits are associated with different media formats. Of course, dynamic negotiation can also be used to define the relationship between the FT value and the media format through signaling other than RTP. In an RTP session (Session), the RTP source can change the PT.

The following field is the serial number of a total of 16 bits. Each time an RTP data packet is sent, the serial number value is incremented by one, so that the receiver can use it to detect the data packet loss and recover the data packet sequence. The initial value of the serial number in one communication can be given randomly. , does not affect communication.

The timestamp occupies 32 bits, which reflects the sampling time of the first byte in the RTP packet. The sampling time here must be derived from a monotonically increasing clock, and the receiver adjusts the media playback time or synchronizes according to it.

The synchronization source SSRC ID occupies 32 bits, and its specific value can be randomly selected. However, to ensure the uniqueness in the same RTP session, it can uniquely identify a media source. If a source changes the source transmission address, a new SSRC must be selected. The identifier.

The source CSRC list can be 0-15 items as needed, each item occupying 32 bits s, and the length of the list, ie the number of CSRC IDs, is exactly indicated by 4 bits of the CC field. In fact, the CSRC identifier used to identify a media source is identical to the SSRC identifier of its corresponding contribution source, except that the role of the different receivers is different and is set to SSRC or CSRC. In multiparty communication, the CSRC ID is inserted by the mixer.

First embodiment

In this embodiment, the sending and receiving parties implement unequal protection based on ER TP. The main steps are as follows:

The transmitting end selects the forward error correction coding scheme to perform erasure coding on the multimedia data, encapsulates the encoded multimedia data with ER TP, and carries relevant information of the forward error correction coding scheme in the ER TP header information, and then sends the information to the receiving end;

The receiving end encapsulates the received ERRTP packet, and extracts the relevant information of the forward error correction coding scheme from the ERRTP header information, and then selects the forward error correction coding scheme to perform the erasure decoding and decoding according to the related information of the forward error correction coding scheme. Get multimedia data.

The unequal protection is reflected in that the sending end selects the forward error correction coding scheme according to the current network transmission status and/or the quality of service level of the multimedia data to be transmitted. First, the specific structure of ERRTP is introduced. The following is an example of the structure of the header information of the specific ERRTP. 4 is a block diagram showing the structure of an ERRTP header according to a first embodiment of the present invention. As can be seen from the figure, the version information field V takes a value of 3, indicating the ERRTP protocol, which is different from the traditional RTP protocol (V=2). The header information extension is finally accompanied by a related information field regarding the forward error correction codec scheme, and the example includes: a forward error correction coding type field, a forward error correction coding parameter field, a packet length field, and data. The number of packages field.

The forward error correction coding type field is used to indicate the erasure code type used by the forward error correction coding scheme, and may also be referred to as an FEC Type field, that is, an FEC coding type, which is 4 bits, and can represent 16 different FEC types. , from the actual application, is enough. The types defined here are actually large types, and will be further subdivided into various schemes, called subtypes. The large types in practical applications are, for example, 0010 for Tornado code and 0011 for RS code. This field can identify 16 different types of FEC codes. The query table (LUT, Look-Up Table), which needs to agree in advance on the correspondence between the FEC encoding type and the encoding type code, is called FECTypeLUT.

The forward error correction coding subtype field is used to indicate the related parameter setting of the forward error correction coding scheme. For each type of FEC coding, it is also necessary to determine the setting of various parameters to be specifically implemented, and this field is to clear specific parameters. The role. Since the resources in the ERRTP header information are limited, it is impossible to list specific parameters corresponding to various FEC encoding schemes, their rules, etc., and the first embodiment of the present invention indicates various alternative parameters by using the concept of subtypes. Set the plan. This field is also known as the FEC coded subtype field, FEC Subtype, which occupies 9 bits. This field mainly represents the subtypes further subdivided under the major types defined in the FECTypeLUT.

The packet length field is used to indicate the length of the data node after the forward error correction coding scheme performs erasure coding on the multimedia data, and is called a Data Length field, which is 11 bits. Since each packet length should be less than the Maximum Transport Unit (MTU), and the current cable channel MTU<1500 = 0x5DC bytes, the wireless channel MTI is 100 bytes, so this field is 11 bits enough to store the data packet. length.

The number of packets field, used to indicate the number of data nodes carried by the ERRTP packet, also known as the Packet Number field, which occupies 8 bits, for example, before a number of NALUs pass through After the error correction code is verified, the packet is encapsulated in multiple ERRTPs, and the number of data nodes carried in each ERRTP.

It can be seen that after these fields are available, the decoding end or the network node can verify the received data packet according to the FEC code type and the check type of the data packet given by the field, and recover the lost data packet.

It is to be noted that the sub-type FEC Subtype field mentioned above has a total of 9 bits for encoding a parameter setting scheme indicating various alternatives, and how to perform the coding indication in the first embodiment of the present invention is given below. technical details.

First, the receiving and receiving party needs to negotiate to determine the field indicating the relationship correspondence table. Before starting the transmission, the sender and the receiver negotiate to determine: for various types of FEC codes, the correspondence between the value of the FEC Subtype and the related parameter setting scheme of the FEC code indicated, and various alternatives. Specific parameter settings.

Then, the sender and the receiver both establish a correspondence table according to the negotiation result, and are configured to query the corresponding FEC coding type or FEC codec processing module according to the FEC Type and FEC Subtype fields;

In the process of transmitting and receiving, the transmitting end calls the corresponding erasure coding processing module to perform erasure coding, and the receiving end calls the corresponding erasure decoding processing module to perform erasure decoding.

In practical applications, subtype information actually indicates two aspects:

A. Generation rules for FEC coding (Generation Rule);

B. Protection strength / protection.

The so-called generation rule is a rule or algorithm (Algorithm) of how the data node is processed at the transmitting end to generate each check node. Of course, the opposite is done at the receiving end. If a packet loss occurs during the transmission, that is, some nodes are lost, the lost node can be recovered or partially recovered according to the generation rule. It can be seen that the generation rule is very important information, according to which both parties of the communication can work based on the FEC mechanism. Each of the FEC types listed in the FECTypeLUT has different generation rules; in each class, such as Tornado code, the following subclass generation rules are combined with specific generation parameters. . So for each subclass here, the claim rule will be combined with the build parameters. For example, for the Tornado code, the generation parameters include the following data: According to the total number of nodes, the total number of check nodes, the number of check node layers, the scaling ratio of the number of power saves between successive layers, and the association of node associations between successive two layers. Matrix, if there is an L-layer check node, then such an associative matrix has L or equivalent bipartite graphs representing the relationship between successive two-layer nodes. Generally, under the premise of large generation rules, the generation is performed. The parameters often determine the protection strength of the subtype. For example, Tornado code, in the various generation parameters given above, the total number of data nodes and the total number of test nodes can basically determine the protection ability to a large extent (of course, strictly speaking, to fully determine the protection capability, all the generation parameters are required. ). In the present invention, for each FEC large type, select some of the main parameters determining the protection capability (the decision is the most important) as the representative generation parameters (representative generation parameters) ₀ by using the representative generation parameters, it is possible to Subclasses are arranged in order of protection from weak to strong (ascending order). Thus creating a LUT is called FECSubTypeLUT.

Each large type specifically supports multiple subtypes below, and can have specific application and communication capabilities (CPU processing speed, memory, program complexity, etc.) and needs to be determined. If the communication environment changes a lot and the performance of the network fluctuates widely, then the subtypes that need to be supported are generally more, but less. This can be agreed upon by the communication parties through the capability negotiation process before the communication begins. Negotiation can be carried out through the current mainstream multimedia communication framework protocols such as H.323 or Session Initial Protocol (SIP).

Assume that for subclasses under a large class, if it is necessary to distinguish S subtypes (S ≤ 29-1), there are k representative generation parameters, denoted by pl, p2, ..., pk, then Table 2 gives An example of a correspondence, the superscript in the table indicates the FEC big type, and the subscript indicates which parameter.

Table 2 FEC Subtype and parameter setting scheme correspondence table

■ FEC Subtype FEC coding subtype (parameter setting)

000000000 FEC subtype 0 (p° p° ₂ , . . , ρ\)

000000001 FEC subtype p^ p' ₂ ,. .,p' _k )

000000010 FEC subtype 2 (p ² p ² ₂ , . . , p\) 000000011 FEC subtype 3 (p ³ p ³ ₂ , . . , p )

S (S^2 ⁹ -1) FEC subtype S (p ^s _h p ^s ₂ , . , , p ^s J For example, for Tornado code, the correspondence can be set to: 000000010 - ( 24, 20 ) (total number of data nodes =20, total number of check nodes = 4), 000000011 - (30, 20), ..., 111111111 - others.

For a subtype of FEC coding of a certain characteristic, a given set of generation rules combined with corresponding generation parameters corresponds to a unique coding scheme, that is, the only decision is how to generate a calibration node from the data node, and how to recover the lost node. A database can be created to store the generation parameters for each of the large types and subtypes. The generation rules themselves are implemented in hardware or software modules. Therefore, each type of macro corresponds to a FEC processing module at the transmitting end, which is responsible for generating a check node; at the receiving end, it also corresponds to an FEC processing module, which is responsible for restoring the node. However, for each large type of module, it is necessary to read the specific generation parameters of each seed type from the above generated parameter database, thereby performing processing. Therefore, both parties are based on

The information of the two information fields FEC Type and FEC Subtype determines which FEC processing module is called and reads those generation parameters.

Due to the development of multimedia communication technology, the H.264 video coding standard has gradually become the mainstream media coding format. Therefore, based on the first embodiment, the second embodiment of the present invention gives the NALU of H.264 with ERRTP. The specific steps of the data stream for FEC encoding and decoding, the flow of which is shown in Figure 5.

Step 501: The sender combines multiple (assumed S) H.264 NALUs into a unified group of coded transmissions, and first re-divides the S NALUs into blocks of equal length, which are assumed to be M, and the M are data. node.

In this step, the S NALUs of H.264 are grouped into one group; then the S NALUs are concatenated end-to-end, connected to form a large block, and then the large block is equally divided into M data blocks, wherein Each data block has a length of K bytes. Here, if the total number of bytes of the large block (set to TB) cannot be divisible by M, then the rounding operation should be performed so that the length of each data block is Ceiling (TBZM) bytes, and the Ceiling function indicates rounding, that is, Ceiling(x) is equal to no The smallest integer less than x, x is any real number. Then in some data blocks, the operation of zero padding may be used, so that the number of bytes is equal to Ceiling (TB/M).

Step 502: Perform FEC encoding on the M data nodes to obtain N check nodes. Using FEC code encoding for M data blocks to generate N check blocks, the generation process uses the method described above to determine which FEC processing module to call for the generation of the check block according to the FEC Type and FEC Subtype information.

Step 503: The sender encapsulates all data nodes and check node packets in an ERRTP packet for transmission. Figure 6 shows the structure of P + ER TP packages carrying M + N data nodes. Combined with the header information format of ERRTP given in Figure 4, in this example the fields should be set as follows:

Type field FEC Type = 0010, indicating the use of Tornado code;

The subtype field is selected by the sender according to the actual situation. For example, the value is FEC Subtype = 000000010, which means that the Tornado (24, 20) code is used, including 20 data nodes and 4 check nodes. The channel coding redundancy is 16.7%; the erasure code can completely recover the lost data packet when the packet loss rate is less than or equal to 3%;

Packet length Data-Length = K Bytes;

Packet Number Packet Number = (M+N)/P, which represents the number of data nodes carried in an ERRTP payload.

Step 504: After receiving the ERRTP packets, the receiving end encapsulates the data node and the check node. The receiving end starts with P packets and starts decoding and recovering every time a group of P packets is received. How many packets of a group are determined by mutual agreement.

Step 505: The receiving end performs forward error correction decoding on the data node according to the check node. Each time after receiving the data packet P+1, it starts to detect whether there is a packet loss in the P packets received before. If there is, the method described above is used to determine which FEC to call according to the FEC Type and FEC Subtype information. The processing module decodes and recovers or partially loses data.

Step 506, finally, after obtaining the complete data node, re-merging to obtain a large block, and dividing the S NALUs in the same manner as the transmitting end.

In practical applications, the above example uses the ERRTP-based anti-data packet loss algorithm, which can greatly improve the anti-data packet loss of the video code stream when the number of codewords is less than 17%. Force. Compared with the RTP payload header structure, only 4 bytes have been added, which shows that there is basically no effect on the transmission efficiency, and significant practical results have been achieved.

Another key technical point that has been mentioned above with respect to the present invention is the implementation of unequal protection. It is mainly embodied in two aspects. One is to select the appropriate codec scheme or parameters according to the multimedia data of different important levels, that is, to determine the aforementioned FEC coding type and subtype, and the other is to select according to the network conditions at different times. Corresponding to these two aspects, they are called mixed and alternate use of various FEC coding schemes. Hybrid refers to the simultaneous use of multiple FEC subtypes at the same time, mainly for protecting data of different importance. The so-called Alternation refers to the use at different times (different network conditions). Different FEC subtypes.

Therefore, in the third embodiment of the present invention, these two unequal protection mechanisms are given based on the first embodiment. For the H.264 NALU data stream, as mentioned above, its first byte reflects the importance of the data, so the sender can evaluate the QoS level according to the NRI field or Type field in the NALU header information, and then select the forward error correction. The coding scheme, that is, the FEC Type field and the FEC Subtype field are determined. For the network condition, the general network transmission has a corresponding network condition monitoring mechanism, and the transmitting end can learn the transmission report fed back by the receiving end according to these mechanisms, so as to evaluate the network transmission status, and then select the forward error correction coding scheme, that is, Determine the FEC Type field and the FEC Subtype field.

The H.264 code stream is transmitted or stored based on the NALU, which consists of NAL header information and NAL payload. In the NALU of H.264, different NALU types have different effects on decoding and restoring images. For example, a NRI of 0 means that a Slice or Slice data strip of a non-reference image in the NALU does not affect subsequent decoding; and a non-zero indicates that a sequence/image parameter set or a slice of the reference image is stored in the NALU or Slice data strips can seriously affect subsequent decoding.

Therefore, when packet protection is applied to the H.264 code stream, it can be based on NRI or

The value of Nal_unit_type divides the data of H.264 into two categories: one is relatively important image data (for example, Nal_ref-idc is equal to 1); the other is secondary image data (for example, Nal). — ref— idc is equal to 0). Then, the important image data is protected by the FECI code with high redundancy and strong anti-dropping ability; while the secondary image data can be used with less redundancy and weaker anti-loss capability. The FEC2 code is protected.

Through this unequal protection algorithm, the correct recovery of all kinds of important information in a high packet loss environment is ensured, and the image information that the FEC2 code still fails to recover adopts techniques such as error concealment and prevention of error diffusion. FEC1, FEC2 are just general representations, representing any two seed types. These two seed types can belong to the same large type, or they can belong to different large types.

Obviously, the above method can be extended to a more general case, and the data is divided into more classes according to the value of NAL_unit-type, such as five categories: the most important data, the second most important data, the general important data, the less important data, The least important data; can also be divided into 7 categories or more, then, can be protected with the same number of FEC subtypes, each type of data corresponds to a different subtype. As long as the protection ability is weak to strong, these subtypes do not necessarily belong to the same large type. The image information that has not been recovered after the protection of the most protected FEC code is protected by error concealment and error-proof diffusion.

Another case of unequal protection according to the present invention is the ability to select FECs of different protection capabilities depending on the real-time conditions of the network. The two sides of the communication are then notified by the header information of ERRTP so that they can correctly decode the data and recover the lost data. It is possible to divide the current situation in which the network is affected by the drop in transmission performance into several levels. For example, five levels: the most serious, the second most serious, the more serious, the less serious, the least serious; can also be divided into 7 or more, then, you can use the same number of FEC subtypes to protect, each level corresponds to a different Subtype. As long as the protection ability is weak to strong, these subtypes do not necessarily belong to the same large type. The image information that has not been recovered after the protection of the FEC code with the strongest protection is protected by error concealment and error-preventing. Network conditions can be monitored through various existing QoS monitoring methods.

More complex applications can also be provided in accordance with the present invention, if a total of T FEC schemes (different types/subtypes) are available (both terminals are supported by both parties). Deciding which FEC to use depends on both the importance of the data and the state of the network. Then you can use a two-dimensional LUT method, as shown in Table 3:

Table 3 Two-dimensional LUTs mixed and alternated with various FEC mechanisms

In the above table, the data importance level and the network status level are in ascending order. its

The subscript of the middle FEC is represented by a two-dimensional subscript. The fault-tolerant elastic mechanism FEC(i,j) in the table, 0<i < U, 0<j < V, may be any of the above T FEC schemes. .

The description of the embodiments of the above invention is exemplified by the FEC erasure code, especially the Tornado code, but can be applied to other similar fault-tolerant elastic mechanisms, especially the FEC coding scheme except the Tornado code, without affecting the essence of the present invention. range.

In another embodiment of the present invention, an improved Tornado erasure code is specifically employed. The improved Tornado erasure code generates only one layer of the check node for a group of data nodes, which can greatly reduce coding. Delay, to meet the needs of real-time communication.

In real-time video communication, packet protection using FEC codes introduces delays, the size of which is related to the size of the image data packets. The S NALUs are grouped into one group, and one NALU contains the code stream data of a Slice. If a frame of image is divided into a slice, the encoding end will have the delay of the S frame, and the decoding end will also have the delay of the S frame. The relationship between NALU and the number of data nodes is as follows:

NalSize _; = PackSize x DataNode ( 1 ) i=0

The sum of the S NALU length values in the equation is equal to the number of data nodes multiplied by the size of each node packet. It can be seen from equation (1) that when the value of S is limited, the value of PackSize xDataNode is also limited. In addition, the value of PackSize Ji cannot be too small due to the validity of IP network transmission, so the value of DataNode is limited. In real-time video communication over IP networks, the delay of one frame of image. , calculated as follows:

The formula ^ ¾ _c delay is introduced after the addition of FEC protection, and ^ Γ "are Η.264 codec for processing the network transmission delay and delay due to the rapid development of digital signal processing technology and IP networks, can be It is assumed that T _c _dec and Τ can satisfy the real-time requirement: Toodec <= T _ih , T _lram <= T _th , where = 1/^ , (3) is the decoding target frame rate in formula (3) (available value) 10Hz, 30Hz, etc.), and set a frame image into a slice, then the formula (2) can be changed to:

T _lolal <= S * T _lh + 2 * T _th = (S + 2r T _lh (4) From equations (4) and (1), the delay of the delay of one frame is basically determined by the value of S. Determine, and the DataNode greatly affects the value of s. Therefore, under the premise of ensuring the ability of video communication to resist packet loss, the delay introduced by FEC is minimized, and the QoS of real-time video communication is further ensured.

The present invention employs an improved Tornado code protection algorithm in the case where the ^{DataNode is} limited. The improved Tornado method does not use a multi-level even graph coding method, but uses only one layer of check node coding. Compared with the original Tornado coding method, the improved coding method greatly improves the flexibility of the algorithm. The number of data nodes and check nodes can be set arbitrarily, and the complexity of the codec algorithm is also reduced. It can be used for real-time video communication. Anti-packet loss.

In addition, the improved anti-data packet loss performance of the Tornado code is basically not reduced in the case where the data node is limited. The specific principle and detailed steps of the improved Tornado coding method are described in Chinese Patent Application No. 200510066146.7, entitled "A Data Transmission Protection Method Based on Erasure Code".

It will be understood by those skilled in the art that the specific parameter settings and values and other implementation details given in the above embodiments may be used in the specific application, and other feasible values or solutions may be used to achieve the object of the present invention without affecting the essence and scope.

Claims

-25- Claims

A real-time transmission method for a multimedia data network supporting fault-tolerant flexibility, comprising:

The transmitting end encapsulates the encoded multimedia data by using a fault-tolerant elastic real-time transmission protocol, and carries the forward error correction coding mode related information in the header information of the fault-tolerant elastic real-time transmission protocol data packet, and sends the information to the receiving end;

2. The method for real-time transmission of a multimedia data network supporting error tolerance resilience according to claim 1, wherein the forward error correction encoded multimedia data comprises a data node and a check node.

The method for real-time transmission of a multimedia data network supporting fault tolerance resilience according to claim 2, wherein the transmitting end selects forward error correction coding according to a current network transmission condition or/and a quality of service level of the multimedia data to be transmitted. The mode, where the service ^: the quantity level is determined according to the relative importance of the data.

The real-time transmission method for supporting a fault-tolerant elastic multimedia data network according to claim 3, wherein the packet header information of the fault-tolerant elastic real-time transmission protocol includes: a forward error correction coding type field, which is used to indicate that a forward error correction code type; a forward error correction coding subtype field, configured to indicate a related parameter setting of the forward error correction coding mode;

a packet length field, configured to perform a forward error correction code on the multimedia data The length of the resulting node;

The real-time transmission method of the multimedia data network supporting the fault-tolerant elasticity according to claim 4, wherein when the multimedia data is an H.264 network abstraction layer unit, the transmitting end shall have at least one of the H. 264, the network abstraction layer unit is divided into at least one data node of equal length, and then subjected to forward error correction coding to obtain at least one check node; the sender encapsulates the data node and the calibration node in a packet Transmitting in at least one of the fault tolerant elastic real-time transport protocol packets;

If the data node is lost during the transmission, the receiving end performs forward error correction decoding on the data node according to the verification node, and divides the H.264 network abstract layer unit.

6. The method for real-time transmission of a multimedia data network supporting fault tolerance resilience according to claim 5, wherein before starting the transmission, the method comprises:

The real-time transmission method of the multimedia data network supporting the fault-tolerant elasticity according to claim 6, wherein the transmitting end and the receiving end are both established according to the indication correspondence relationship of the forward error correction coding subtype field Corresponding relationship table, configured to query, according to the forward error correction coding type field and the forward error correction coding subtype field, a forward error correction coding or a forward error correction decoding processing module;

The transmitting end invokes a corresponding forward error correction coding processing module to perform forward error correction coding; the receiving end invokes a corresponding forward error correction decoding processing module to perform forward error correction decoding.

The real-time transmission method of the multimedia data network supporting the fault-tolerant elasticity according to claim 7, wherein the transmitting end is based on a network abstraction layer reference identifier field in the header information of the H.264 network abstraction layer unit or / Correspond to the network abstraction layer unit type field evaluation The relative importance of the data, determining the quality of service level, selecting a corresponding forward error correction coding mode, and determining the forward error correction coding type field and the forward error correction coding subtype field.

The real-time transmission method of the multimedia data network supporting the fault-tolerant elasticity according to claim 7, wherein the transmitting end evaluates the network transmission status according to the transmission report fed back by the receiving end, and further selects the forward direction. The error correction coding mode determines the forward error correction coding type field and the forward error correction coding subtype field.

10. The method for real-time transmission of a multimedia data network supporting error tolerance resilience according to claim 8 or 9, wherein:

The forward error correction coding type field is located after the contribution source identifier list;

The forward error correction coding subtype field is located after the forward error correction coding type field;

11. The method for real-time transmission of a multimedia data network supporting error tolerance resilience according to claim 8 or 9, wherein the forward error correction coding method uses an improved "Tornado" erasure code;

The improved "Tornado" erasure code generates only one layer of the check node for a set of said data nodes.