CN1859580A

CN1859580A - Multimedia data network realtime transfer method for supporting error elasticity

Info

Publication number: CN1859580A
Application number: CN 200510113940
Authority: CN
Inventors: 罗忠; 宋彬
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2005-10-17
Filing date: 2005-10-17
Publication date: 2006-11-08
Anticipated expiration: 2025-10-17
Also published as: CN100450187C; WO2007045141A1

Abstract

The present invention discloses error resilient multimedia data network real time transmitting method relating to multimedia data real time transmitting technology. It contains first of all on current RTP protocol basic, providing ERRTP protocol capable of carrying forward error correction coding scheme relevant information to realize media data real time transmitting, to make multimedia data on ERRTP transmitting and simultaneously marking relevant forward error correction coding scheme messages, thereby merging error elasticity system in transport layer; secondly transmitting end capable of according to current network status multimedia data importance grade etc factor to select adopting different kinds of spare forward error correction coding scheme, thereby reaching inequality protection and cascade protection target, to realize protective capability and transmission efficiency equalization.

Description

Support the multimedia data network realtime transfer method of error elasticity

Technical field

The present invention relates to the real-time tranmission techniques of multi-medium data, particularly support the multi-medium data method for real-time transmitting of error elasticity.

Background technology

Along with the develop rapidly of computer internet (Internet) and mobile communications network, the application of stream media technology more and more widely, broadcasting, film are played to remote teaching and online news website etc. and have all used stream media technology from network.Current online transmission video, audio frequency mainly contain download (Download) and streaming transmits (Streaming) dual mode.It is to transmit the video/audio signal continuously that streaming transmits, when Streaming Media remainder when client computer is play continues on the backstage to download.Streaming transmits has progressive streaming to transmit (Progressive Streaming) and real-time streaming transmission (Realtime Streaming) dual mode.It is real-time transmission that real-time streaming transmits, and is particularly suitable for live event, and real-time streaming transmits and must coupling connect bandwidth, this means that picture quality can reduce variation because of network speed, to reduce the demand to transmission bandwidth.The notion of " in real time " is meant that the payment of data in an application must keep the precise time relation with the generation of data.

Especially along with 3-G (Generation Three mobile communication system) (3rd Generation, be called for short " 3G ") appearance and generally based on Internet protocol (Internet Protocol, abbreviation " IP ") network develops rapidly, and video communication just progressively becomes one of main business of communication.And both sides or multi-party video communication service, as video telephone, video conference, mobile multimedia terminal service etc., more transmission and the service quality to multimedia data stream proposes harsh requirement.It is better not only to require network to transmit real-time, and equivalence also require video data compaction coding efficient higher.

Demands status in view of media communication, (the InternationalTelecommunication Union Telecommunication Standardization Sector of ITU-T, be called for short " ITU-T ") after having formulated H.261, H.263, H.263+ waiting video compression standard, formally issued H.264 standard in 2003.This is ITU-T and (the InternationalStandardization Organization of International Standards Organization, abbreviation " ISO ") Motion Picture Experts Group (Moving PictureExperts Group is called for short " MPEG ") unites the adaptation new stage network media transmission of formulation and the efficient compressed encoding standard of communication requirement together.It also is the main contents of MPEG-4 standard the 10th part simultaneously.

Formulating H.264, the purpose of standard is to improve more effectively video coding efficient and its suitability to network.In fact owing to its superiority, H.264 the video compression coding standard has become the mainstream standard in the current multimedia communication soon gradually.A large amount of employing H.264 multimedia realtime communication product (as video conferencing, video telephone, 3G mobile communication terminal) and network flow-medium product is successively come out, and whether supports H.264 to have become the key factor of decision product competitiveness in this market segment.Can predict that along with H.264 formal promulgation be extensive use of, the multimedia communication of IP based network and 3G, back 3G wireless network must enter a new stage of developing by leaps and bounds.

Multimedia communication not only requires media compression code efficiency height as previously mentioned, and the real-time that requires network to transmit.Media stream transmission at present all is to adopt real time transport protocol (Real-timeTransport Protocol is called for short " RTP ") and control protocol (Real-time Transport ControlProtocol is called for short " RTCP ") thereof basically.RTP is a transportation protocol going up multimedia data stream at Internet, by the Internet engineering duty group (Internet Engineering Task Force is called for short " IETF ") issue.RTP be defined in one to one or the transmission situation of one-to-many under work, its objective is provides temporal information and realizes that stream is synchronously.The typical case of RTP uses and is based upon User Datagram Protocol (UserDatagram Protocol, be called for short " UDP ") on, but also control protocol (TransportControl Protocol can transmitted, be called for short " TCP ") or asynchronous transfer mode (Asynchronous Transfer Mode is called for short " ATM ") wait on other agreements and work.

RTP itself only guarantees the transmission of real time data, can not provide reliable transfer mechanism for transfer data packets in order, and flow control or congested control are not provided yet, and it relies on RTCP that these services are provided.RTCP is in charge of delivery quality exchange of control information between the current application process.During the RTP session, each participant periodically transmits the RTCP bag, contains the quantity of data packets that has sent, the statistics of losing such as quantity of data packets in the bag, therefore, server can utilize these information dynamically to change transfer rate, even changes PT Payload Type.RTP and RTCP are used, and can make the transmission efficiency optimization with effective feedback and minimum expense, so be fit to transmit online real time data.

And H.264 multi-medium data transmits on IP network, the unexceptional Real-time Transport Protocol that also is based on UDP and its upper strata.RTP itself structurally can both be suitable for for different media data type, but different upper-layer protocol or media compression coding standard in multimedia communication are (as H.261, H.263, MPEG-1/-2/-4, MP3 etc.), IETF can formulate the authority file at RTP payload (Payload) packaging method of this agreement, the method for the big bag of specified in more detail RTP encapsulation, for this concrete agreement through optimization.Same, be H.264 Video of RFC 3984:RTPPayload Format for for H.264 also having corresponding ietf standard.Be the main standard that on IP network, transmits of video code flow H.264 before this standard mesh, use very extensive.In field of video communication, each mainstream vendor's product all is based on RFC 3984, also is only H.264/RTP load mode at present.

In fact, H.264 be H.264 to define a new aspect, be called network abstract layer (Network Abstract Layer is called for short " NAL ") with other the different key place of video compression coding agreement in the past.H.264 in order to increase its video coding layer (Video Coding Layer, be called for short " VCL ") with the separating and independence of the network transportation protocol layer of following mask body, bring bigger application flexibility, defined this new aspect of NAL, this layer, all do not have in H.263/H.263+/H.263++ such as H.261 in the early stage video compression coding agreement of ITU-T.Yet, how in NAL and Real-time Transport Protocol carrying collaborative work at H.264 higher, the better scheme of advantage design efficiency, make RTP better for load-carrying properties H.264, be very worth research and the problem that very big using value is arranged.

The method of RTP that the RFC3984 standard is proposed carrying NAL layer data H.264 is present main flow transfer approach, and this scheme is encapsulated in the NAL layer data in the RTP payload and carries on the basis of Real-time Transport Protocol (RFC 3550).The NAL layer is between VCL and RTP, and regulation will be divided into a series of NAL data cell (NAL Units is called for short " NALU ") to rule and the structure of video code flow according to definition.In RFC3984, defined the encapsulation format of RTP payload for NALU.Simply introduce the method for packing of NALU in the frame format of RTP and the prior art below successively.

The main purpose of RTP design is real-time multimedia meeting and continuous data storage, mutual distributed emulation, control and measurement application etc.RTP is carried on the udp protocol usually, to utilize its multiplexed and function verification.If bottom provides the multiple spot distribution, RTP supports multiaddress to transmit.The function that RTP provides comprises: load type discriminating, sequence numbering, timestamp and transmitting supervisory.

The packet format of RTP is as follows: RTP header most basic option takies 12 bytes (minimum), and the header of IP agreement and udp protocol takies 20 bytes and 8 bytes respectively, therefore RTP seals and is contained in the UDP bag and is encapsulated in the IP bag again, and it is the 12+8+20=40 byte that total header takies byte number.The detailed structure of the header of RTP bag as shown in Figure 1.

Shown in Fig. 1 from front to back the RTP header be followed successively by: the 1st byte (byte 0) is some fields about header structure itself, the 2nd byte (byte 1) is the definition payload type, the 3rd, 4 bytes (byte 2,3) be bag sequence number (Sequence Number), the 5-8 byte is timestamp (timestamp), the 9-12 byte is to contribute source identifier (Synchronous Source Identifier synchronously, be called for short " SSRCID "), be contribution source identifier (Contributing Source Identifiers at last, abbreviation " CSRCIDs ") tabulation, its number is uncertain.Notice, the byte 0 of the 1st byte in this paper describes for marking, the rest may be inferred afterwards.

Wherein preceding 12 bytes appear in all dissimilar RTP packets, and other data in the header just have such as contributing the source identifier sign to have only when blender inserts.Situation when therefore CSRC generally is used to exist medium to mix, such as in Multi-Party Conference, audio frequency needs to mix, and video also can provide the function of many pictures in this way.And Synchronization Source SSRC is exactly the sign of institute's carrying media stream in fact.

The concrete meaning and the full name of above-mentioned each field are described below respectively:

The V field is version (Version) information, account for 2 bits (bits), the version that adopts is 2 at present, therefore put V=2, and other values such as V=1 represent RTP version more early, V=0 represents the most original RTP predecessor, promptly adopt in voice IP (VOIP) communication system of using on the Mbone network in early days, be evolved into RTP afterwards, V=3 is then still undefined, so the present invention can use.；

The P field accounts for 1 bit for filling sign (Padding), if P set represents that then the packet end comprises one or more byte of paddings (Padding), fills a part that does not belong to payload;

X field is expansion sign bit (Extension), account for 1 bit, if X set, then the RTP head must expand with the head of a variable length at last (if the CSRC tabulation is arranged, the head expansion will be followed thereafter), mainly be to be preserved for the not enough situation of following information field of some applied environment, the length field that this header expansion comprises one 16 bit is counted the word what 32 bit long are arranged in the expansion, preceding 16 bits of head expansion are that a left side is open, so that distinguishing identifier symbol and parameter, the form of this 16 bit is by concrete aspect normalized definition, and the formal definition of this expansion has a detailed description in RFC3550 5.3.1 joint, no longer provides as space is limited herein;

The CC field is contribution source number (CSRC Count), accounts for 4 bit s, indicates the number of the rearmost CSRC identifier of header, and the recipient can determine the CSRCIDs list length of header back according to the CC field;

The M field is sign bit (Marker), account for 1 bit, the explanation of this sign bit defines in specific aspect (Profile), it allows to identify the critical event in the data packet stream, an aspect can define additional sign bit or stipulate not identify bit, here so-called aspect just is meant concrete applied environment setting, is specifically reached an agreement on by communicating pair, is not subjected to the qualification of agreement;

The PT field is load type (Payload Type is called for short " PT "), and totally 7 bit s identify the form of RTP load and determine his explanation in application program; Flag bit and load type totally one byte are carried the aspect provisioning information, this byte may be redefined to adapt to different demands by concrete aspect, in concrete the application, can define so-called profile, be exactly one group of static state (being that communicating pair is appointed in advance) corresponding relation in fact, the value that the PT bit is different is mapped with different media formats.Can certainly carry out relation between dynamic negotiation definition PT value and the media formats by the signaling outside the RTP.In a RTP session (Session), PT can be changed in the RTP source.

Field then is exactly sequence number totally 16 bits, RTP packet of every transmission, and this sequence number value adds one, and the recipient can detect data-bag lost and recovery data packets order with it like this, and once the sequence number initial value in the communication can be given at random, do not influence communication.

Timestamp accounts for 32 bits, and it has reflected the sampling time of first byte in the RTP packet, and the sampling time here must derive from the clock of a dull linear growth, and the recipient adjusts the media play time according to it or carries out synchronously.

Synchronisation source SSRC ID accounts for 32 bits, and its occurrence can be selected at random, but will guarantee the uniqueness in the same RTP session, can source of media of unique identification, if a source has changed the transfer address, source, must select a new SSRC identifier.

CSRC tabulation in contribution source can be the 0-15 item as required, and every accounts for 32 bit s, and the length of this tabulation is that the number of CSRC ID is just in time marked by 4 bits of CC field.In fact, the CSRC identifier that is used to identify certain source of media is consistent with the SSRC identifier in its corresponding contribution source, only in role's difference of different recipients, and is changed to SSRC or CSRC.In multi-party communication, CSRC ID is inserted by blender.

Carrying under the situation of video H.264, RTP is packaged into RTP bag stream to NALU encapsulation H.264.In RFC 3984 files, mainly defined NALU, and provided the H.264 encapsulation packing form of layer NAL data in RTP based on this.The RTP encapsulation format of this NALU as shown in Figure 2.

Provide the encapsulating structure of a NALU in the payload of RTP among Fig. 2, first byte of front is the NALU header, be the data content of NALU afterwards, in the payload that is filled into the RTP bag that a plurality of NALU are end to end, in the end also have optional RTP to fill, this is the content of RTP packet format regulation, is in order to make the length of RTP bag meet certain particular requirement (such as reaching regular length), optionally generally all zero fillings of RTP padding data.

The NALU header i.e. the 1st byte, is also referred to as eight bit groups (Octet), and it has three fields, and meaning and full name are described below respectively:

The F Field Definition is for forbidding bit (forbidden_zero_ bit), account for 1 bit, be used to identify situations such as grammer mistake, if syntax clash arranged then be changed to 1, when having the bit mistake in Network Recognition this element, it can be made as 1,, be mainly used in and adapt to different types of network environment (environment that combines such as wire and wireless) so that the recipient loses this unit;

The NRI Field Definition is NAL reference identification (nal_ref_idc), account for 2 bits, be used to indicate the significance level of NALU data, its value is that the content of 00 expression NALU is not used in the reference picture of rebuilding inter prediction, but not 00 current NALU of expression is band (slice) or sequence parameter set (the Sequence Parameter Set that belongs to reference frame, abbreviation " SPS "), picture parameter set (PictureParameter Set, be called for short " PPS ") wait significant data, it is important more that this is worth the current NAL of big more expression;

The type field is defined as NALU type (Nal_unit_type), and totally 5 bits can have the type of 32 kinds of NALU, and the corresponding relation of its value and particular type provides in table 1 in detail.

The type field value and type mapping table in the table 1 NALU header

The Type value	The type of NALU content
The Type value	The type of NALU content	0	Do not specify
1	The coding slice of non-IDR image	0	Do not specify
1	The coding slice of non-IDR image	2	Coding slice data are divided A
3	Coding slice data are divided B	2	Coding slice data are divided A
3	Coding slice data are divided B	4	Coding slice data are divided C
5	Coding slice in the IDR image	4	Coding slice data are divided C
5	Coding slice in the IDR image	6	SEI (supplemental enhancement information)
7	SPS (sequence parameter set)	6	SEI (supplemental enhancement information)
7	SPS (sequence parameter set)	8	PPS (picture parameter set)
9	The access unit delimiter	8	PPS (picture parameter set)
9	The access unit delimiter	10	EOS
11	Code stream finishes	10	EOS
11	Code stream finishes	12	Padding data
13-23	Keep	12	Padding data
13-23	Keep	24-31	Do not specify

As seen, the information spinner that provides in the byte of the header of NALU will comprise validity, the importance rate of NALU, can determine RTP institute data carried by data importance according to these information.

After having understood transfer structure H.264/RTP, with the closely-related error resilience mechanism that also has multi-media network to transmit of content of the present invention.Below simple error elasticity and the background of related that transmits relevant for video network of introducing.

H.264 video is the main agreement of following multimedia communication, and the network that following multimedia communication is used mainly is to be the packet-switched network and the wireless network of representative with IP.This two big class network all can't provide good service quality (Quality of Service, be called for short " QoS ") to guarantee, so video transmits on network and will inevitably be subjected to various transmission mistakes and the influence of packet loss, thereby makes the communication quality reduction.Topmost problem here is that IP network realizes that " doing the best " (best effort) transmits, and can not guarantee to transmit the QoS of vision signal.Particularly to the H.264 code stream through efficient compressed encoding, problem is more outstanding.Transmission as possible on the IP network can not guarantee the QoS of real-time video communication, is in particular in three aspects: data-bag lost, time delay and delay variation.Wherein, data-bag lost is to recovering the quality influence maximum of video, because H.264 Coding Compression Algorithm is used estimating motion and motion compensation technique, in case there is data-bag lost to exist, not only influence current decoded picture, and can influence the subsequent decoding image, be i.e. error code diffusion.The influence that the error code diffusion couple recovers video quality is very big, has only in conjunction with coding side and decoding end and unites anti-error code, could avoid the error code diffusion fully.

Error elasticity (Error Resilience) be meant that transfer mechanism has that guard against false takes place or the ability that after mistake takes place, can correct with certain ability (mistake intensity can be corrected within the specific limits fully; Surpass certain limit, can only partly correct).In the multimedia communication environment of extensive (we can say omnipresent) in future, whether a kind of video transfer mechanism has error elasticity will be very crucial.

There is multiple error resilience mechanism, such as forward error correction (Forward Error Correction, abbreviation " FEC "), automatic re-send request may (Automatic Retransmission Request, be called for short " ARQ "), error concealment (Error Concealment), message source and channel combined coding (Joint Source-ChannelCoding is called for short " JSCC "), interweave (Interleaving) and eliminate error code diffusion etc.Transmit on packet networks for video H.264, FEC is a kind of very practical technology, and effect is fine.This method mainly adopts multiple error correction coding to encode for the data that will protect, and essence is to form data redundancy, thereby increases the wrong ability of resisting.

Cardinal error is the packet loss mistake on packet networks, and this mistake is called deletion error (Erasure Error) in Coding Theory.Error correction coding at deletion error is that a big class is called correcting and eleting codes (Erasure Codes).So-called correcting and eleting codes is exactly that the data code flow order is divided into the identical unit one by one (Unit) of size piecemeal, also is called back end (Data Nodes), for representing conveniently, supposes total n data node.Calculate for these back end according to certain mathematical operation rule then and produce check-node (Parity Nodes or Check Nodes); in order to strengthen protective capability; can also continue to produce second layer check-node for these check-nodes according to identical or different mathematical operation rule computings; and the like; can generate the 3rd layer; the 4th layer, until N layer check-node.

In general, if relate to the multilayer check-node, the interstitial content on every layer successively decreases according to certain rule (modal is the geometric ratio rule) with respect to last layer, so just becomes one successively to pass the multilayer node structure that contracts.Can be expressed as a pyramid that turn 90 degrees visually to the right.Wherein, Far Left is the back end layer, and arranging to the right is the ground floor check-node successively, second layer check-node ..., N layer check-node.

Wherein a class correcting and eleting codes has a kind of very important character, and the time complexity that promptly handle to need is to have linear relationship with the data section n that counts, and therefore is called linear session characteristic (linear-time).And much other correcting and eleting codes will be much higher such as the time complexity of famous Reed-Solomon sign indicating number needs, be n*log2n*log (logn) order of magnitude.Therefore, it is far better to have its purposes in real time communication of correcting and eleting codes of linear session.

Tornado correcting and eleting codes (hereinafter all being called for short the Tornado sign indicating number) is a kind of novel correcting and eleting codes that occurs before and after 1998.The Tornado code structure is simple; Computing is efficient, because it has linear session; Protective capability is strong.In actual applications, obtained good effect.Obtained at present to use comparatively widely.Dynamic according to up-to-date ITU-T, SG16 is wherein considering to carry out standardized possibility for error control coding class (Error Control Codes) technology at present, mainly is to transmit at the video/audio network to protect.Tornado sign indicating number and a plurality of mutation thereof are likely important technology wherein.

In the Tornado sign indicating number, successively produce a plurality of check-node layers from back end.Check-node and back end all send to receiving terminal by transmitting terminal by network.If in the network transport process, part of nodes has been lost, because upper layer node has been participated in the generation of lower level node, therefore the information of upper layer node has been included in lower level node and more in the lower level node, therefore lose node information can by working majority purpose lower level node or more lower level node recover fully.If each node is a bag, the bag of then losing can be recovered fully by other bag that correctly receives.If the back end number is n, the check-node number of generation is 1.The code check and the redundancy rate that then define correcting and eleting codes are respectively: r=n/ (n+1), 1-r=1/ (n+1); Under other condition same case (protective capability, the delay that causes etc.), code check high more (inevitably, redundancy rate is low more), then the efficient of correcting and eleting codes is high more.

Fig. 3 shows the relation between a kind of typical Tornado code data node and each layer check-node.Line among the figure between the node is called the limit, and the left node on expression limit participates in calculating the right side node, is a kind of logical relation of multi-to-multi between the two-layer node of visible front and back.In the Tornado sign indicating number production process the computational methods of normal employing be XOR because XOR has restore funcitons very easily, after any one node loss, all can recover by all the other nodes.Owing to the ratio difference that contracts of passing of last one deck check-node, the error correction coding scheme of therefore general employing routine is calculated, such as the Reed-Solomon sign indicating number.

In fact, the scope of correcting and eleting codes is very big, and the Tornado sign indicating number is just wherein a kind of, also has in addition such as RS (Reed-Solomon) sign indicating number, loe-density parity-check code (Low Density Parity Codes is called for short " LDPC ") etc.

Important performance index of correcting and eleting codes are exactly its error correcting capability (perhaps being called protective capability); directly be presented as the maximum packet loss quantity (under the total precursor of certain bag) that to correct the packet loss mistake fully and be allowed; perhaps work as packet loss and be higher than under this maximum allowable quantity condition, can correctly correct the percentage of bag.In general, under other condition same cases, protective capability is high more, and redundancy rate is high more.

Protective capability is not only applicable to correcting and eleting codes, and in a wider context, all FEC codings can be measured with protective capability.In video data, some data relative importance height is such as the structural parameters of the structural parameters of video sequence, image, header etc.; The importance of other data is low relatively, such as picture material data etc.When using FEC to protect, for the stronger coding of important relatively The data protective capability; And for the more weak coding of unessential relatively The data protective capability.Can between protective capability and efficient, reach balance like this.Can not emphasize protective capability simply,, sacrifice efficient because can cause very high redundancy like this.This method of carrying out the FEC protection of different protective capabilities according to the data relative importance is called unequal loss protection (Unequal Protection is called for short " UEP ").By unequal loss protection, realize that easily the QoS of video communication services guarantees.

The Real-time Transport Protocol that transmits the video multimedia data at present is that error elasticity is not provided, and lean on higher application layer to provide.H.264, prior art is for waiting during the video data network transmits, and generally is to use correcting and eleting codes protection to realize error elasticity.H.264 to be example, the measure that the prior art scheme is taked is:

Transmitting terminal directly uses certain correcting and eleting codes in H.264 NALU aspect for the NALU data cell, then result's (comprising back end and check-node) directly is encapsulated in the RTP packet, transmits then.

Receiving terminal is after receiving the RTP packet, go encapsulation, extract back end and check-node, if generation packet loss, it is certain or some RTP data-bag losts, so according to having encapsulated which back end or check-node in these lost package, can judge whether to utilize the back end that correctly receives and check-node recovers fully or part is recovered the node lost, and carry out recovery operation.

Certainly also have and adopt other error resilience mechanisms realizations except correcting and eleting codes, but correcting and eleting codes can provide error resilience mechanism the most efficiently for Data Protection H.264.

As seen, prior art is to delete coding then in the RTP transmission at high level by multi-medium datas such as NALU are entangled, and entangles accordingly at receiving terminal and deletes decoding.It should be noted that H.323/H.245 receiving-transmitting sides generally is to be provided with etc. by the parameter that the Signalling exchange before transmitting comes consult to decide to adopt what forward error correction coding scheme and this scheme to adopt, waits protocol channel confession both sides to hold consultation such as adopting.

In actual applications, there is following problem in such scheme: error resilience mechanism is all realized at the high level of RTP in the prior art scheme, wherein both sides consult or inform that the correcting and eleting codes type that adopted and parameter thereof need realize by other logic channel when being provided with, have a strong impact on the multimedia transmission efficiency, consume network bandwidth resources;

And transmit aspect for RTP, error resilience mechanism is transparent, therefore the RTP layer can not be learnt the structure of the encoded multimedia data that the FEC coding and decoding scheme produces, thereby can't encapsulate targetedly, can't simplify and transmit the level framework, the lengthening network transmits time-delay, transfer equipment becomes complicated;

For transmitting-receiving two-end after consult determining the FEC encoding scheme; promptly transmit multi-medium data according to this scheme always; data and different network delivery statuss constantly for different importance can't realize unequal loss protection mechanism, also can't realize that QoS guarantees by error resilience mechanism.

Cause the main cause of this situation to be, prior art realizes error resilience mechanisms such as FEC at high level, do not use Real-time Transport Protocol and encapsulation thereof, therefore receiving-transmitting sides need be set up logic channel in addition or utilize concrete application layer protocol such as some agreement in the protocol architecture H.323 H.245, consult or notify the FEC that is adopted type of coding, information such as structural parameters; Do not relate to the error elasticity correlative detail, do not formulate the RTP packet and how to encapsulate back end and the check-node that protection produces through FEC at the RTP layer; Do not select the FEC coding and decoding scheme yet, lack to provide and carry out the mechanism of different protective capability FEC protections, promptly can't realize unequal loss protection for having different relative importance data according to the importance of network condition and multi-medium data.

Summary of the invention

In view of this, main purpose of the present invention is to provide a kind of multimedia data network realtime transfer method of supporting error elasticity, and the error resilience mechanism that makes multi-medium data transmit in real time can be achieved in the transportation protocol aspect; Further aim of the present invention is unequal loss protection mechanism and the cascade protection mechanism that realizes at different pieces of information and network condition.

For achieving the above object, the invention provides a kind of multimedia data network realtime transfer method of supporting error elasticity, comprise following steps:

The A transmitting terminal selects the forward error correction coding scheme that described multi-medium data is carried out forward error correction coding;

Multi-medium data after the described transmitting terminal of B is encoded with the encapsulation of error elasticity real time transport protocol, the multi-medium data that carried out forward error correction coding is divided into back end and check-node two classes, and in described error elasticity real time transport protocol header packet information, carry described forward error correction coding scheme relevant information, send to receiving terminal then;

The described receiving terminal of C goes the error elasticity real time transport protocol bag of receiving to encapsulation, and extracts described forward error correction coding scheme relevant information from described error elasticity real time transport protocol header packet information;

If the losing of error elasticity real time transport protocol bag of back end correspondence taken place in D in transport process, so described receiving terminal is according to described forward error correction coding scheme relevant information, select described decoding FEC scheme to carry out decoding FEC, recover or partly recover the described multi-medium data of losing.

Wherein, in described steps A, described transmitting terminal is selected described forward error correction coding scheme according to one of following factor at least:

Current network transmission status and service of multimedia data credit rating to be sent, and the latter is decided by the relative importance of different pieces of information.

In this external described method, comprise in the described error elasticity real time transport protocol header packet information:

The forward error correction coding type field is used to the forward error correction code type of indicating described forward error correction coding scheme to adopt;

The forward error correction coding sub-type field is used to indicate the relative parameters setting of described forward error correction coding scheme;

The data packet length field is used to indicate the length of the node that described forward error correction coding scheme obtains after described multi-medium data is entangled forward error correction;

The number of data packets field is used to indicate the number of the described node that this error elasticity real time transport protocol bag carried.

In this external described method, when described multi-medium data during for network abstraction layer unit H.264,

In described steps A, described transmitting terminal is divided at least one isometric back end with at least one described H.264 network abstraction layer unit, then it is carried out forward error correction coding, obtains at least one check-node;

In described step B, described transmitting terminal sends described back end and described check-node packet encapsulation at least one described error elasticity real time transport protocol bag;

In described step C, described receiving terminal goes encapsulation to obtain described back end and described check-node after receiving described error elasticity real time transport protocol bag;

In described step D, if the back end in the transport process has taken place to be lost, then described receiving terminal carries out recovering based on the recovery or the part of decoding FEC to described obliterated data node according to described check-node, and division obtains described H.264 network abstraction layer unit.

In this external described method, before beginning transmission, also comprise step,

Described transmitting terminal and described receiving terminal are consulted to determine: for various described forward error correction code types, the corresponding relation of the relative parameters setting of this kind forward error correction that the value of described wrong forward error correction sub-type field is indicated with it.

In this external described method, described transmitting terminal and described receiving terminal are all set up mapping table according to the indication corresponding relation of described forward error correction coding sub-type field, are used for inquiring about pairing forward error correction coding or decoding FEC processing module according to described forward error correction coding type field and described forward error correction coding sub-type field;

In described steps A, described transmitting terminal calls corresponding forward error correction coding processing module and carries out forward error correction coding;

In described step D, described receiving terminal calls corresponding decoding FEC processing module and carries out decoding FEC.

In this external described method, when described multi-medium data is described H.264 network abstraction layer unit,

In described steps A, described transmitting terminal is assessed the relative importance of corresponding data according to network abstract layer reference identification field in the header of described H.264 network abstraction layer unit and any one or both in the network abstraction layer unit type field, thereby determine described service quality rating, and then select described forward error correction coding scheme, determine described forward error correction coding type field and forward error correction coding sub-type field.

In this external described method, in described steps A, described transmitting terminal is estimated described network transmission status according to the transmission report of described receiving terminal feedback, and then selects described forward error correction coding scheme, determines described forward error correction coding type field and forward error correction coding sub-type field.

In this external described method, the version information field value in the described error elasticity real time transport protocol header packet information is binary one 1 (decimal system 3), to be different from real time transport protocol;

Described forward error correction coding type field is positioned at after the tabulation of contribution source identifier, accounts for 4 bits;

Described forward error correction coding sub-type field is positioned at after the described forward error correction coding type field, accounts for 9 bits;

Described data packet length field is positioned at after the described forward error correction coding sub-type field, accounts for 11 bits;

Described number of data packets field is positioned at after the described data packet length field, accounts for 8 bits.

In this external described method, described forward error correction coding scheme comprises improved " Tornado " correcting and eleting codes;

Described improved " Tornado " correcting and eleting codes only generates the described check-node of one deck for one group of described back end.

By relatively finding, the main distinction of technical scheme of the present invention and prior art is, at first the present invention provides the ERRTP transport layer that can carry forward error correction coding scheme relevant information encapsulation format on existing RTP basis, make its corresponding forward error correction coding scheme information of same tense marker that multi-medium data transmits on ERRTP, thereby error resilience mechanism is incorporated transport layer;

Secondly can also select to adopt various standby forward error correction coding schemes according to factors such as current network conditions and multi-medium data importance rates at transmitting terminal, thereby reach unequal loss protection and purpose cascade protection, realize the equilibrium of protective capability and transmission efficiency;

At last for the correcting and eleting codes protection scheme of H.264 NALU, provided back end and check-node generation, transmission, encapsulate and go method for packing.

Difference on this technical scheme has brought comparatively significantly beneficial effect, promptly simplifies the error elasticity transfer structure greatly at transport layer realization error resilience mechanism, has saved the network transmission bandwidth; The realization of unequal loss protection has reached the equilibrium of protective capability and transmission efficiency, is convenient to the realization of the QoS assurance of multimedia transmission; H.264 the realization of the concrete transfer scheme of data can improve performance and the user experience used on IP network such as video conferencing, video telephone based on multimedia communication product H.264 greatly, improves product competitiveness, brings remarkable economic efficiency.

Description of drawings

Fig. 1 is the header structural representation of RTP packet;

Fig. 2 is the encapsulation format schematic diagram of RTP bag payload to the NALU data;

Fig. 3 is a Tornado correcting and eleting codes principle schematic;

Fig. 4 is the EERTP header packet information structural representation of first embodiment of the invention;

Fig. 5 is a H.264 multimedia data transmitting method flow chart second embodiment of the invention;

Fig. 6 is a H.264NALU division encoding-decoding process schematic diagram second embodiment of the invention.

Embodiment

For making the purpose, technical solutions and advantages of the present invention clearer, the present invention is described in further detail below in conjunction with accompanying drawing.

Problems at the prior art existence, the present invention proposes a kind of Real-time Transport Protocol of improved support error elasticity, be intended to error resilience mechanism is incorporated transport-layer protocols, not only can simplify transfer structure and reduce complexity, and can also improve error resilience mechanism flexibility enhancing and transmit reliability.Owing to have error elasticity, the present invention claims that this improved Real-time Transport Protocol is error elasticity real time transport protocol (ErrorResilience Real-time Transport Protocol is called for short " ERRTP " or " ER2TP ").The main distinction of ERRTP and RTP is that ERRTP protocol data header packet information expansion can carry forward error correction coding and decoding scheme relevant information, such as FEC type, protective capability, coding parameter etc.

On the ERRTP basis; the present invention has realized unequal loss protection easily; at first provide the different safeguard measure of kinds of protect ability available use; after transmitting terminal obtains information such as network condition and multi-medium data importance in collection then; can select suitable safeguard measure according to these factors; thereby reach the purpose of unequal loss protection, realize the equilibrium of protective capability and transmission efficiency.Owing to all carried the FEC relevant information that it adopted on each ERRTP packet, so transmitting terminal only needs the information of selected scheme is inserted in the ERRTP header packet information, and receiving terminal just can correctly recover or error correction according to it.

At last, provided specific implementation method, the step that comprises division, generates, encapsulates and go encapsulation of data node and check-node based on the correcting and eleting codes protection for H.264 NALU data-transfer applications.Continuous a string NALU waited together be divided into several back end longways, produce check-node with the Tornado sign indicating number then, all these nodes are distributed in again in several ERRTP bags and transmit, and receiving terminal then carries out this inverse process.

In the first embodiment of the present invention, receiving-transmitting sides is realized unequal loss protection based on ERRTP, and key step is as described below:

Transmitting terminal is selected the forward error correction coding scheme that multi-medium data is entangled and is deleted coding, with the multi-medium data behind the ERRTP encapsulation coding, and carries forward error correction coding scheme relevant information in the ERRTP header packet information, sends to receiving terminal then;

Receiving terminal goes the ERRTP bag of receiving to encapsulation, and extracts forward error correction coding scheme relevant information from the ERRTP header packet information, then according to forward error correction coding scheme relevant information, selects the forward error correction coding scheme to entangle and deletes decoding, obtains multi-medium data.

Wherein, unequal loss protection is embodied in transmitting terminal and selects the forward error correction coding scheme according to current network transmission status and/or service of multimedia data credit rating to be sent.

At first introduce the concrete structure of ERRTP, provide the header structure embodiment of concrete ERRTP below.Fig. 4 is the ERRTP header structural representation according to the first embodiment of the present invention.As can be seen from the figure, version information field V value is 3, and expression ERRTP agreement is to be different from traditional Real-time Transport Protocol (V=2).Wherein just last with relevant information field about the forward error correction coding and decoding scheme in the header expansion, comprise in this example: forward error correction coding type field, forward error correction coding parameter field, data packet length field, number of data packets field.

The forward error correction coding type field, the correcting and eleting codes type that is used to indicate the forward error correction coding scheme to adopt also can be called the FEC the type field, promptly indicate the FEC type of coding, account for 4 bits, can represent 16 kinds of different FEC types, from practical application, be enough.Here Ding Yi type is big type in fact, and the back also is subdivided into various scheme with continuing, and is called subtype, the major types in the practical application for example: 0010 expression Tornado sign indicating number, 0011 expression RS sign indicating number etc.This field can identify 16 kinds of different FEC sign indicating number major types, and table look-up (Look-Up Table is called for short " LUT ") of corresponding relation is called FECTypeLUT between communicating pair needs FEC type of coding of prior agreement and the type of coding code name.

The forward error correction coding sub-type field is used to indicate the relative parameters setting of forward error correction coding scheme, needs also to determine that for every type FEC coding being provided with of various parameters could specifically implement, and this field is exactly to play the effect of clear and definite concrete parameter.Because resource-constrained in the ERRTP header can not be enumerated pairing concrete parameter of various FEC encoding schemes and rule thereof etc. one by one, the first embodiment of the present invention is by indicating the various alternative parameter plans of establishment with the notion of subtype.This field is also referred to as FEC coding sub-type field, and FEC Subtype accounts for 9 bits.This territory mainly is illustrated in below each major types that defines among the FECTypeLUT the further subtype of segmentation.

The data packet length field is used to indicate the forward error correction coding scheme in the back end length of multi-medium data being entangled after deleting coding, is called Data Length field, accounts for 11 bits.Because each data packet length should transmit maximum transmission unit (Maximum Transport Unit less than network, be called for short " MTU "), and present wire message way MTU＜1500=0 * 5DC byte, wireless channel MTU＜100 bytes, so 11 bits of this field are enough to the length of store data bag.

The number of data packets field is used to indicate this ERRTP to wrap the number of institute's data carried by data node, is called Packet Number field again, account for 8 bits, such as for several NALU through forward error correction code checks after, packet encapsulation in a plurality of ERRTP, institute's data carried by data node number among each ERRTP.

As seen had after these fields, the FEC code type that decoding end or network node can provide according to this field and the verification type of packet are carried out verification to the packet that receives, and recover the packet lost.

It should be noted that subtype FEC Subtype field above-mentioned totally 9 bits be to be used for encoding the various alternative parameter plans of establishment of indication, just provide the ins and outs of the indication of how encoding in the first embodiment of the present invention below.

At first receiving-transmitting sides need consult to determine this field indexical relation correspondence table.Before beginning transmission, transmitting terminal and receiving terminal are consulted to determine: for various FEC sign indicating number major types, the corresponding relation of the relative parameters setting scheme of this kind FEC sign indicating number that the value of FEC Subtype is indicated with it, and the concrete parameter of various alternatives is provided with situation.

Then, transmitting terminal and receiving terminal are all set up mapping table according to negotiation result, are used for inquiring about pairing FEC type of coding or FEC encoding and decoding processing module according to FECType and FEC Subtype field;

In the transmitting-receiving process, transmitting terminal calls corresponding entangling and deletes the encoding process module and entangle and delete coding, and receiving terminal calls corresponding entangling and deletes decoding process module and entangle and delete decoding.

In actual applications, in fact the information of subtype indicate two aspects:

The create-rule (Generation Rule) of A.FEC coding;

B. protect intensity/protective capability.

So-called create-rule is exactly how back end to be handled rule or the algorithm (Algorithm) that generates each check-node at transmitting terminal.Certainly receiving terminal do just in time opposite, if packet loss has taken place in transport process, i.e. some node loss can be recovered according to create-rule so or part is recovered the node lost.As seen create-rule is very important information, and according to it, the both sides of communication just can come work based on FEC mechanism.Each class in the FEC type of listing in FECTypeLUT all has different create-rules; And in each class, such as the Tornado sign indicating number, the create-rule of following subclass also will be in conjunction with concrete generation parameter (generation parameters).Therefore specific to each subclass here, claim rule will with generate parameter and combine.

Such as for the Tornado sign indicating number, generate parameter and comprise following data: back end sum, check-node sum, the check-node number of plies, two-layer in succession between the economize on electricity number pass the ratio of contracting, expression two-layer in succession between the incidence matrices of node incidence relation, if L layer check-node arranged, so such incidence matrices just has the individual or equivalent expression of the L parametrization mathematical notation of the bigraph (bipartite graph) of two-layer node incidence relation (Bipartite) (parametric mathematical representation) in succession.

In general, under the identical prerequisite of big create-rule, generate the often protection intensity of determinant type of parameter.Such as the Tornado sign indicating number, in the every generation parameter that provides in the above, back end sum and check node sum can determine protective capability (certainly strictly speaking, determine protective capability fully, need whole generation parameters) basically to a great extent.In the present invention, for each FEC major types, the major parameter (decisive action maximum) of selecting some decision protective capabilities is as the representative parameter (representative generation parameters) that generates.By using the representative parameter that generates, just can line up the subclass below the big class according to protective capability order (ascending order) from weak to strong.Thereby set up a LUT and be called FECSubTypeLUT.

Mask body is supported a plurality of subtypes under each major types, the concrete application and the communication capacity of communicating pair (factors such as CPU processing speed, internal memory, program complexity) can be arranged and need decision.If communication environment alters a great deal, the performance inconsistency scope of network is very big, and in general the subtype that needs so to support want many, on the contrary can be less.This fully can be in handling capacity negotiations process before the communication beginning, the agreement of being reached an agreement by communicating pair.Negotiation can pass through H.323 or the multimedia communication framework agreement of session initiation protocol present main flows such as (SessionInitial Protocol are called for short " SIP ") is carried out.

Suppose that at the subclass below certain big class (S≤29-1), the representative parameter that generates has k, uses p1 to distinguish S subtype if desired, p2 ..., pk represents, table 2 provides the example of a corresponding relation so, and subscript is represented the FEC major types in the table, and subscript is represented concrete which parameter.

Table 2 FEC Subtype and parameter plan of establishment mapping table

FEC Subtype	The FEC subtype (parameter setting) of encoding
FEC Subtype	The FEC subtype (parameter setting) of encoding	000000000	FEC subtype 0 (p ⁰ ₁，p ⁰ ₂，..，p ⁰ _k)
000000001	FEC subtype 1 (p ¹ ₁，p ¹ ₂，..，p ¹ _k)	000000000	FEC subtype 0 (p ⁰ ₁，p ⁰ ₂，..，p ⁰ _k)
000000001	FEC subtype 1 (p ¹ ₁，p ¹ ₂，..，p ¹ _k)	000000010	FEC subtype 2 (p ² ₁，p ² ₂，..，p ² _k)
000000011	FEC subtype 3 (p ³ ₁，p ³ ₂，..，p ³ _k)	000000010	FEC subtype 2 (p ² ₁，p ² ₂，..，p ² _k)
000000011	FEC subtype 3 (p ³ ₁，p ³ ₂，..，p ³ _k)	…………………	……………..
S(S≤2 ⁹-1)	FEC subtype S (p ^S ₁，p ^S ₂，..，p ^S _k)	…………………	……………..

Such as, for the Tornado sign indicating number, corresponding relation can be set be: 000000010-(24,20) (back end sum=20, check-node sum=4), 000000011-(30,20) ..., 111111111-other.

At the subtype of the FEC of certain specific character coding, one group of given create-rule is in conjunction with the corresponding corresponding unique encoding scheme of parameter that generates, and how promptly unique the decision generates check-node by back end, and the node how to recover to lose.Can set up a database, store the generation parameter of every kind of major types and subtype correspondence.And create-rule itself is realized with hardware or software module.Therefore, every kind of major types is responsible for generating check-node in the corresponding FEC processing module of transmitting terminal; In the same corresponding FEC processing module of receiving terminal, be responsible for recovery nodes.But the module of corresponding every kind of major types need read the generation parameter of concrete every seed category, thereby handle from above-mentioned generation parameter database.Therefore, current both sides are that information according to FEC Type and two information fields of FEC Subtype decides to call which FEC processing module and read those and generates parameter.

Because the development of multimedia communication technology at present, H.264 video encoding standard becomes mainstream media's coded format gradually, therefore the second embodiment of the present invention is on the basis of first embodiment, provided the concrete steps of NALU data flow H.264 being carried out the FEC encoding and decoding with ERRTP, its flow process as shown in Figure 5.

In step 501, H.264NALU transmitting terminal merges into one group of unification transmissions of encoding with a plurality of (are assumed to be S), earlier S NALU is reclassified as isometric piece, and it is individual to be assumed to be M, and this M is individual to be exactly back end.

In this step, H.264 S NALU is divided into one group; With S NALU end to end (concatenated), be connected to form a bulk then, then this bulk be divided into M data block, wherein the length of each data block is K byte.If total byte number (being made as TB) of this bulk can not be divided exactly by M here, should carry out rounding operation so, make that the length of each data block is Ceiling (TB/M) byte, the Ceiling function representation rounds, be the smallest positive integral that Ceiling (x) equals to be not less than x, x is any real number.Back in some data block is adopted the operation of filling zero string (zeropadding) possibly so, makes the byte number Ceiling (TB/M) that gathers together enough).

Then in step 502, it carries out the FEC coding to M data node, obtains N check-node.Use FEC sign indicating number coding to generate N check block to M data block, the method that generative process adopts the front to describe according to FEC Type and FEC Subtype information, is determined to call concrete which FEC processing module and is carried out the generation of check block.

In step 503, transmitting terminal sends all back end and check-node packet encapsulation in the ERRTP bag then.Fig. 6 shows the structure that P ERRTP packing carried M+N data node.The header form of the ERRTP that provides in conjunction with Fig. 4, each field should be by following setting in this example:

Type field FEC Type=0010, the Tornado sign indicating number is used in expression;

Sub-type field is then specifically selected according to actual conditions by transmitting terminal, is FECSubtype=000000010 such as value, and Tornado (24,20) sign indicating number is used in expression, and wherein back end is 20, and 4 of check-nodes, code redundancy degree are 16.7%; This correcting and eleting codes smaller or equal to 3% o'clock, can recover the packet of losing at packet loss fully;

Data packet length Data-Length=K Bytes;

Number of data packets Packet Number=(M+N)/P represents data carried by data node number in the ERRTP load.

In step 504, receiving terminal goes encapsulation to obtain back end and check-node after receiving these ERRTP bags.Receiving terminal is the cycle with P packet, whenever receives one group of P packet and just begins the recovery of once decoding.One group of what packet is consulted is by both party determined.

In step 505, receiving terminal carries out decoding FEC according to check-node to the data node.After receiving packet P+1, begin to detect in P the packet of receiving previously whether data-bag lost is arranged at every turn, if have and just adopt previously described method, according to FEC Type and FEC Subtype information, determine to call that concrete which FEC processing module is decoded and the data of recovery or partial loss.

In step 506, after obtaining complete back end, reconsolidate and just obtain a bulk at last at last, adopt the mode identical, divide and obtain S NALU with transmitting terminal.

Find that in actual applications last example adopts the anti-data-bag lost algorithm based on ERRTP, can under situation about increasing, improve the anti-data-bag lost ability of video code flow greatly less than 17% code word.And compare with RTP payload header structure, only increased by 4 bytes, as seen transmission efficiency there is not influence substantially, obtained significant actual effect.

Having mentioned about another one key technology point of the present invention in front is exactly the realization of unequal loss protection.Be mainly reflected in two aspects, one is to select suitable coding and decoding scheme or parameter according to the multi-medium data of different important level, promptly determines aforementioned FEC type of coding and subtype, and another is exactly to select according to difference network condition constantly.Corresponding these two aspects are called mixing and are used alternatingly various FEC encoding schemes.So-called mix (Hybrid) uses multiple FEC subtype in being meant at one time simultaneously, is mainly used in the data of the different importance of protection; And so-called alternately (Alternation) is meant at different time (under the different network conditions) and uses different FEC subtype.

Therefore in the third embodiment of the present invention, provide this two kinds of unequal loss protection mechanism based on first embodiment.For data flow H.264NALU, the front is mentioned, its byte has embodied the significance level of data, therefore transmitting terminal can be assessed the QoS grade according to NRI field in the header of NALU or the type field, and then select the forward error correction coding scheme, promptly determine FEC the type field and FEC Subtype field.And for network condition, general network transmits all corresponding network condition monitoring mechanism, transmitting terminal can be known the transmission report of receiving terminal feedback according to these mechanism, with this evaluating network transmission status, and then select the forward error correction coding scheme, promptly determine FEC the type field and FEC Subtype field.

H.264 code stream is based on NALU and transmits or store, and NALU is made up of NAL header and NAL load.In NALU H.264, different N ALU type is recovered the difference that influences of image to decoding.For example, NRI gets Slice or the Slice data band that non-reference picture is deposited in 0 expression among the NALU, can not influence subsequent decoding; Show Slice or the Slice data band of depositing a sequence/picture parameter set or reference picture among the NALU and NRI gets non-0, can have a strong impact on subsequent decoding.

Therefore, when H.264 code stream being carried out the packet protection, can data H.264 be divided into two classes according to the value of NRI or Nal_unit_type: a class is relative important images data (for example Nal_ref_idc equals 1); Another kind of is less important view data (for example Nal_ref_idc equals 0).Then, use redundancy FEC1 sign indicating number big, that anti-packet loss ability is strong to protect to important view data; And less important view data can use the FEC2 sign indicating number that redundancy is less, anti-packet loss ability is more weak to protect.

By this unequal loss protection algorithm, guaranteed the correct recovery of all kinds of important informations under high data-bag lost environment, adopt code error shielding and prevent technology such as error code diffusion and the FEC2 sign indicating number is still failed image restored information.FEC1, FEC2 are general method for expressing here, represent any two seed categories.This two seed category can belong to same major types, also can belong to different major types.

Obviously, said method can be generalized to situation more generally, and data are divided into more multiclass according to the value of NAL_unit-type, such as five classes: most important data, inferior significant data, general significant data, more inessential data, significant data least; Also can be divided into 7 classes or more, so, can protect, the corresponding a kind of different subtype of every class data with the FEC subtype of equal number.As long as protective capability is just passable from weak to strong, these subtypes not necessarily belong to same major types.And adopt code error shielding and prevent technology such as error code diffusion still failing image restored information after the protection of the strongest FEC sign indicating number of protective capability.

Another situation of unequal loss protection also within the scope of the present invention, exactly can according to the network real time status select different protective capabilities FEC.Come the both sides of notifying communication then by the header of ERRTP, make the data that they can correctly be decoded and recover to lose data.Can be divided into several levels to the situation that the current conveying function that is affected of network descends.Such as Pyatyi: the most serious, inferior seriously, generally serious, more not serious, least serious; Also can be divided into 7 grades or more, so, can protect, every grade of a kind of different subtype of correspondence with the FEC subtype of equal number.As long as protective capability is just passable from weak to strong, these subtypes not necessarily belong to same major types.And adopt code error shielding and prevent technology such as error code diffusion still failing image restored information after the protection of the strongest FEC sign indicating number of protective capability.The sensing network situation can realize by existing various QoS monitoring methods.

More complicated application scheme can be used (the communicating pair terminal is all supported) if always have T kind FEC scheme (dissimilar/subtype) also within the scope of the present invention.Which kind of FEC decision adopts, and depend on data importance and network conditions simultaneously.Can adopt the method for a two-dimentional LUT so, as shown in table 3:

The two-dimentional LUT that many kinds of FEC mechanism of table 3 are mixed and are used alternatingly

Severity level 1	FEC(1，1)	FEC(1，2)	FEC(1，V)
Severity level 1	FEC(1，1)	FEC(1，2)	FEC(1，V)	Severity level 2	FEC(2，1)
…				Severity level 2	FEC(2，1)
…				Severity level U	FEC(U，1)	FEC(U，2)	FEC(U，V)

In the above form, data importance rank and network condition rank are all arranged according to ascending order.Wherein the subscript of FEC is represented with two-dimentional subscript, and the error resilience mechanism FEC in the table (i, j), 0＜i≤U, 0＜j≤V can be in above-mentioned T the FEC scheme any one.

It should be noted that, during the embodiment of foregoing invention describes all with the FEC correcting and eleting codes particularly the Tornado sign indicating number be example, but for other similar error resilience mechanisms particularly the FEC encoding scheme except that the Tornado sign indicating number can be suitable for, do not influence the spirit and scope of the invention.

And in another one embodiment of the present invention, adopted a kind of improved Tornado correcting and eleting codes specially, this improved Tornado correcting and eleting codes only generates the described check-node of one deck for one group of back end, can significantly reduce encoding time delay, satisfies the demand of real time communication.

In real-time video communication, use the protection of FEC code data bag can introduce time delay, the size of time delay is relevant with the size of view data packet.S NALU is divided into one group, and one of them NALU comprises the bit stream data of a Slice.If a two field picture is divided into a Slice, then coding side just has the time delay of S frame, and same decoding end also has the time delay of S frame.The relation of NALU and back end number is shown below:

Σ_{i = 0}^{s} {NalSize}_{i} = PackSize \times DataNode \cdot \cdot \cdot (1)

The addition of S NALU length value equals the size that the back end number is multiplied by each node data bag in the formula.As can be seen when the S value is limited, the value of PackSize * DataNode also can be limited by formula (1), and in addition because the validity that IP network transmits causes the PackSize value can not be too little, so the value of DataNode is limited.

During real-time video is communicated by letter on the IP network, the time-delay T of a two field picture _TotalBe calculated as follows:

T _total＝T _FEC+T _codec+T _trans (2)

T in this formula _FECBe to add the time delay that introduce FEC protection back, T _CodecAnd T _TransBe respectively H.264 codec handling time delay and network transfer delay.Because developing rapidly of Digital Signal Processing and IP network can be supposed T _CodecAnd T _TransCan both requirement of real time:

T _Codec＜=T _Th, T _Trans＜=T _Th, T wherein _Th=1/F _Target(3)

F in the formula (3) _TargetBe decoding target frame rate (but value 10Hz, 30Hz etc.), and establish a two field picture and be divided into a Slice that this up-to-date style (2) can change into:

T _total＜＝S*T _th+2*T _th＝(S+2)*T _th (4)

By formula (4) and formula (1) as can be known, the time-delay T of a two field picture _TotalTime-delay determine by the value of S that substantially and DataNode influences the value of S greatly.Therefore, under the prerequisite that can guarantee the anti-data-bag lost ability of video communication, reduce the time delay that FEC introduces as far as possible, further guarantee the QoS of real-time video communication.

The present invention adopts improved Tornado sign indicating number protection algorithm under the limited situation of DataNode.This improved Tornado method does not adopt the coded system of multistage bigraph, but only uses the coded system of one deck check-node.Compare with original T ornado coded system, coding method after the improvement has improved the flexibility of algorithm greatly, the number of back end and check-node can be provided with arbitrarily, has also reduced the complexity of code decode algorithm, can be used for the anti-data-bag lost of real-time video communication.In addition, under the limited situation of back end, the anti-data-bag lost performance of improving the Tornado sign indicating number does not have to descend substantially.This concrete principle of improved Tornado coding method and detailed step, with reference to Chinese patent " a kind of data based on correcting and eleting codes transmit guard method ", the number of accepting is 200510066146.7.

Those of ordinary skill in the art are appreciated that, given concrete parameter setting and numerical value and other realize that details for example in the foregoing description, in concrete the application, can adopt other feasible value or schemes, realize the object of the invention and do not influence essence and scope.

Claims

1. a multimedia data network realtime transfer method of supporting error elasticity is characterized in that, comprises following steps:

Multi-medium data after the described transmitting terminal of B is encoded with the encapsulation of error elasticity real time transport protocol, and in described error elasticity real time transport protocol header packet information, carry described forward error correction coding scheme relevant information, send to receiving terminal then;

2. the multimedia data network realtime transfer method of support error elasticity according to claim 1 is characterized in that, the multi-medium data behind the described forward error correction coding is divided into back end and check-node two classes.

3. the multimedia data network realtime transfer method of support error elasticity according to claim 2 is characterized in that, in described steps A, described transmitting terminal is selected described forward error correction coding scheme according to one of following factor at least:

4. the multimedia data network realtime transfer method of support error elasticity according to claim 3 is characterized in that, comprises in the described error elasticity real time transport protocol header packet information:

5. the multimedia data network realtime transfer method of support error elasticity according to claim 4 is characterized in that, when described multi-medium data during for network abstraction layer unit H.264,

In described step D, if having taken place, loses the back end in the transport process, then described receiving terminal carries out decoding FEC according to described check-node to described back end, and division obtains described H.264 network abstraction layer unit.

6. the multimedia data network realtime transfer method of support error elasticity according to claim 5 is characterized in that, before beginning transmission, also comprises step,

For various described forward error correction code types, the corresponding relation of the relative parameters setting of this kind forward error correction that the value of described wrong forward error correction sub-type field is indicated with it is determined in described transmitting terminal and the negotiation of described receiving terminal.

7. the multimedia data network realtime transfer method of support error elasticity according to claim 6, it is characterized in that, described transmitting terminal and described receiving terminal are all set up mapping table according to the indication corresponding relation of described forward error correction coding sub-type field, are used for inquiring about pairing forward error correction coding or decoding FEC processing module according to described forward error correction coding type field and described forward error correction coding sub-type field;

8. the multimedia data network realtime transfer method of support error elasticity according to claim 7, it is characterized in that, in described steps A, described transmitting terminal is assessed the relative importance of corresponding data according to network abstract layer reference identification field in the header of described H.264 network abstraction layer unit and any one or both in the network abstraction layer unit type field, thereby determine described service quality rating, and then select described forward error correction coding scheme, determine described forward error correction coding type field and forward error correction coding sub-type field.

9. the multimedia data network realtime transfer method of support error elasticity according to claim 7, it is characterized in that, in described steps A, described transmitting terminal is estimated described network transmission status according to the transmission report of described receiving terminal feedback, and then select described forward error correction coding scheme, determine described forward error correction coding type field and forward error correction coding sub-type field.

10. according to Claim 8 or the multimedia data network realtime transfer method of 9 described support error elasticities, it is characterized in that, version information field value in the described error elasticity real time transport protocol header packet information is a binary one 1, to be different from real time transport protocol;

11. according to Claim 8 or the multimedia data network realtime transfer method of 9 described support error elasticities, it is characterized in that described forward error correction coding scheme is used improved " Tornado " correcting and eleting codes;