CN1863302A - Multimedia communication method and terminal thereof - Google Patents
Multimedia communication method and terminal thereof Download PDFInfo
- Publication number
- CN1863302A CN1863302A CN200510110013.5A CN200510110013A CN1863302A CN 1863302 A CN1863302 A CN 1863302A CN 200510110013 A CN200510110013 A CN 200510110013A CN 1863302 A CN1863302 A CN 1863302A
- Authority
- CN
- China
- Prior art keywords
- error
- real time
- data
- transport protocol
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 209
- 230000006854 communication Effects 0.000 title claims abstract description 188
- 238000004891 communication Methods 0.000 title claims abstract description 174
- 230000005540 biological transmission Effects 0.000 claims abstract description 78
- 230000007246 mechanism Effects 0.000 claims abstract description 66
- 230000004224 protection Effects 0.000 claims description 131
- 238000012937 correction Methods 0.000 claims description 89
- 238000009792 diffusion process Methods 0.000 claims description 57
- 238000005538 encapsulation Methods 0.000 claims description 55
- 238000012545 processing Methods 0.000 claims description 31
- 238000012546 transfer Methods 0.000 claims description 28
- 230000008569 process Effects 0.000 claims description 27
- 238000004364 calculation method Methods 0.000 claims description 18
- 230000011218 segmentation Effects 0.000 claims description 18
- 230000000153 supplemental effect Effects 0.000 claims description 13
- 238000011084 recovery Methods 0.000 claims description 12
- 230000003044 adaptive effect Effects 0.000 claims description 11
- 239000000284 extract Substances 0.000 claims description 11
- 230000001360 synchronised effect Effects 0.000 claims description 8
- 230000002708 enhancing effect Effects 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000004458 analytical method Methods 0.000 claims description 3
- GOLXNESZZPUPJE-UHFFFAOYSA-N spiromesifen Chemical compound CC1=CC(C)=CC(C)=C1C(C(O1)=O)=C(OC(=O)CC(C)(C)C)C11CCCC1 GOLXNESZZPUPJE-UHFFFAOYSA-N 0.000 claims description 2
- 238000012856 packing Methods 0.000 abstract description 9
- 238000013461 design Methods 0.000 abstract description 6
- 238000012986 modification Methods 0.000 abstract description 2
- 230000004048 modification Effects 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 44
- 230000001681 protective effect Effects 0.000 description 42
- 238000005516 engineering process Methods 0.000 description 27
- 230000000694 effects Effects 0.000 description 14
- 238000012544 monitoring process Methods 0.000 description 14
- 230000008901 benefit Effects 0.000 description 9
- 238000007906 compression Methods 0.000 description 9
- 230000006835 compression Effects 0.000 description 9
- 238000009826 distribution Methods 0.000 description 9
- 230000033001 locomotion Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000001965 increasing effect Effects 0.000 description 7
- 239000000203 mixture Substances 0.000 description 7
- 230000008030 elimination Effects 0.000 description 6
- 238000003379 elimination reaction Methods 0.000 description 6
- 239000000047 product Substances 0.000 description 6
- 230000002829 reductive effect Effects 0.000 description 6
- 230000033228 biological regulation Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 230000007423 decrease Effects 0.000 description 5
- 238000012217 deletion Methods 0.000 description 5
- 230000037430 deletion Effects 0.000 description 5
- 238000012795 verification Methods 0.000 description 5
- 238000009825 accumulation Methods 0.000 description 4
- 230000006978 adaptation Effects 0.000 description 4
- 230000009977 dual effect Effects 0.000 description 3
- 230000008713 feedback mechanism Effects 0.000 description 3
- 239000012467 final product Substances 0.000 description 3
- 238000002156 mixing Methods 0.000 description 3
- 238000010295 mobile communication Methods 0.000 description 3
- 230000035945 sensitivity Effects 0.000 description 3
- 230000001174 ascending effect Effects 0.000 description 2
- 230000003139 buffering effect Effects 0.000 description 2
- 238000000205 computational method Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 238000007429 general method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000036961 partial effect Effects 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 238000005728 strengthening Methods 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 206010011469 Crying Diseases 0.000 description 1
- 241001269238 Data Species 0.000 description 1
- 208000007542 Paresis Diseases 0.000 description 1
- 101150034459 Parpbp gene Proteins 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005056 compaction Methods 0.000 description 1
- 238000007596 consolidation process Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000012010 growth Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 208000012318 pareses Diseases 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 230000001846 repelling effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
- 238000003892 spreading Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/65—Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/0001—Systems modifying transmission characteristics according to link quality, e.g. power backoff
- H04L1/0023—Systems modifying transmission characteristics according to link quality, e.g. power backoff characterised by the signalling
- H04L1/0025—Transmission of mode-switching indication
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/0001—Systems modifying transmission characteristics according to link quality, e.g. power backoff
- H04L1/0023—Systems modifying transmission characteristics according to link quality, e.g. power backoff characterised by the signalling
- H04L1/0028—Formatting
- H04L1/0029—Reduction of the amount of signalling, e.g. retention of useful signalling or differential signalling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/0078—Avoidance of errors by organising the transmitted data in a format specifically designed to deal with errors, e.g. location
- H04L1/0079—Formats for control data
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention relates to multimedia communication technique, disclosing a multimedia communication method and the terminal design and implementation thereof, improving transmission reliability and communication quality. And the invention adopts ERRTP to provide a transmission layer packing form able to carry error resilient encoding solution related information on the basis of the existing RTP protocol, making multimedia data label the corresponding error resilient encoding solution related information while transmitted on ERRTP, thus merging error resilient mechanism into the transmission layer; in accordance with H.264 NALU structure, the invention provides a special ERRTP packing method and protocol header modification solution, able to integrate all NALU header bytes in the same ERRTP packet into its header, thus expressing important information of NALU in the ERRTP header and raising transmitting efficiency.
Description
Technical field
The present invention relates to multimedia communication technology, particularly support the multimedia communication technology of error elasticity.
Background technology
Along with the develop rapidly of computer internet (Internet) and mobile communications network, the application of stream media technology more and more widely, broadcasting, film are played to remote teaching and online news website etc. and have all used stream media technology from network.Current online transmission video, audio frequency mainly contain download (Download) and streaming transmits (Streaming) dual mode.It is to transmit the video/audio signal continuously that streaming transmits, when Streaming Media remainder when client computer is play continues on the backstage to download.Streaming transmits has progressive streaming to transmit (Progressive Streaming) and real-time streaming transmission (Realtime Streaming) dual mode.It is real-time transmission that real-time streaming transmits, and is particularly suitable for live event, and real-time streaming transmits and must coupling connect bandwidth, this means that picture quality can reduce variation because of network speed, to reduce the demand to transmission bandwidth.The notion of " in real time " is meant that the payment of data in an application must keep the precise time relation with the generation of data.
Especially along with 3-G (Generation Three mobile communication system) (3rd Generation, be called for short " 3G ") appearance and generally based on Internet protocol (Internet Protocol, abbreviation " IP ") network develops rapidly, and video communication just progressively becomes one of main business of communication.And both sides or multi-party video communication service, as video telephone, video conference, mobile multimedia terminal service etc., more transmission and the service quality to multimedia data stream proposes harsh requirement.It is better not only to require network to transmit real-time, and equivalence also require video data compaction coding efficient higher.
Demands status in view of media communication, (the InternationalTelecommunication Union Telecommunication Standardization Sector of ITU-T, be called for short " ITU-T ") after having formulated H.261, H.263, H.263+ waiting video compression standard, formally issued H.264 standard in 2003.This is ITU-T and (the InternationalStandardization Organization of International Standards Organization, abbreviation " ISO ") Motion Picture Experts Group (Moving PictureExperts Group is called for short " MPEG ") unites the adaptation new stage network media transmission of formulation and the efficient compressed encoding standard of communication requirement together.It also is the main contents of MPEG-4 standard the 10th part simultaneously.
Formulating H.264, the purpose of standard is to improve more effectively video coding efficient and its suitability to network.In fact owing to its superiority, H.264 the video compression coding standard has become the mainstream standard in the current multimedia communication soon gradually.A large amount of employing H.264 multimedia realtime communication product (as video conferencing, video telephone, 3G mobile communication terminal) and network flow-medium product is successively come out, and whether supports H.264 to have become the key factor of decision product competitiveness in this market segment.Can predict that along with H.264 formal promulgation be extensive use of, the multimedia communication of IP based network and 3G, back 3G wireless network must enter a new stage of developing by leaps and bounds.
Simply introducing H.264 below, the message of standard constitutes and transmit mechanism: H.264 standard adopts layered model, defined video coding layer (Video Coding Layer, be called for short " VCL ") and network abstract layer (Network Abstraction Layer, be called for short " NAL "), the latter aims at network and transmits design, the video that can adapt in the heterogeneous networks transmits, and further improves " compatibility " of network.H.264 introduced towards the encoding mechanism of IP bag, the grouping that helps in the network transmits, and the Streaming Media of video transmits in the network enabled; Have stronger anti-bit error performance, adapt to the requirement of the wireless video transmission of packet loss height, serious interference especially.H.264 all data to be transmitted comprise that view data and other message all are encapsulated as the bag transmission of consolidation form, i.e. network abstraction layer unit (NAL Unit is called for short " NALU ").Each NALU is the variable length byte character string of certain syntactic element, comprises the header that comprises a byte, can be used to represent data type, and the load data of some integer bytes.NAL unit can carry a coded slice, categorical data is cut apart or sequence or picture parameter set separately.In order to strengthen data reliability, every two field picture all is divided into several bands (Slice), and each Slice is carried by a NALU, and Slice is made up of several littler macro blocks again, is minimum processing unit.In general, the Slice of front and back frame correspondence position is interrelated, and the Slice of diverse location is separate, can avoid taking place between the Slice counterdiffusion of error code phase like this.
H.264 data comprise data texturing, sequential parameter, image parameter, additional message (Supplemental Enhancement Information is called for short " SEI "), the reference frame data texturing etc. of strengthening of non-reference frame.Wherein, SEI message is the general designation in decoding, the demonstration of video H.264 and the message that helps out aspect other.Prior art has defined all kinds of SEI message, has kept the SEI reserving message simultaneously, for various may the application in future stayed expansion leeway.According to H.264, SEI message is not as necessary at decode procedure reconstruct brightness and chromatic diagram.Meeting the H.264 decoder of standard, is need not do any processing for SEI.That is to say, what the not all terminal that meets H.264 basic demand can both treatment S EI message, but for terminal that can not treatment S EI message, sending SEI does not influence for it, and it can neglect the SEI message that it can not be handled simply.According to the SEI syntax rule, the user can utilize reserving message to transmit self-defined message, realizes the function expansion.
At first introduce related datas such as the structure of SEI message and the syntax below.H.264 provide the multiple mechanism that can carry out extension of message in, comprising SEI.H.264 defined supplemental enhancement information (SEI) in, its data representation zone is independent with video data encoder, provides in the description of its using method NAL in agreement H.264.H.264 the base unit of code stream is NALU, NALU can carry various H.264 data types, such as video sequence parameter (Sequence parameters), and image parameter (Picture parameters), Slice data (being concrete view data), and SEI message data.SEI is used to transmit various message, supports extension of message.Therefore be used to be transmitted as specific purpose and self-defining message in the SEI territory, and can not influence based on the compatibility of video communication system H.264.The NALU of carrying SEI message is called SEI NALU.A SEI NALU contains one or more SEI message.Each SEI message contains some variablees, mainly is load type (payloadType) and magnitude of load (payloadSize), and these variablees have indicated the type and size of message load.At AnnexD.8 H.264, defined the syntax and the meaning of one's words of some H.264 SEI message commonly used in D.9.The load that comprises among the NALU is called raw byte sequence payload (Raw-Byte Sequence Payload is called for short " RBSP "), and SEI is a type of RBSP.
The data representation zone of SEI abbreviates the SEI territory as.Each SEI territory comprises one or more SEI message, and SEI message is made up of SEI header and SEI payload.The SEI header comprises two fields: a type that provides load in the SEI message, another provides the size of load.When load type is represented to 0xFE with a byte 0x00 between 255 the time 0, when type is represented to 0xFFFE with two byte 0xFF00 between 511 the time 256, when type greater than 511 the time method for expressing by that analogy, the user can self-defined any multiple load type like this.Wherein type 0 has been defined as information specific such as caching period, image timing etc. in Class1 8 standards.The SEI territory of definition can be deposited abundant customized information according to demand in this shows H.264.For the H.264 decoder of not supporting to resolve these customized informations, can abandon the data in the SEI territory automatically.Therefore, charging to useful self-defined information in the SEI territory can not influence based on the compatibility of video communication system H.264.
Multimedia communication not only requires media compression code efficiency height as previously mentioned, and the real-time that requires network to transmit.Media stream transmission at present all is to adopt real time transport protocol (Real-timeTransport Protocol is called for short " RTP ") and control protocol (Real-time Transport ControlProtocol is called for short " RTCP ") thereof basically.RTP is a transportation protocol going up multimedia data stream at Internet, by the Internet engineering duty group (Internet Engineering Task Force is called for short " IETF ") issue.RTP be defined in one to one or the transmission situation of one-to-many under work, its objective is provides temporal information and realizes that stream is synchronously.The typical case of RTP uses and is based upon User Datagram Protocol (UserDatagram Protocol, be called for short " UDP ") on, but also control protocol (TransportControl Protocol can transmitted, be called for short " TCP ") or asynchronous transfer mode (Asynchronous Transfer Mode is called for short " ATM ") wait on other agreements and work.
RTP itself only guarantees the transmission of real time data, can not provide reliable transfer mechanism for transfer data packets in order, and flow control or congested control are not provided yet, and it relies on RTCP that these services are provided.RTCP is in charge of delivery quality exchange of control information between the current application process.During the RTP session, each participant periodically transmits the RTCP bag, contains the quantity of data packets that has sent, the statistics of losing such as quantity of data packets in the bag, therefore, server can utilize these information dynamically to change transfer rate, even changes PT Payload Type.RTP and RTCP are used, and can make the transmission efficiency optimization with effective feedback and minimum expense, so be fit to transmit online real time data.
And H.264 multi-medium data transmits on IP network, the unexceptional Real-time Transport Protocol that also is based on UDP and its upper strata.RTP itself structurally can both be suitable for for different media data type, but different upper-layer protocol or media compression coding standard in multimedia communication are (as H.261, H.263, MPEG-1/-2/-4, MP3 etc.), IETF can formulate the authority file at RTP payload (Payload) packaging method of this agreement, the method for the big bag of specified in more detail RTP encapsulation, for this concrete agreement through optimization.Same, be H.264 Video of RFC 3984:RTPPayload Format for for H.264 also having corresponding ietf standard.Be the main standard that on IP network, transmits of video code flow H.264 before this standard mesh, use very extensive.In field of video communication, each mainstream vendor's product all is based on RFC 3984, also is only H.264/RTP load mode at present.
In fact, H.264 be H.264 to define a new aspect, be called network abstract layer (Network Abstract Layer is called for short " NAL ") with other the different key place of video compression coding agreement in the past.H.264 in order to increase its video coding layer (Video Coding Layer, be called for short " VCL ") with the separating and independence of the network transportation protocol layer of following mask body, bring bigger application flexibility, defined this new aspect of NAL, this layer, all do not have in H.263/H.263+/H.263++ such as H.261 in the early stage video compression coding agreement of ITU-T.Yet, how in NAL and Real-time Transport Protocol carrying collaborative work at H.264 higher, the better scheme of advantage design efficiency, make RTP better for load-carrying properties H.264, be very worth research and the problem that very big using value is arranged.
The method of RTP that the RFC3984 standard is proposed carrying NAL layer data H.264 is present main flow transfer approach, and this scheme is encapsulated in the NAL layer data in the RTP payload and carries on the basis of Real-time Transport Protocol (RFC 3550).The NAL layer is between VCL and RTP, and regulation will be divided into a series of NAL data cell (NAL Units is called for short " NALU ") to rule and the structure of video code flow according to definition.In RFC3984, defined the encapsulation format of RTP payload for NALU.Simply introduce the method for packing of NALU in the frame format of RTP and the prior art below successively.
The main purpose of RTP design is real-time multimedia meeting and continuous data storage, mutual distributed emulation, control and measurement application etc.RTP is carried on the udp protocol usually, to utilize its multiplexed and function verification.If bottom provides the multiple spot distribution, RTP supports multiaddress to transmit.The function that RTP provides comprises: load type discriminating, sequence numbering, timestamp and transmitting supervisory.
The packet format of RTP is as follows: RTP header most basic option takies 12 bytes (minimum), and the header of IP agreement and udp protocol takies 20 bytes and 8 bytes respectively, therefore RTP seals and is contained in the UDP bag and is encapsulated in the IP bag again, and it is the 12+8+20=40 byte that total header takies byte number.The detailed structure of the header of RTP bag as shown in Figure 1.
Shown in Fig. 1 from front to back the RTP header be followed successively by: the 1st byte (byte 0) is some fields about header structure itself, the 2nd byte (byte 1) is the definition payload type, the 3rd, 4 bytes (byte 2,3) be bag sequence number (Sequence Number), the 5-8 byte is timestamp (timestamp), the 9-12 byte is to contribute source identifier (Synchronous Source Identifier synchronously, be called for short " SSRCID "), be contribution source identifier (Contributing Source Identifiers at last, abbreviation " CSRCIDs ") tabulation, its number is uncertain.Notice, the byte 0 of the 1st byte in this paper describes for marking, the rest may be inferred afterwards.
Wherein preceding 12 bytes appear in all dissimilar RTP packets, and other data in the header just have such as contributing the source identifier sign to have only when blender inserts.Situation when therefore CSRC generally is used to exist medium to mix, such as in Multi-Party Conference, audio frequency needs to mix, and video also can provide the function of many pictures in this way.And Synchronization Source SSRC is exactly the sign of institute's carrying media stream in fact.
The concrete meaning and the full name of above-mentioned each field are described below respectively:
The V field is version (Version) information, account for 2 bits (bits), the version that adopts is 2 at present, therefore put V=2, and other values such as V=1 represent more RTP version early, and V=0 represents the most original RTP predecessor, promptly adopt in voice IP (VOIP) communication system of using on the Mbone network in early days, be evolved into RTP afterwards, V=3 is then still undefined.
The P field accounts for 1 bit for filling sign (Padding), if P set represents that then the packet end comprises one or more byte of paddings (Padding), fills a part that does not belong to payload;
X field is expansion sign bit (Extension), account for 1 bit, if X set, then the RTP head must expand with the head of a variable length at last (if the CSRC tabulation is arranged, the head expansion will be followed thereafter), mainly be to be preserved for the not enough situation of following information field of some applied environment, the length field that this header expansion comprises one 16 bit is counted the word what 32 bit long are arranged in the expansion, preceding 16 bits of head expansion are that a left side is open, so that distinguishing identifier symbol and parameter, the form of this 16 bit is by concrete aspect normalized definition, and the formal definition of this expansion has a detailed description in RFC3550 5.3.1 joint.
The CC field is contribution source number (CSRC Count), accounts for 4 bits, indicates the number of the rearmost CSRC identifier of header, and the recipient can determine the CSRCIDs list length of header back according to the CC field;
The M field is sign bit (Marker), account for 1 bit, the explanation of this sign bit defines in specific aspect (Profile), it allows to identify the critical event in the data packet stream, an aspect can define additional sign bit or stipulate not identify bit, here so-called aspect just is meant concrete applied environment setting, is specifically reached an agreement on by communicating pair, is not subjected to the qualification of agreement;
The PT field is load type (Payload Type is called for short " PT "), and totally 7 bits identify the form of RTP load and determine his explanation in application program; Flag bit and load type totally one byte are carried the aspect provisioning information, this byte may be redefined to adapt to different demands by concrete aspect, in concrete the application, can define so-called profile, be exactly one group of static state (being that communicating pair is appointed in advance) corresponding relation in fact, the value that the PT bit is different is mapped with different media formats.Can certainly carry out relation between dynamic negotiation definition PT value and the media formats by the signaling outside the RTP.In a RTP session (Session), PT can be changed in the RTP source.
Field then is exactly sequence number totally 16 bits, RTP packet of every transmission, and this sequence number value adds one, and the recipient can detect data-bag lost and recovery data packets order with it like this, and once the sequence number initial value in the communication can be given at random, do not influence communication.
Timestamp accounts for 32 bits, and it has reflected the sampling time of first byte in the RTP packet, and the sampling time here must derive from the clock of a dull linear growth, and the recipient adjusts the media play time according to it or carries out synchronously.
Synchronisation source SSRC ID accounts for 32 bits, and its occurrence can be selected at random, but will guarantee the uniqueness in the same RTP session, can source of media of unique identification, if a source has changed the transfer address, source, must select a new SSRC identifier.
CSRC tabulation in contribution source can be the 0-15 item as required, and every accounts for 32 bits, and the length of this tabulation is that the number of CSRC ID is just in time marked by 4 bits of CC field.In fact, the CSRC identifier that is used to identify certain source of media is consistent with the SSRC identifier in its corresponding contribution source, only in role's difference of different recipients, and is changed to SSRC or CSRC.In multi-party communication, CSRC ID is inserted by blender.
Carrying under the situation of video H.264, RTP is packaged into RTP bag stream to NALU encapsulation H.264.In RFC 3984 files, mainly defined NALU, and provided the H.264 encapsulation packing form of layer NAL data in RTP based on this.The RTP encapsulation format of this NALU as shown in Figure 2.
Provide the encapsulating structure of a NALU in the payload of RTP among Fig. 2, first byte of front is the NALU header, be the data content of NALU afterwards, in the payload that is filled into the RTP bag that a plurality of NALU are end to end, in the end also have optional RTP to fill, this is the content of RTP packet format regulation, is in order to make the length of RTP bag meet certain particular requirement (such as reaching regular length), optionally generally all zero fillings of RTP padding data.
The NALU header i.e. the 1st byte, is also referred to as eight bit groups (Octet), and it has three fields, and meaning and full name are described below respectively:
The F Field Definition is for forbidding bit (forbidden_zero_ bit), account for 1 bit, be used to identify situations such as grammer mistake, if syntax clash arranged then be changed to 1, when having the bit mistake in Network Recognition this element, it can be made as 1,, be mainly used in and adapt to different types of network environment (environment that combines such as wire and wireless) so that the recipient loses this unit;
The NRI Field Definition is NAL reference identification (nal_ref_idc), account for 2 bits, be used to indicate the significance level of NALU data, its value is that the content of 00 expression NALU is not used in the reference picture of rebuilding inter prediction, but not 00 current NALU of expression is band (slice) or sequence parameter set (the Sequence Parameter Set that belongs to reference frame, abbreviation " SPS "), picture parameter set (PictureParameter Set, be called for short " PPS ") wait significant data, it is important more that this is worth the current NAL of big more expression;
The type field is defined as NALU type (Nal_unit_type), and totally 5 bits can have the type of 32 kinds of NALU, and the corresponding relation of its value and particular type provides in table 1 in detail.
The type field value and type mapping table in the table 1NALU header
| The Type value | The type of |
| 0 | Do not specify |
| 1 | The coding slice of |
| 2 | Coding slice data are divided A |
| 3 | Coding slice data are divided |
| 4 | Coding slice data are divided |
| 5 | Coding slice in the |
| 6 | SEI (supplemental enhancement information) |
| 7 | SPS (sequence parameter set) |
| 8 | PPS (picture parameter set) |
| 9 | The access unit delimiter |
| 10 | EOS |
| 11 | Code stream finishes |
| 12 | Padding data |
| 13-23 | Keep |
| 24-31 | Do not specify |
As seen, the information spinner that provides in the byte of the header of NALU will comprise validity, the importance rate of NALU, can determine RTP institute data carried by data importance according to these information.
Under the current multimedia communication framework H.264/RTP, mainly cooperate control protocol RCTP to finish service quality (Quality of Service is called for short " QoS ") monitoring by utilization, and based on this congested control and flow control.RTCP is mainly used in the control and the report of Real-time Transport Protocol.The main contents of report are exactly the information relevant with QoS.The method for reporting that RTCP takes is a periodic report, and promptly periodicity all participant communications of control data bags in two sides or multi-party conversation (Session) are reported and adopted and the same distribution mechanisms of RTP packet.Underlying protocol provides multiplexed (for example use separately independent udp port number etc.) of data and control data bag.In the RFC3550 file, advise that the session bandwidth that increases for RTCP is 5% of a medium bandwidth.
Introduce the type and the structure of RTCP packet below.Define following several RTCP type of data packet among the RTCP and carried various control information: transmit leg report (Sender Report is called for short " SR "), about the transmission of active transmit leg and the statistical information of reception; Recipient's report (Receiver Report is called for short " RR ") never is the participant receiving and counting information of active transmit leg; Resource description item (SourceDescription is called for short " SDES "), the inside comprises CNAME; Participant finishes (withdrawing from) sign (BYE); Special function (Application-specific function is called for short " APP ").
The pack arrangement that RTCP sends and reception is reported as shown in Figure 3, can be divided into three sections by content type, top is header, it then is caller information, next is the report content piece, the expansion of last is specific aspect (Profile) (so-called aspect represent at certain application-specific scene needs and the concrete regular special case of formulating).The meaning and the full name of the concrete field of each shown in Fig. 3 are described in detail as follows:
The V field is version information (Version is called for short " V "), accounts for 2 bits, and the version number of current RTCP is V=2;
The P field accounts for 1 bit for filling flag bit (Padding), if P set represent that then this RTCP packet comprises the additional byte of padding that some do not belong to control information at afterbody, but these bytes is calculated in length field;
The RC field accounts for 5 bits for receiving report counting (Reception Report Count is called for short " RC "), and the number of the reception report blocks that comprises in the packet allows for 0;
The PT field is type of data packet (Payload Type, be called for short " PT "), accounts for 8 bits, and identifying this in the time of value 200 is a SR packet;
Length (Length) field accounts for 16 bits, and equaling the RTCP packet is that the length of unit subtracts 1 with 32-bit words (32bitWord), comprises head and any filling;
The SSRC of transmit leg accounts for 32 bits, indicates the promoter's of this SR packet synchronous source identifier (Synchronous Source Identifier is called for short " SSRC "), and media data source of the synchronisation source unique identification here is such as the source of one road video;
NTP timestamp field is NTP (Network Time Protocol) timestamp (Network Time Protocol is called for short " NTP "), accounts for 64 bits, after this report sends, has indicated wallclock (absolute date and time), is used in combination with the RTP timestamp;
RTP timestampe field is the RTP timestamp, accounts for 32 bits, i.e. the timestamp of Real-time Transport Protocol generation;
The packet count area of transmit leg accounts for 32 bits, and indication is established to from transmission and produces the RTP packet sum that transmit leg transmits during this SR packet;
The transmit leg byte count field, totally 32 bits, indication is established to from transmission during this SR packet of generation, and transmit leg transmits the total bytes (not comprising head or filling) of load (Payload) in the RTP packet, and this field can be used for estimating the Mean Speed of load;
Field has afterwards comprised 0 or a plurality of reception report blocks, concrete piece number depends on other source number of learning from last report transmit leg, each receives the statistical information of the RTP packet that report blocks transmission receives from single synchronisation source, and these statistical informations comprise:
Fragment is lost (fraction lost) and is accounted for 8 bits, and expression is lost the fragment number from the medium in this source after Previous Report sends; Accumulation lost package number accounts for 24 bits, the number of accumulation packet loss since expression begins to receive;
Next is exactly the maximum sequence number of expansion, the arrival delay variation that receives, and all reflects the network transmission status;
A last SR (Last SR is called for short " LSR ") accounts for 32 bits, is meant the timestamp mark of a SR report on this source, and value is centre 32 bits of the NTP of a last SR;
Time delay since a last SR (Delay since Last SR, be called for short " DLSR ") accounts for 32 bits, is meant the time interval length during this SR from a last SR, and this parameter is to be used for calculating the key parameter that QoS reports.
Receiving report (RR) data packet format with the difference that sends report (SR) is: the value of type of data packet field is 201; There is not the caller information part.
According to the RTP/RTCP consensus standard, it is as follows that RTCP finishes four functions:
Basic function, for the real-time multimedia data delivery quality provides feedback report mechanism, this is the organic component of RTP as transport-layer protocols, this feedback function transmits transmit leg report (SR) by RTCP and recipient's report (RR) realizes;
RTCP is that a permanent transport layer sign is transmitted in each RTP source, be called canonical name (CanonicalName, be called for short " CNAME "), SSRC is identified at and finds may change when conflict or program are restarted, so the recipient need follow the tracks of each participant by CNAME;
Preceding two functions need all participants all to send the RTCP packet, therefore in order to allow RTP can increase the number of participant pari passu, and speed that must control RTCP packet;
The 4th is optional function, promptly transmits the least possible control information.
As seen, adopt rtcp protocol to transmit the QoS report, report these QoS information, realize the H.264 QoS monitoring of equivalent-load medium based on this according to the report content of rtcp protocol regulation.
Yet should be noted that, RTCP bring the QoS report mechanism can be provided in because adopt the periodic report method, caused the expense of additional networks bandwidth, the highlyest can reach 5%.If congested (Congestion) appears in network, cause transmitting QoS and descend, the added flow of RTCP generation will make that problem worsens more so.
Understood H.264/RTP and transfer structure based on the services quality monitoring of RTCP after, below simple error elasticity and the background of related that transmits relevant for video network of introducing.
H.264 video is the main agreement of following multimedia communication, and the network that following multimedia communication is used mainly is to be the packet-switched network and the wireless network of representative with IP.This two big class network all can't provide good service quality (Quality of Service, be called for short " QoS ") to guarantee, so video transmits on network and will inevitably be subjected to various transmission mistakes and the influence of packet loss, thereby makes the communication quality reduction.Topmost problem here is that IP network realizes that " doing the best " (best effort) transmits, and can not guarantee to transmit the QoS of video data.Particularly to the H.264 code stream through efficient compressed encoding, problem is more outstanding.Transmission as possible on the IP network can not guarantee the QoS of real-time video communication, is in particular in three aspects: data-bag lost, time delay and delay variation.Wherein, data-bag lost is to recovering the quality influence maximum of video, because H.264 Coding Compression Algorithm is used estimating motion and motion compensation technique, in case there is data-bag lost to exist, not only influence current decoded picture, and can influence the subsequent decoding image, be i.e. error code diffusion.The influence that the error code diffusion couple recovers video quality is very big, has only in conjunction with coding side and decoding end and unites anti-error code, could avoid the error code diffusion fully.
Error elasticity (Error Resilience) be meant that transfer mechanism has that guard against false takes place or the ability that after mistake takes place, can correct with certain ability (mistake intensity can be corrected within the specific limits fully; Surpass certain limit, can only partly correct).In the multimedia communication environment of extensive (we can say omnipresent) in future, whether a kind of video transfer mechanism has error elasticity will be very crucial.
There is multiple error resilience mechanism, such as forward error correction (Forward Error Correction, abbreviation " FEC "), automatic re-send request may (Automatic Retransmission Request, be called for short " ARQ "), error concealment (Error Concealment), message source and channel combined coding (Joint Source-ChannelCoding is called for short " JSCC "), interweave (Interleaving) and eliminate error code diffusion etc.Transmit on packet networks for video H.264, FEC is a kind of very practical technology, and effect is fine.This method mainly adopts multiple error correction coding to encode for the data that will protect, and essence is to form data redundancy, thereby increases the wrong ability of resisting.
Cardinal error is the packet loss mistake on packet networks, and this mistake is called deletion error (Erasure Error) in Coding Theory.Error correction coding at deletion error is that a big class is called correcting and eleting codes (Erasure Codes).So-called correcting and eleting codes is exactly that the data code flow order is divided into the identical unit one by one (Unit) of size piecemeal, also is called back end (Data Nodes), for representing conveniently, supposes total n data node.Calculate for these back end according to certain mathematical operation rule then and produce check-node (Parity Nodes or Check Nodes); in order to strengthen protective capability; can also continue to produce second layer check-node for these check-nodes according to identical or different mathematical operation rule computings; and the like; can generate the 3rd layer; the 4th layer, until N layer check-node.
In general, if relate to the multilayer check-node, the interstitial content on every layer successively decreases according to certain rule (modal is the geometric ratio rule) with respect to last layer, so just becomes one successively to pass the multilayer node structure that contracts.Can be expressed as a pyramid that turn 90 degrees visually to the right.Wherein, Far Left is the back end layer, and arranging to the right is the ground floor check-node successively, second layer check-node ..., N layer check-node.
Wherein a class correcting and eleting codes has a kind of very important character, and the time complexity that promptly handle to need is to have linear relationship with the data section n that counts, and therefore is called linear session characteristic (linear-time).And much other correcting and eleting codes will be much higher such as the time complexity of famous Reed-Solomon sign indicating number needs, be n*log2n*log (logn) order of magnitude.Therefore, it is far better to have its purposes in real time communication of correcting and eleting codes of linear session.
Tornado correcting and eleting codes (hereinafter all being called for short the Tornado sign indicating number) is a kind of novel correcting and eleting codes that occurs before and after 1998.The Tornado code structure is simple; Computing is efficient, because it has linear session; Protective capability is strong.In actual applications, obtained good effect.Obtained at present to use comparatively widely.Dynamic according to up-to-date ITU-T, SG16 is wherein considering to carry out standardized possibility for error control coding class (Error Control Codes) technology at present, mainly is to transmit at the video/audio network to protect.Tornado sign indicating number and a plurality of mutation thereof are likely important technology wherein.
In the Tornado sign indicating number, successively produce a plurality of check-node layers from back end.Check-node and back end all send to receiving terminal by transmitting terminal by network.If in the network transport process, part of nodes has been lost, because upper layer node has been participated in the generation of lower level node, therefore the information of upper layer node has been included in lower level node and more in the lower level node, therefore lose node information can by working majority purpose lower level node or more lower level node recover fully.If each node is a bag, the bag of then losing can be recovered fully by other bag that correctly receives.If the back end number is n, the check-node number of generation is l.The code check and the redundancy rate that then define correcting and eleting codes are respectively: r=n/ (n+l), 1-r=l/ (n+l); Under other condition same case (protective capability, the delay that causes etc.), code check high more (inevitably, redundancy rate is low more), then the efficient of correcting and eleting codes is high more.
Fig. 4 shows the relation between a kind of typical Tornado code data node and each layer check-node.Line among the figure between the node is called the limit, and the left node on expression limit participates in calculating the right side node, is a kind of logical relation of multi-to-multi between the two-layer node of visible front and back.If the back end number is n, total check-node number is m, then defines the code check r=n/ (n+m) and the redundancy rate 1-r=m/ (n+m) of correcting and eleting codes; (protective capability under same case; the delays that cause etc.), code check is high more, redundancy rate is low more, and then the efficient of correcting and eleting codes is high more.The structure of Tornado sign indicating number and performance are mainly by the decision of three factors: (a) number of back end and successively pass the rule that contracts, and generally pass and contract by equal proportion; (b) produce the computational methods of next node layer; (c) incidence relation between the adjacent two layers node.
Can push away down between each parameter of Tornado sign indicating number and concern that the number of back end is made as n, the check-node number is made as m, passs the ratio of contracting and is made as p, and the check-node number of plies is i, then before the number of i-1 layer check-node be respectively np, np
2..., np
I-1, and the number that last one deck is the i layer is decided to be np
i/ (1-p), obtain total node like this and count n+m=n+n+np
2+ ..+np
I-1+ np
i/ (1-p)=and n/ (1-p), m=np/ (1-p) is then arranged, be and pass the implication relation that satisfies between the ratio of contracting and the check-node number.Because the node that guarantees every layer is counted np, np
2..., np
I-1And np
i/ (1-p) all be integer, can calculate the feasible value of n according to given i and p, such as i=4, p=1/2 then can extrapolate n and be necessary for 16 multiple.
In the Tornado sign indicating number production process the computational methods of normal employing be XOR because XOR has restore funcitons very easily.For two isometric bit sequence A=[a
0, a
1, a
2..., a
L], B=[b
0, b
1, b
2..., b
L], carry out XOR by bit and obtain long equally bit sequence C, then there are following character: A and C XOR to obtain B, B and C XOR obtain A; Same for the XOR between a plurality of sequences, corresponding restoration methods is also arranged.As seen, through behind the XOR, i.e. foundation connects each other between back end or the check-node, after any one node loss, all can be recovered by all the other nodes.Owing to the ratio difference that contracts of passing of last one deck check-node, the error correction coding strategy of therefore general employing routine calculates, such as the Reed-Solomon sign indicating number.
Another key factor of Tornado sign indicating number be exactly before and after incidence relation between the layer, promptly certain node of lower floor is which node by preceding one deck calculates.According to graph theory, form a bigraph (bipartite graph) between the two-layer node of front and back, at preceding one deck and back one deck, last node layer is also referred to as left node respectively at the two ends on any limit, and back one node layer is called the right side node, the bar number degree of being called on the limit that each node is related with it.According to people's such as Luby Random Graph opinion mathematical proof, the vector of the degree of two side gussets of the bigraph (bipartite graph) that layer constituted before and after the parameter of the protective capability of decision Tornado sign indicating number was actually, and this degree vector produces at random.In actual applications, need to determine earlier the random distribution of node degree vector before Tornado encodes, distributing to mate at random according to this then produces bigraph (bipartite graph)s at different levels, the incidence relation before and after promptly having determined according to the internodal association in the bigraph (bipartite graph) left and right sides between the node layer.
In present Tornado sign indicating number strategy, by given protective capability and other requirement, such as the back end size reasonable, acceptable maximum network delay etc.; determine parameter n, m, i; p etc., and the random distribution of given node degree vector, and can carry out the Tornado coding.When receiving terminal is decoded, bigraph (bipartite graph) according to each grade, if there is a right node correctly to be received, and has only a node loss in all left siblings that are associated with it, this node of losing just can recover to obtain by this right node and left sibling that all are not lost so, has promptly reached the effect of error correction.
In fact, the scope of correcting and eleting codes is very big, and the Tornado sign indicating number is just wherein a kind of, also has in addition such as RS (Reed-Solomon) sign indicating number, loe-density parity-check code (Low Density Parity Codes is called for short " LDPC ") etc.
Important performance index of correcting and eleting codes are exactly its error correcting capability (perhaps being called protective capability); directly be presented as the maximum packet loss quantity (under the total prerequisite of certain bag) that to correct the packet loss mistake fully and be allowed; perhaps work as packet loss and be higher than under this maximum allowable quantity condition, can correctly correct the percentage of bag.In general, under other condition same cases, protective capability is high more, and redundancy rate is high more.
Protective capability is not only applicable to correcting and eleting codes, and in a wider context, all FEC codings can be measured with protective capability.In video data, some data relative importance height is such as the structural parameters of the structural parameters of video sequence, image, header etc.; The importance of other data is low relatively, such as picture material data etc.When using FEC to protect, for the stronger coding of important relatively The data protective capability; And for the more weak coding of unessential relatively The data protective capability.Can between protective capability and efficient, reach balance like this.Can not emphasize protective capability simply,, sacrifice efficient because can cause very high redundancy like this.This method of carrying out the FEC protection of different protective capabilities according to the data relative importance is called unequal loss protection (Unequal Protection is called for short " UEP ").By unequal loss protection, realize that easily the QoS of video communication services guarantees.
The thought of unequal loss protection is protected for the protection mechanism of the different protective capabilities of the The data that has different importance (relative) in the multi-medium data/protection intensity.Different protection mechanisms can refer to big class or group, and is different on principle such as big class, and group is only different on structure or parameter.Cascade protection is to be divided into a plurality of ranks for protection mechanism according to protective capability, and these ranks can be striden big class, also can stride group in a big class, promptly only is the difference on the structure and parameter.In network transmitted, network condition changed in time, and this Changing Pattern is complicated can't determining in advance; therefore; cascade protection is a kind of adaptive strategy in fact, and the situation instantaneous according to network selected optimum safeguard measure according to one group of predefined rule.Unequal loss protection and cascade protection are to combine, and form more complicated and strong protection strategy.
H.264 in the video communication, is very serious because the deletion error that packet loss etc. cause causes deteriroation of image quality in reality, more is far more than the collapse that causes the decoding end system.This is that the ability to bear for deletion error is also lower conversely because H.264 ability is stronger with respect to other video encoding standard, efficient is higher, function is abundanter.Therefore, in video communication, except error elasticity protection strategy, must adopt the technology of deletion errors such as effective anti-packet loss, and, guarantee to recover the quality of image in conjunction with the various video anti-error code method based on standard H.264.
Existing anti-packet loss mistake technology can be divided into two classes substantially: (a) initiatively prevent shift: take safeguard measure in advance, such as the introducing redundancy scheme, guarantee that as far as possible packet do not lose or guarantee the data that receiving terminal can recover to lose on a small quantity; (b) mistake offset-type: taking place to take certain indemnifying measure under the error condition, such as worsening in network condition under the serious situation, packet loss is very high, and initiatively error-preventing method loses effect, at this moment just need the error code that has taken place be compensated.
The error code removing method of mistake compensation is divided into error concealment again according to the emphasis difference and the error code diffusion is eliminated two kinds.Wherein, error concealment is to lay particular emphasis on the current influence of compensation error code, loses such as receiving terminal current video frame or Slice, and then image can't correctly show, promptly takes certain measure to compensate, the feasible minimum that influences for user's generation.It then is to eliminate error code to spread the subsequent affect of bringing on room and time that the error code diffusion is eliminated, the LOF or the partial loss of receiving such as receiving terminal, because this frame may be the prediction reference frame of subsequent frame, its error code will be diffused on time-domain in the subsequent frame; Perhaps because the infra-frame prediction that may exist H.264, and loop filtering, all may make the error code of this frame be diffused into other position of this frame by spatial prediction.It is exactly to adopt certain measure that the error code diffusion is eliminated, and spatially will limit the error code influence in finite region, will be limited in the finite time in time, avoids the video communication failure, more is far more than the disorderly and collapse of decoder system work.
As seen under the error code environment, the error code diffusion not only makes the recovery image quality decrease of the frame of makeing mistakes, and may cause expendable loss to subsequent frame, even decoding end has been used the error concealment technology, can not avoid recovering decrease in image quality.In addition, because the requirement of the real-time of video communication is strong, do not adopt the ARQ mode to retransmit the data that make a mistake usually.
By error concealment, promptly by simply replacing or complicated prediction with the correct data that the error code part spatially goes up adjacent part with the time takes place, the cover one's fault method of part such as interpolation can solve the compensation error code problem.This can realize at receiving terminal, need not transmitting terminal and participates in.Error code diffusion problem is then complicated more, needs transmitting terminal and receiving terminal to cooperate and takes suitable strategy to suppress and to eliminate.
It should be noted that error concealment also can cause the generation of error code diffusion.In fact, because error concealment can cause coding side and decoding end reconstructed image cache contents not to match, thereby cause the diffusion of error code on time-domain.For example, when decoding n-1 frame has packet loss, decoding end can use n-2 frame correspondence position view data to carry out error concealment, and when transmitting terminal is encoded and do not know that the n-1 frame has packet loss, can use the correct n-1 two field picture n two field picture of encoding, replace the n-1 frame decoding and when receiving terminal is decoded the n frame, use the n-2 frame, cause the error code diffusion thus.
Existing H.264/RTP transport architecture and based on the QoS method for reporting of RTCP adopts RTP directly to encapsulate NALU and transmits, and with the SR/RR report monitoring QoS information of RTCP, the correlation technique details has been introduced in the front.
In addition, the Tornado sign indicating number of available technology adopting is a kind of scheme of more complicated.Adopt the Tornado sign indicating number to realize that transmitting guard method based on the data of video compression coding H.261/H.263/H.263+/H.263++/H.264 comprises the steps:
Structure at step 1, design Tornado sign indicating number.Be specially: according to given protective capability, other requires, concrete data type such as audio frequency still be factor determination data node size such as video, speed size be the L1 bit, according to the scale factor that contracts of passing of interstitial content between the number of stories m of factor specified data interstitial content n, integrity checking interstitial content L such as acceptable maximum network delay in the concrete application, middle verification layer, each layer
Deng, determine left side degree distribution vector and the right degree distribution vector of the bigraph (bipartite graph) between the node of any adjacent two layers.
To step 2, adopt the mode of Random Graph coupling to generate bigraph (bipartite graph) at random between each adjacent two layers node according to above-mentioned left side degree distribution vector and the right degree distribution vector.
To step 3, beginning data transfer procedure, the H.261/H.263/H.263+/H.263++/H.264 video encoder of transmitting terminal produces data code flow, and this data code flow need carry out data and transmit protection.
To step 4, the data code flow that needs protection, be divided into equal-sized back end D with L1 bit
0, D
1, D
2, D
3... .D
T
To step 5, make t=0.
To step 6, obtain D
t, D
T+1... .D
T+n-1Be total to n data node.
To step 7, according to the bigraph (bipartite graph) at random between the definite adjacent two layers node of step 2 is a said n data node D
t, D
T+1... .D
T+n+1Successively generate each middle check-node layer MC
(0), MC
(1)... MC
(m)With terminal check node layer FC.
To step 8, with back end D
t, D
T+1... .D
T+n-1With check-node in each the check-node layer that generates by step 7 all by being sent to receiving terminal after the packing of network packing manner such as UDP/IP or TCP/IP.
Judge according to the numerical value of t, T whether all back end dispose to step 9, transmitting terminal, if all back end dispose, to step 10; Otherwise t=t+n is to step 6.
Communication process at step 10, transmitting terminal finishes.
Because transmitting terminal carries out on one side often that encoding compression sends in the real time communication process, be not temporal sequencing so above-mentioned steps 3 to step 8 represents, but sequencing in logic.
Said process is described transmitting terminal, for receiving terminal, after receiving terminal receives a collection of back end and check-node, at first should judge which back end be receive and in fact lost, and recoverable back end, then, for above-mentioned that lost and recoverable back end according to the recovery of decoding of the general decode procedure of Tornado sign indicating number.Receiving terminal repeats above-mentioned reception and recovery process, finishes up to communication process.
In addition, existing error code removing method all is an error concealment method or error code diffusion removing method independently, and a variety of realization ins and outs diverse ways are all arranged.Cover in the free territory of error concealment method, spatial domain is covered, space-time unite is covered etc.Error code diffusion is eliminated has in intraframe coding, sign, the adaptive frame piece to refresh etc. again.
The time-domain concealing method is exactly to adopt the information of frame adjacent on the time shaft to calculate obliterated data.The method of calculating can be: simply adopt the data of consecutive frame same position to replace obliterated data; Consider the motion prediction factor, carry out motion prediction according to the consecutive frame data.Remove this and also have the complicated more strategy of covering, but amount of calculation is very big.
The spatial domain concealing method is exactly to utilize the space adjacent area in obliterated data zone to carry out error concealment.Same has: simply substitute with the field; Calculate obliterated data based on a plurality of spaces of having of data fusion adjacent area, such as space interpolation; The algebraically method of inversion, with a linear model modeling, its input is data before the packet loss the packet loss process, output is the data that correctly receive, and utilizes the method for algebraically inverting, such as least square method, come the inverting input from output, substitute misdata with inversion result, this method amount of calculation is big.
The space-time unite concealing method then is the error concealment of associating usage space territory and time-domain.Such as, according to the characteristics of obliterated data and the situation of adjacent time data and spatial data, adopting certain strategy to determine to cover with spatial domain still is that time-domain is covered better, implements this strategy of better covering then.Perhaps, merge spatial data and time data, cover jointly.
Error code diffusion removing method based on intraframe coding is that the macro block that will influence by error code adopts intraframe coding, promptly utilize the forward direction dependence of motion vector to carry out accurate error code tracking, and the macro block employing intraframe coding to influenced by error code, can prevent the error code diffusion effectively.At first provide because the caused inter-frame dependencies of motion compensation; According to " energy " of the correlation calculations error code of motion vector forward direction dependence and weight factor, the macro block maximum to " energy " uses intraframe coding then, thereby prevents the error code diffusion.
Error code diffusion removing method based on sign is that the macro block that influenced by error code is done sign, avoids adopting the macro block that identified to do reference frame when making coding, has directly prevented diffusion.This method need be set up and retransmit the feedback mechanism of sending end to receiving terminal, receiving terminal arrives transmitting terminal with the feedback information of obliterated data, coding side all identifies with a specific value according to the error message pixel that error macro block in the same group is later, after the coding of some frames in not with reference to the zone that has identified, avoided the error code diffusion of receiving terminal.
The error code diffusion removing method that refreshes strategy based on piece in the adaptive frame is based on " the error code sensitivity yardstick " of coding side and weighs the vulnerability of each coded macroblocks to channel error code, carries out then that piece refreshes in the adaptive frame.This method does not need feedback channel.Coding side elder generation's initialization " error code sensitivity yardstick " value:, high more to the sensitiveness of error code apart from synchronous mark macro block far away more; The bit number of coded macroblocks is many more, and the easy more error code that is subjected to destroys.In cataloged procedure, upgrade this yardstick by the accumulation of calculating each macro block " error code sensitivity yardstick " value, select macro block to carry out intraframe coding according to the ESM yardstick then.
In addition, owing to do not have the scheme of network condition monitoring and the description of data relative importance can be provided easily, all do not realize multi-stage protection and unequal loss protection in the prior art.
In actual applications, there is following problem in such scheme: Tornado code plan of the prior art is too complicated, and efficient is low, is applied to the protection of video data, and time-delay is big, can't satisfy the performance requirement of real time communication.
Lack a kind of mechanism that can the reporting ent situation simultaneously; therefore; communicating pair can't be made a strategic decision according to network condition and be adopted suitable protection mechanism, thus unequal loss protection and adaptive hierarchical protection all can't effectively use, the reliability of multimedia communication can not reach requirement.
And the diffusion of error concealment and error code eliminates two kinds of methods and well do not unite, and conflicting sometimes, its effect is cancelled out each other.
In the prior art, H.264NAL the associated working mechanism with Real-time Transport Protocol lacks, and how based on H.264NAL providing not definition of protection mechanism with corresponding RTP method for packing, is a blank.Furthermore, good method can't be applied to H.264 video data such as high efficiency Tornado coding now with other safeguard measures and unequal loss protection and adaptive hierarchical protection etc.
Prior art does not have to utilize H.264 extension of message mechanism to realize the report of network condition and QoS information, lacks this mechanism, and a lot of good technology have not just had the prerequisite condition of using.
Prior art is all more scattered, lacks integratedly, and effect does not each other mutually strengthen.Simultaneously a lot of technology also rest on the scholastic pursuit stage, do not enter into the definition and the exploitation of communication protocol aspect, have influenced practical application.In the integration of these technology, must consider the constraint of real time communication performance requirement, the technology of selection must performance well be calculated too complexity simultaneously.
Cause the main cause of this situation to be, prior art can't adapt to network service and change with fixing correcting and eleting codes strategy protection video communication stream; The replacement mechanism that error concealment method adopts can cause the error code diffusion; Error code diffusion removing method all needs complicated mechanism or extra feedback channel, expends system handles resource and network bandwidth resources.
In the prior art scheme header of NALU is encapsulated in the middle of the payload fully, makes Real-time Transport Protocol can't directly know attribute, rank, significance level of relevant payload etc., thereby can't realize QoS mechanism based on this.Secondly, such encapsulation format has also caused the NALU header to take the payload resource because each NALU all attach header, cause under many circumstances, because the header of the NALU of a plurality of same types all is the same among RTP, thereby has wasted RTP transmission bandwidth resource.
H.264/RTP multimedia communication framework has adopted a kind of general cooperation control protocol RTCP to transmit the QoS report, to realize the QoS monitoring, yet RTCP itself is not necessarily only for H.264 such particular video frequency communications applications, because the logical channel of the band of itself reopening outward transmits the QoS report, influence network condition, caused the generation of contradiction.
Crucial place is that prior art is not provided by the error elasticity protection strategy of transport layer, the reliability and the communication quality that can't provide multimedia to transmit.
Summary of the invention
In view of this, main purpose of the present invention is to provide a kind of multimedia communication method and terminal thereof, makes transmission reliability and communication quality be improved.
For achieving the above object, the invention provides a kind of multimedia communication method, its communication process comprises following steps,
The A transmitting terminal is protected multi-medium data according to error elasticity protection strategy, and sends it to receiving terminal;
The described receiving terminal of B receives described multi-medium data, is occurring under the wrong situation of transmission, recovers or partly recover described multi-medium data by described error elasticity protection strategy.
Wherein, its communication process also comprises following steps,
The described receiving terminal statistics of C communication quality generates service quality report, and it is sent back to described transmitting terminal;
In the described steps A, described transmitting terminal is adjusted described error elasticity protection strategy according to described service quality report.
In this external described method, its communication process also comprises following steps,
The described receiving terminal statistics of D transmits error message, and implementation mistake is covered strategy;
Among the described step C, described receiving terminal also feeds back to described transmitting terminal with described transmission error message;
The described transmitting terminal of E is implemented the error code diffusion according to described transmission error message and is eliminated strategy.
In this external described method, described steps A comprises following substep:
The A1 transmitting terminal selects the error elasticity encoding scheme that multi-medium data is carried out forward error correction coding;
Multi-medium data after the described transmitting terminal of A2 is encoded with the encapsulation of error elasticity real time transport protocol, and in described error elasticity real time transport protocol header packet information, carry described error elasticity encoding scheme relevant information, send to receiving terminal then;
Described step B comprises following substep:
The described receiving terminal of B1 goes the error elasticity real time transport protocol bag of receiving to encapsulation, and extracts described forward error correction coding scheme relevant information from described error elasticity real time transport protocol header packet information;
If the losing of error elasticity real time transport protocol bag of back end correspondence taken place in B2 in transport process, so described receiving terminal is according to described error elasticity encoding scheme relevant information, select described decoding FEC scheme to carry out decoding FEC, recover or partly recover the described multi-medium data of losing; If losing of back end do not taken place, then do not need to carry out decoding FEC.
In this external described method, the multi-medium data behind the described forward error correction coding is divided into back end and check-node two classes.
In this external described method, in described steps A 1, described transmitting terminal is selected described forward error correction coding scheme according to one of following factor at least:
Current network transmission status and service of multimedia data credit rating to be sent, service of multimedia data credit rating wherein to be sent depends on the relative importance of different pieces of information.
In this external described method, comprise in the described error elasticity real time transport protocol header packet information:
Error elasticity real time transport protocol identification field is used for indication to be different from real time transport protocol;
The forward error correction coding type field is used to the forward error correction code type of indicating described forward error correction coding scheme to adopt;
The forward error correction coding sub-type field is used to indicate the relative parameters setting of described forward error correction coding scheme;
The data packet length field is used to indicate the length of described forward error correction coding scheme at the node that described multi-medium data is carried out obtain behind the forward error correction coding;
The number of data packets field is used to indicate the number of the described node that this error elasticity real time transport protocol bag carried.
In this external described method, in described steps A 1, described transmitting terminal is divided at least one isometric back end with at least one described H.264 network abstraction layer unit, then it is carried out forward error correction coding, obtains at least one check-node;
In described steps A 2, described transmitting terminal sends described back end and described check-node packet encapsulation at least one described error elasticity real time transport protocol bag;
In described step B1, described receiving terminal goes encapsulation to obtain described back end and described check-node after receiving described error elasticity real time transport protocol bag;
In described step B2, if the back end in the transport process has taken place to be lost, then described receiving terminal carries out recovering based on the recovery or the part of decoding FEC to the described back end of losing according to described check-node, and division obtains described H.264 network abstraction layer unit.
In this external described method, before beginning transmission, also comprise step,
Described transmitting terminal and described receiving terminal are consulted to determine: for various described forward error correction code types, the corresponding relation of the relative parameters setting of this kind forward error correction that the value of described forward error correction sub-type field is indicated with it.
In this external described method, described transmitting terminal and described receiving terminal are all set up mapping table according to the corresponding relation of described forward error correction coding sub-type field indication, are used for inquiring about pairing forward error correction coding or decoding FEC processing module according to described forward error correction coding type field and described forward error correction coding sub-type field;
In described steps A 1, described transmitting terminal calls corresponding forward error correction coding processing module and carries out forward error correction coding;
In described step B2, described receiving terminal calls corresponding decoding FEC processing module and carries out decoding FEC.
In this external described method, in described steps A 1, described transmitting terminal is according to the combination of network abstract layer reference identification field in the header of described H.264 network abstraction layer unit and any one or both in the network abstraction layer unit type field, and any other can predefined rule assess the relative importance of corresponding data, thereby determine described service quality rating, and then select described forward error correction coding scheme, determine described forward error correction coding type field and forward error correction coding sub-type field.
In this external described method, in described steps A, described transmitting terminal is estimated described network transmission status according to the transmission report of described receiving terminal feedback, and then selects described forward error correction coding scheme, determines described forward error correction coding type field and forward error correction coding sub-type field.
In this external described method, the version information field value in the described error elasticity real time transport protocol header packet information is binary value " 11 " or decimal value " 3 ", to be different from real time transport protocol;
Described forward error correction coding type field is positioned at after the tabulation of contribution source identifier, accounts for 4 bits;
Described forward error correction coding sub-type field is positioned at after the described forward error correction coding type field, accounts for 9 bits;
Described data packet length field is positioned at after the described forward error correction coding sub-type field, accounts for 11 bits;
Described number of data packets field is positioned at after the described data packet length field, accounts for 8 bits.
In this external described method, in the described steps A, divide together again, encode after at least one network abstraction layer unit that described transmitting terminal is identical with header is removed its header and be encapsulated into described error elasticity real time transport protocol bag, and with identical header that this network abstraction layer unit had comprehensively in the header of this error elasticity real time transport protocol bag;
Among the described step B, described receiving terminal obtains the header that is carried from the header of the described error elasticity real time transport protocol bag that receives, and add the head of the network abstraction layer unit of having peeled off header that extracts from described error elasticity real time transport protocol bag to, obtain whole network level of abstraction unit; Transmit mistake if exist, then have the decoding FEC of carrying out to recover or part restore data node, and then therefrom extract network abstraction layer unit according to presetting strategy.
In this external described method, in described error elasticity real time transport protocol header, network abstract layer reference identification field in the described network abstraction layer unit header and type field are filled in the payload type field of described error elasticity real time transport protocol header packet information, and this payload type field is positioned at back 7 bits of the 2nd byte of described error elasticity real time transport protocol header packet information.
In this external described method, described error elasticity real time transport protocol identification field is the version information field of described error elasticity real time transport protocol header packet information, and this version information field is positioned at preceding 2 bits of the 1st byte of described error elasticity real time transport protocol header packet information.
In this external described method, in described error elasticity real time transport protocol encapsulation format, the bit field of forbidding in the described network abstraction layer unit header is filled in the tag field of described error elasticity real time transport protocol header packet information, and this tag field is positioned at preceding 1 bit of the 2nd byte of described error elasticity real time transport protocol header packet information;
And in described step B, receiving terminal judges according to the tag field of described error elasticity real time transport protocol bag whether its network abstraction layer unit of carrying makes mistakes.
In this external described method, in described steps A, B, described error elasticity real time transport protocol is designated the tag field value of described error elasticity real time transport protocol header packet information, and this tag field is positioned at preceding 1 bit of the 2nd byte of described error elasticity real time transport protocol header packet information.
In this external described method, described steps A comprises following substep:
Described transmitting terminal at first judges to forbid whether bit field is effective in the header of at least one described network abstraction layer unit, in view of the above it is divided into proper network level of abstraction unit and the network abstraction layer unit of makeing mistakes;
By described error elasticity real time transport protocol encapsulation format described proper network level of abstraction unit package is become described error elasticity real time transport protocol bag then, and establish described error elasticity real time transport protocol sign;
By described real time transport protocol encapsulation format the described network abstraction layer unit of makeing mistakes is packaged into described real time transport protocol bag;
Described step B comprises following substep:
Described receiving terminal judges that at first whether the header of the bag that receives establishes described error elasticity real time transport protocol sign, is divided into described error elasticity real time transport protocol bag and described real time transport protocol bag with it;
Handle described error elasticity real time transport protocol bag according to described error elasticity real time transport protocol encapsulation format then, seal the described real time transport protocol bag of dress format analysis processing according to described real time transport protocol.
In this external described method, the following substep of described step C,
The described receiving terminal statistics of C1 generates described service quality report;
The described receiving terminal of C2 carries described service quality report with extended message H.264, issues described transmitting terminal.
In this external described method, described H.264 extended message is a supplemental enhancement information;
The encapsulation format of described service quality report in described supplemental enhancement information is as follows:
The 1st byte is load type field, and being used to indicate load is the corresponding with service quality report;
2nd, 3 bytes are payload length field, are used to indicate corresponding with service quality report length;
After the 4th byte reaches is load, is used to fill the corresponding with service quality report.
In this external described method, described service quality report is divided into transmit leg report and recipient's report, is distinguished by described load type field indication;
When described service quality report was filled in the load of described supplemental enhancement information, the load of described supplemental enhancement information comprised:
The version information field accounts for 2 bits;
Fill field, account for 1 bit, be used to indicate whether to have the filling content;
Receive the number of reports field, account for 5 bits, be used for indicating this service quality report to report and receive the report blocks number;
Transmit leg synchronous source identifier field accounts for 32 bits, is used to identify the transmit leg of this service quality report;
When described service quality report is transmit leg when report, also comprise the caller information piece, be used to describe the relevant information of the transmit leg of this service quality report;
Comprise at least one described reception report blocks, be used to describe from the multimedia statistical information of homology not;
Comprise specific aspect expansion, be used for the reservation function expansion of specific aspect.
In this external described method, the described supplemental enhancement information that is used to carry described service quality report is further carried by abstract network layer unit;
The reliability requirement that described communication terminal transmits according to described service quality report is provided with the network abstract layer reference identification of this abstract network layer unit.
In this external described method, described communication terminal is dynamically adjusted the statistics generation of described service quality report and the cycle that sends according to current network state and higher layer applications demand.
In this external described method, when described communication terminal mixes the service quality report of at least a Media Stream of carrying with described additional enhancing message,
Comprise the corresponding described reception report blocks of at least a Media Stream that is carried in this service quality report.
In this external described method, among the described step C, described receiving terminal is according to the network abstraction layer unit sequence number of the described video stream data that receives, and the described network abstraction layer unit number that statistics is lost generates described service quality report, sends back to described transmitting terminal;
In the described steps A, described transmitting terminal calculates described accumulative total packet loss according to described network abstraction layer unit sequence number of losing, and adjusts described error elasticity protection strategy in view of the above.
In this external described method, described receiving terminal is according to the service quality report that receives, analytical calculation network condition parameter; Described parameter comprises instant bandwidth, time-delay and shake end to end.
In this external described method, described transmitting terminal is provided with the error elasticity protection strategy series of different brackets, selects to use corresponding described error elasticity protection strategy according to described service quality report in described steps A.
In this external described method, among the described step C, described receiving terminal is added up the locating information that obtains losing video stream data according to the network abstraction layer unit sequence number of the described video stream data that receives, and it is sent back to described transmitting terminal;
In the described steps A, described transmitting terminal resends the described video stream data of losing and gives described receiving terminal according to described locating information of losing video stream data.
In this external described method, in described step e, described transmitting terminal obtains described locating information of losing band according to described transmission error message, carries out segmentation intraframe coding one by one by this being lost band, eliminates strategy to realize described error code diffusion.
In this external described method, described segmentation intraframe coding one by one comprises following steps,
E1 is cut apart one group of continuous macro block from described losing the band, form new band, and remaining described macro block still belongs to the described band of losing, and enters step e 2;
E2 carries out intraframe coding to described new band, sends when next frame, and this new band is done conventional coding after this, enters step e 3;
When E3 encodes at next frame, judge whether the described band of losing also comprises untreated macro block, if then return step e 1, otherwise finish intraframe coding.
In this external described method, in described step e 1, the size of each described one group of continuous macro block of separating satisfies: after this was organized continuous macro block and carries out intraframe coding, the data transfer rate of this frame was in data transfer rate control range H.264.
In this external described method, described step D comprises following substep,
The described receiving terminal of D1 detects and transmits mistake, and statistics transmits error message;
The described receiving terminal of D2 carries out video information and weighs synchronously after taking place to transmit mistake;
The described receiving terminal of D3 is implemented described error concealment strategy according to described transmission error message.
In this external described method, among the described step D1, described receiving terminal detects and adds up the transmission error message according to the discontinuous situation of network abstraction layer unit sequence number.
In this external described method, in described step D1, the locating information that described receiving terminal is lost band according to the interruption situation acquisition of described network abstraction layer unit sequence number, this locating information comprises described frame number and described position of losing band at this frame of losing the band place.
In this external described method, described error concealment strategy comprises step: described receiving terminal substitutes this and loses band with described respective strap of losing the former frame of band place frame.
In this external described method, described error elasticity encoding scheme comprises improved " Tornado " correcting and eleting codes;
Described improved " Tornado " correcting and eleting codes only generates the described check-node of one deck for one group of described back end.
In this external described method, the transmission mistake among the described step B comprises data-bag lost or random bit mistake.
The present invention also provides a kind of multimedia communication terminal, comprises the basic function module that is used to realize multimedia communication, wherein comprises and comprises the coding/decoding module that is used to realize the multi-media decoding and encoding function at least, also comprises with lower module:
Error elasticity implements to transmit the control protocol module; be used for to be undertaken by the described multi-medium data behind the described coding/decoding module coding transmitting at network side again after the error elasticity protection, the described multi-medium data from network side carried out passing to described coding/decoding module again after the error correction decode.
Wherein, also comprise with lower module:
Guard method and policy conferring module are used to be responsible for carry out error elasticity protection policy conferring between communicating pair, determine the protection strategy set, implement to transmit the control protocol module for described error elasticity and select;
Forward error correction block; be used to realize at least a forward error correction guard method; safeguard the relevant parameter of described forward error correction guard method; wherein said guard method and the described forward error correction block of policy conferring module controls are to realize unequal loss protection and adaptive hierarchical defencive function, and described error elasticity is implemented to transmit the control protocol module and realized error elasticity protection and error correction by calling this forward error correction block.
In addition, also comprise
The error concealment module is used to realize the error concealment function;
Described coding/decoding module is used to realize H.264 encoding and decoding standard, also is used for the error code diffusion and eliminates function;
Also comprise network condition analytical calculation module, be used for the analytical calculation network condition, and provide information to described error concealment module and described coding/decoding module.
In addition, also comprise
Replenish enhancing extension of message processing module, be used to realize service quality report and network condition function of reporting, and report is sent to described network condition analytical calculation module.
In addition, its transport layer is used to realize supporting the multimedia transmitting function of error elasticity based on described error elasticity real time transport protocol/transmit control protocol in real time;
Its application protocol layer comprises protection mechanism and policy conferring sublayer, is used to realize cascade protection and unequal loss protection function;
Its H.264 the video coding layer comprise and replenish to strengthen extension of message report layer, be used to realize the function of reporting that strengthens extension of message based on replenishing;
H.264, it comprises the forward error correction coding layer in network abstract layer, is used to realize the forward error correction coding function.
In addition, the described basic function module that is used to realize multimedia communication comprises one of following or its combination in any:
Main control module is used for being responsible for the control of whole terminal;
Subscriber Interface Module SIM is used for the demonstration of the mutual and information of responsible user's input and output;
Network communication module is used for being responsible for and network communicates, and provides lower floor to transmit passage;
Input and output and bottom layer driving module are used for being responsible for driving for hardware device;
Business module is used to realize high-level business;
The communication course control module is used to control communication process;
The application protocol module is used to realize the application protocol function;
Implement to transmit the control protocol module, be used for realizing implementing to transmit the control protocol function;
H.264 the network abstract layer module is used to realize the network abstract layer function;
The audio coding decoding module is used to realize the audio coding decoding function.
By relatively finding, the main distinction of technical scheme of the present invention and prior art is, adopted error elasticity real time transport protocol (ERRTP), on existing RTP basis, provide the transport layer encapsulation format that can carry error elasticity encoding scheme relevant information, make its corresponding error resilient coding scheme information of same tense marker that multi-medium data transmits on ERRTP, thereby error resilience mechanism is incorporated transport layer;
Provide the special-purpose ERRTP method for packing and the modification scheme of protocol header information at structure H.264NALU, be attached in its header by header byte all NALU in the same ERRTP bag, adopted a kind of combination cleverly to make neither influence have the running of ERRTP agreement and equipment now, and the attribute of NALU payload can be embodied directly in the ERRTP header, make load-carrying efficiency improve greatly on the one hand, the basis that provides QoS mechanism to realize on the other hand;
Based on extension of message mechanism H.264, add up communication quality and feed back to transmitting terminal by receiving terminal, directly adopt the H.264 extended message mechanism of high-rise media protocol own to carry the QoS report information, avoid using extra channel, realized a kind of " in the band " QoS report mechanism;
Can also select to adopt various standby error elasticity encoding schemes according to factors such as current network conditions and multi-medium data importance rates at transmitting terminal, thereby reach the purpose of unequal loss protection, realize the equilibrium of protective capability and transmission efficiency;
On the basis of the feedback mechanism from the receiving terminal to the transmitting terminal, realize using alternately mixing of unequal loss protection and multiple error elasticity scheme, transmitting terminal is according to the QoS report and the network of relation transmission status message of receiving terminal feedback, select to use the protection strategy of different brackets, based on data importance grade, also can select the data of different brackets are used suitable protection strategy in addition from the reflection of ERRTP header;
At data flow H.264NALU, provide the scheme that combination is eliminated in the diffusion of error concealment and error code, comprehensively embody the advantage of two kinds of technology, by error code feedback information mechanism and segmentation one by one intraframe coding realize error code diffusion elimination;
A kind of code plan of Tornado efficiently also is provided, guaranteeing that data transmit protective capability and do not have under the situation of significantly decline, by the correcting and eleting codes that only has one deck check-node layer is set, reduced the operand of correcting and eleting codes generation check-node layer, reduced the data transfer delay time, made data transmit protective value and be improved with the cost ratio;
At last, the enhancement techniques that above-mentioned various multimedia communications are relevant is incorporated on the multimedia communications system, and modular implementation various technology and protocol infrastructure, the mutual co-ordination of various technology further strengthens the multimedia communication reliability each other.
Difference on this technical scheme has brought comparatively significantly beneficial effect, and promptly ERRTP agreement transport architecture is simplified the error elasticity transfer structure greatly at transport layer realization error resilience mechanism, has saved the network transmission bandwidth; The realization of unequal loss protection has reached the equilibrium of protective capability and transmission efficiency, is convenient to the realization of the QoS assurance of multimedia transmission, further improve service quality, reduce redundant, improve transmission efficiency, realized the compatibility with prior art, all improved the robustness of this new method of ERRTP;
QoS report based on H.264 extension of message mechanism, in band, realize the QoS monitoring, reduce bandwidth cost, and the complexity of reduction system realization, improve the present H.264 effect and the efficient of the report mechanism of video network delivery quality, thereby promote H.264 video network delivery quality;
Unequal loss protection and multi-stage protection strategy adapt to network more flexibly, accurately and timely and transmit demand, improve protective capability, improve system effectiveness and reliability, can guarantee that statistical information is accurately errorless and save system resource;
Eliminate in conjunction with error concealment and error code diffusion, the error code of avoiding being caused by error concealment spreads, and under simple complexity prerequisite, reaches desirable error code and eliminates effect, improves the video delivery quality, pares down expenses, simplifies mechanism, and guarantee system compatibility;
Use improved Tornado correcting and eleting codes scheme, improve data and transmit the protectiveness price ratio, improve data-transmission efficiency, promote H.264 to wait new technology to use;
With multiple enhancement techniques comprehensively in multimedia communications system, the common multimedia communication quality that improves, can improve performance and the user experience used on IP network such as video conferencing, video telephone based on multimedia communication product H.264 greatly, improve product competitiveness, bring remarkable economic efficiency, limitless practical value is arranged.
Description of drawings
Fig. 1 is the header structural representation of RTP packet;
Fig. 2 is the encapsulation format schematic diagram of RTP bag payload to the NALU data;
Fig. 3 is based on the QoS report data packet format schematic diagram of rtcp protocol;
Fig. 4 is a Tornado correcting and eleting codes principle schematic;
Fig. 5 is the support error elasticity multimedia communication terminal modular structure schematic diagram of first embodiment of the invention;
Fig. 6 is the multimedia communication protocol stack architecture schematic diagram of first embodiment of the invention;
Fig. 7 is the header structural representation according to the ERRTP packet of second and third execution mode of the present invention;
Fig. 8 is the SEI encapsulation format schematic diagram according to the carrying QoS report of the 4th execution mode of the present invention;
Fig. 9 be according to the 6th execution mode of the present invention based on segmentation one by one the error code diffusion of intraframe coding eliminate principle schematic.
Figure 10 is a correcting and eleting codes structural representation of the present invention.
Embodiment
For making the purpose, technical solutions and advantages of the present invention clearer, the present invention is described in further detail below in conjunction with accompanying drawing.
The present invention comprehensively realizes various enhancement techniques on a multimedia communications system, the advantages separately of various enhancement techniques is improved systematic function jointly, transmits reliability and communication quality.These enhancement techniques comprised the error elasticity that FEC is blended in Real-time Transport Protocol implement transportation protocol (ErrorResilience Real-time Transport Protocol is called for short " ERRTP "), with the NALU header be incorporated into RTP packet header technology, adopt the feedback technique of SEI extended message carrying QoS report and network condition, the multi-stage protection and the unequal loss protection mechanism that realize based on feedback, adopt error concealment and diffusion to eliminate the technology and the improved Tornado encoding scheme of combination.
The present invention is with after the various enhancement techniques modularizations; be combined in the multimedia communications system; realize H.264 video communication of error elasticity; this system comprises general main control module; user interface; network communication module; I/O and bottom layer driving module; the miscellaneous service module; the communication course control module; application protocol modules etc. also comprise the guard method and the policy conferring module that realize various enhancement techniques; the FEC module; the ERRTP module; the RTCP module; H.264NAL module; H.264 coder module; H.264 decoder module; the audio coding decoding module; the error concealment module; SEI extension of message processing module; network condition analytical calculation module.
In the first embodiment of the present invention, multiple enhancement techniques is realized that also modularization comprehensive is in a multimedia communications system, mainly be meant multimedia communication terminal, at first carry out the realization description of this device below from each composition function module of terminal, a complete terminal inner modular structure figure as shown in Figure 5.Should illustrate that said functional module all defines from function here, concrete implementation can be software, hardware, firmware (firmware) and hardware and software hybrid mode.A complete multimedia communication terminal at first must comprise with lower module:
Main control module: be responsible for the control of whole terminal system;
User interface (perhaps crying the interface) module: be responsible for the mutual of user's input and output, the user operates by Interface Control element such as menu button etc., shows feedback information simultaneously, such as current system mode, and parameter, network condition etc.;
Network communication module: be responsible for and network communicates, TCP is provided, UDP, IP and the more communication protocol stack such as the Ethernet of lower floor, PPP, ATM etc.;
I/O and bottom layer driving module: be responsible for driving for hardware device, such as video, video, the input and output of voice data are responsible in the driving of audio collecting device and display/play equipment simultaneously;
The miscellaneous service module: realize various concrete business, such as video telephone, Multi-Party Conference, video-mail, timely message, Video chat or the like;
The communication course control module: in concrete communication process, control, such as in Multi-Party Conference, realizing request Chair, release Chair, request floor, browse etc. in certain meeting-place of control broadcasting, meeting-place;
Application protocol module: can be concrete application protocol such as system (comprise under it H.225.0, RAS, H.245, H.235, H.460 etc.) and SIP H.323.In general, this agreement is the general name of series of protocols, is called " agreement umbrella " (protocol umbrella);
Corresponding to various enhancement techniques, in lower module, realize respectively in addition:
Guard method and policy conferring module: this module is responsible for carrying out guard method and is consulted between communicating pair, determines to allow set, according to allowing set, consults the strategy that one group of guard method mixes and is used alternatingly then.Consult to communicate and finish by " application protocol module ".This module controls FEC module, the latter realizes different FEC protected modes, functions such as unequal loss protection and adaptive hierarchical protection;
The FEC module: this module is supported multiple FEC guard method, and they can belong to a plurality of big classes as subclass, supposes to support altogether T kind diverse ways.According to the result's (from " guard method and policy conferring module ") who consults, for H.264 video data and voice data (not in this patent scope) are protected.This inside modules has been preserved the create-rule and the parameter of various FEC subclass correspondences, therefore contains the database of an inside, is used to store these data.This module can realize the mixing and the alternate application of different guard methods;
ERRTP module: realize the ERRTP agreement, go encapsulation step also can describe in detail in the following embodiments about protocol encapsulation form and the correspondence relevant encapsulation H.264 of ERRTP;
RTCP module: realize normal RTCP function, though the invention provides based on the report mechanism of SEI extension of message H.264, can realize the report of main RTCP information, but do not get rid of the use of RTCP, two kinds of report mechanisms can and be deposited, consider compatibility and interoperability if decide like this, so distant terminal may not support to adopt SEI extension of message report mechanism;
H.264 NAL module: realize the H.264 function of network abstract layer;
H.264 coder module: except realizing normally H.264 encoder functionality, realized that also error code diffusion of the present invention eliminates function, so according to information from " network condition analytical calculation module ";
H.264 decoder module: realize normally H.264 decoder function;
The audio coding decoding module: realize the codec functions of audio frequency, the agreement of support can be ITU-TG.711, G.722, and G.723.1, G.728, G.728, G.722.2 (3GPP AMR), the MP3 of MPEG tissue, AAC or the like;
Error concealment module: realize error concealment function provided by the invention.The information of institute's foundation is from " network condition analytical calculation module " and " H.264 coder module ";
SEI extension of message processing module: realize QoS and the network condition function of reporting based on the SEI extension of message of the present invention,, collect data and form RTCP SR that the RR report sends by the encapsulation of SEI extended message then at transmitting terminal; Extract RTCP SR at receiving terminal from the SEI extended message, the RR report is issued these data " network condition analytical calculation module " then and is analyzed and calculate;
Network condition analytical calculation module: according to data from " SEI extension of message processing module ", carry out analytical calculation and obtain the network condition data, such as packet loss, shake, time-delay, up time is bandwidth (throughput) or the like end to end, then, control " H.264 coder module " and " error concealment module " with these data, also these data are sent to " Subscriber Interface Module SIM " simultaneously and can show and see to the user.
After integral body is understood the module formation of communication terminal, come again to describe this terminal from the level aspect of protocol stack.A communication terminal system has been realized the agreement of a plurality of different aspects, and these agreements have constituted protocol stack (Protocol Stack).For terminal of the present invention, its protocol stack has identical place with common multimedia communication terminal, and different places is also arranged, and has increased some new levels in some place.Fig. 6 shows the multimedia communication protocol stack architecture schematic diagram according to the first embodiment of the present invention.
H.264/ERRTP multimedia transport architecture and the difference of traditional H.264/RTP framework of the present invention mainly is:
Replaced RTP/RTCP layer in the general terminal protocol stack with the ERRTP/RTCP layer, use the ERRTP that supports error elasticity after, the error elasticity protection mechanism is combined in transport layer realizes;
Increased by one " protection mechanism and policy conferring layer " in application protocol layer, this one deck is mainly used in communicating pair and consults various protection class and relevant protection scheme thereof when realizing multi-stage protection with unequal loss protection;
H.264 increasing " SEI extended report layer " between VCL layer and the NAL layer, this one deck is convenient to receiving-transmitting sides and realizes feeding back based on the QoS monitoring and the network transmission status of SEI extended message;
H.264 increasing " FEC layer " between NAL layer and the ERRTP/RTCP layer, this one deck has been realized node division, coding and the encapsulation for H.264 NALU data flow.
Those of ordinary skill in the art are appreciated that, among first embodiment of the invention described above with typically H.264 business be that example provides basic module structure and protocol stack and forms, for other agreements or following multimedia communication protocol or the application that occurs, only need on principle of the present invention basis, to realize the correlation technique details according to concrete application, reach goal of the invention, do not influence the spirit and scope of the invention.
Under the prerequisite that provides the entire system framework, the realization details of each enhancement techniques will be described successively below.
Problems at the prior art existence, the present invention proposes a kind of Real-time Transport Protocol of improved support error elasticity, be intended to error resilience mechanism is incorporated transport-layer protocols, not only can simplify transfer structure and reduce complexity, and can also improve error resilience mechanism flexibility enhancing and transmit reliability.Owing to have error elasticity, the present invention claims that this improved Real-time Transport Protocol is error elasticity real time transport protocol (ErrorResilience Real-time Transport Protocol is called for short " ERRTP " or " ER2TP ").The main distinction of ERRTP and RTP is that ERRTP protocol data header packet information expansion can carry error elasticity encoding scheme relevant information, such as FEC type, protective capability, coding parameter etc.
On the ERRTP basis; the present invention has realized unequal loss protection easily; at first provide the different safeguard measure of kinds of protect ability available use; after transmitting terminal obtains information such as network condition and multi-medium data importance in collection then; can select suitable safeguard measure according to these factors; thereby reach the purpose of unequal loss protection, realize the equilibrium of protective capability and transmission efficiency.Owing to all carried the FEC relevant information that it adopted on each ERRTP packet, so transmitting terminal only needs the information of selected scheme is inserted in the ERRTP header packet information, and receiving terminal just can correctly recover or error correction according to it.
At last, provided specific implementation method, comprised divisions, generations, encapsulate the step of reconciliation encapsulation of data node and check-node based on the correcting and eleting codes protection for H.264 NALU data-transfer applications.Continuous a string NALU waited together be divided into several back end longways, produce check-node with the Tornado sign indicating number then, all these nodes are distributed in again in several ERRTP bags and transmit, and receiving terminal then carries out this inverse process.
The second embodiment of the present invention is on the basis of first embodiment, and receiving-transmitting sides is realized unequal loss protection based on ERRTP, and key step is as described below:
Transmitting terminal is selected the error elasticity encoding scheme that multi-medium data is entangled and is deleted coding, with the multi-medium data behind the ERRTP encapsulation coding, and carries error elasticity encoding scheme relevant information in the ERRTP header packet information, sends to receiving terminal then;
Receiving terminal is with the ERRTP bag decapsulation of receiving, and from the ERRTP header packet information, extract error elasticity encoding scheme relevant information, according to error elasticity encoding scheme relevant information, select the error elasticity encoding scheme to carry out the error elasticity decoding then, obtain multi-medium data.
Wherein, unequal loss protection is embodied in transmitting terminal and selects the error elasticity encoding scheme according to current network transmission status and/or service of multimedia data credit rating to be sent.
At first introduce the concrete structure of ERRTP, provide the header structure embodiment of concrete ERRTP below.Fig. 7 is the ERRTP header structural representation according to the first embodiment of the present invention.As can be seen from the figure, version information field V value is 3, and expression ERRTP agreement is to be different from traditional Real-time Transport Protocol (V=2).Wherein just last with relevant information field about the error elasticity encoding scheme in the header expansion, comprise in this example: error elasticity type of coding field, error elasticity coding parameter field, data packet length field, number of data packets field.
Error elasticity type of coding field, the correcting and eleting codes type that is used to indicate the error elasticity encoding scheme to adopt also can be called the FEC the type field, promptly indicate the FEC type of coding, account for 4 bits, can represent 16 kinds of different FEC types, from practical application, be enough.Here Ding Yi type is big type in fact, and the back also is subdivided into various scheme with continuing, and is called subtype, the major types in the practical application for example: 0010 expression Tornado sign indicating number, 0011 expression RS sign indicating number etc.This field can identify 16 kinds of different FEC sign indicating number major types, and table look-up (Look-Up Table is called for short " LUT ") of corresponding relation is called FECTypeLUT between communicating pair needs FEC type of coding of prior agreement and the type of coding code name.
Error elasticity coding sub-type field is used to indicate the relative parameters setting of error elasticity encoding scheme, needs also to determine that for every type FEC coding being provided with of various parameters could specifically implement, and this field is exactly to play the effect of clear and definite concrete parameter.Because resource-constrained in the ERRTP header can not be enumerated pairing concrete parameter of various FEC encoding schemes and rule thereof etc. one by one, the first embodiment of the present invention is by indicating the various alternative parameter plans of establishment with the notion of subtype.This field is also referred to as FEC coding sub-type field, and FEC Subtype accounts for 9 bits.This territory mainly is illustrated in below each major types that defines among the FECTypeLUT the further subtype of segmentation.
The data packet length field is used to indicate the error elasticity encoding scheme in the back end length of multi-medium data being entangled after deleting coding, is called Data Length field, accounts for 11 bits.Because each data packet length should transmit maximum transmission unit (Maximum Transport Unit less than network, be called for short " MTU "), and present wire message way MTU＜1500=0x5DC byte, wireless channel MTU＜100 bytes, so 11 bits of this field are enough to the length of store data bag.
The number of data packets field is used to indicate this ERRTP to wrap the number of institute's data carried by data node, is called Packet Number field again, account for 8 bits, such as for several NALU through forward error correction code checks after, packet encapsulation in a plurality of ERRTP, institute's data carried by data node number among each ERRTP.
As seen had after these fields, the FEC code type that decoding end or network node can provide according to this field and the verification type of packet are carried out verification to the packet that receives, and recover the packet lost.
It should be noted that subtype FEC Subtype field above-mentioned totally 9 bits be to be used for encoding the various alternative parameter plans of establishment of indication, just provide the ins and outs of the indication of how encoding in the first embodiment of the present invention below.
At first receiving-transmitting sides need consult to determine this field indexical relation correspondence table.Before beginning transmission, transmitting terminal and receiving terminal are consulted to determine: for various FEC sign indicating number major types, the corresponding relation of the relative parameters setting scheme of this kind FEC sign indicating number that the value of FEC Subtype is indicated with it, and the concrete parameter of various alternatives is provided with situation.
Then, transmitting terminal and receiving terminal are all set up mapping table according to negotiation result, are used for inquiring about pairing FEC type of coding or FEC encoding and decoding processing module according to FECType and FEC Subtype field;
In the transmitting-receiving process, transmitting terminal calls corresponding entangling and deletes the encoding process module and entangle and delete coding, and receiving terminal calls corresponding entangling and deletes decoding process module and entangle and delete decoding.
In actual applications, in fact the information of subtype indicate two aspects:
The create-rule (Generation Rule) of A.FEC coding;
B. protect intensity/protective capability.
So-called create-rule is exactly how back end to be handled rule or the algorithm (Algorithm) that generates each check-node at transmitting terminal.Certainly receiving terminal do just in time opposite, if packet loss has taken place in transport process, i.e. some node loss can be recovered according to create-rule so or part is recovered the node lost.As seen create-rule is very important information, and according to it, the both sides of communication just can come work based on FEC mechanism.Each class in the FEC type of listing in FECTypeLUT all has different create-rules; And in each class, such as the Tornado sign indicating number, the create-rule of following subclass also will be in conjunction with concrete generation parameter (generation parameters).Therefore specific to each subclass here, claim rule will with generate parameter and combine.
Such as for the Tornado sign indicating number, generate parameter and comprise following data: back end sum, check-node sum, the check-node number of plies, two-layer in succession between the economize on electricity number pass the ratio of contracting, expression two-layer in succession between the incidence matrices of node incidence relation, if L layer check-node arranged, so such incidence matrices just has the individual or equivalent expression of the L parametrization mathematical notation of the bigraph (bipartite graph) of two-layer node incidence relation (Bipartite) (parametric mathematical representation) in succession.
In general, under the identical prerequisite of big create-rule, generate the often protection intensity of determinant type of parameter.Such as the Tornado sign indicating number, in the every generation parameter that provides in the above, back end sum and check node sum can determine protective capability (certainly strictly speaking, determine protective capability fully, need whole generation parameters) basically to a great extent.In the present invention, for each FEC major types, the major parameter (decisive action maximum) of selecting some decision protective capabilities is as the representative parameter (representative generation parameters) that generates.By using the representative parameter that generates, just can line up the subclass below the big class according to protective capability order (ascending order) from weak to strong.Thereby set up a LUT and be called FECSubTypeLUT.
Mask body is supported a plurality of subtypes under each major types, the concrete application and the communication capacity of communicating pair (factors such as CPU processing speed, internal memory, program complexity) can be arranged and need decision.If communication environment alters a great deal, the performance inconsistency scope of network is very big, and in general the subtype that needs so to support want many, on the contrary can be less.This fully can be in handling capacity negotiations process before the communication beginning, the agreement of being reached an agreement by communicating pair.Negotiation can pass through H.323 or the multimedia communication framework agreement of session initiation protocol present main flows such as (SessionInitial Protocol are called for short " SIP ") is carried out.
Suppose at the subclass below certain big class, distinguish S subtype (S≤2 if desired
9-1), the representative parameter that generates has k, uses p
1, p
2..., p
kExpression, table 2 provides the example of a corresponding relation so, and subscript is represented the FEC major types in the table, and subscript is represented concrete which parameter.
Table 2FEC Subtype and parameter plan of establishment mapping table
| FEC?Subtype | The FEC subtype (parameter setting) of encoding |
| 000000000 | FEC subtype 0 (p 0 1，p 0 2，..，p 0x) |
| 000000001 | FEC subtype 1 (p 1 1，p 1 2，..，p 1 k) |
| 000000010 | FEC subtype 2 (p 2 1，p 2 2，..，p 2 k) |
| 000000011 | FEC subtype 3 (p 3 1，p 3 2，..，p 3 k) |
| ................. | .................. |
| S(S≤2 9-1) | FEC subtype S (p S 1，p S 2，..，p S k) |
Such as, for the Tornado sign indicating number, corresponding relation can be set be: 000000010-(24,20) (back end sum=20, check-node sum=4), 000000011-(30,20) ..., 111111111-other.
At the subtype of the FEC of certain specific character coding, one group of given create-rule is in conjunction with the corresponding corresponding unique encoding scheme of parameter that generates, and how promptly unique the decision generates check-node by back end, and the node how to recover to lose.Can set up a database, store the generation parameter of every kind of major types and subtype correspondence.And create-rule itself is realized with hardware or software module.Therefore, every kind of major types is responsible for generating check-node in the corresponding FEC processing module of transmitting terminal; In the same corresponding FEC processing module of receiving terminal, be responsible for recovery nodes.But the module of corresponding every kind of major types need read the generation parameter of concrete every seed category, thereby handle from above-mentioned generation parameter database.Therefore, communicating pair all is that information according to FEC Type and two information fields of FEC Subtype decides to call which FEC processing module and which reads and generates parameter.
Because the development of multimedia communication technology at present, H.264 video encoding standard becomes mainstream media's coded format gradually, therefore the second embodiment of the present invention is on the basis of first embodiment, provided the concrete steps of NALU data flow H.264 being carried out the FEC encoding and decoding with ERRTP, its flow process is as described below.
H.264NALU, transmitting terminal is merged into one group of unification transmissions of encoding with a plurality of (are assumed to be S), earlier S NALU is reclassified as isometric piece, and it is individual to be assumed to be M, and this M is individual to be exactly back end.
In this step, H.264 S NALU is divided into one group; With S NALU end to end (concatenated), be connected to form a bulk then, then this bulk be divided into M data block, wherein the length of each data block is K byte.If total byte number (being made as TB) of this bulk can not be divided exactly by M here, should carry out rounding operation so, make that the length of each data block is Ceiling (TB/M) byte, the Ceiling function representation rounds, be the smallest positive integral that Ceiling (x) equals to be not less than x, x is any real number.Back in some data block is adopted the operation of filling zero string (zeropadding) possibly so, makes the byte number Ceiling (TB/M) that gathers together enough).
Then, it carries out the FEC coding to M data node, obtains N check-node.Use FEC sign indicating number coding to generate N check block to M data block, the method that generative process adopts the front to describe according to FEC Type and FEC Subtype information, is determined to call concrete which FEC processing module and is carried out the generation of check block.
Then, transmitting terminal sends all back end and check-node packet encapsulation in the ERRTP bag.Each field should be by following setting in this example:
Type field FEC Type=0010, the Tornado sign indicating number is used in expression;
Sub-type field is then specifically selected according to actual conditions by transmitting terminal, is FECSubtype=000000010 such as value, and Tornado (24,20) sign indicating number is used in expression, and wherein back end is 20, and 4 of check-nodes, code redundancy degree are 16.7%; This correcting and eleting codes smaller or equal to 3% o'clock, can recover the packet of losing at packet loss fully;
Data packet length Data-Length=K Bytes;
Number of data packets Packet Number=(M+N)/P represents data carried by data node number in the ERRTP load.
Receiving terminal is after receiving these ERRTP bags, and decapsulation obtains back end and check-node.Receiving terminal is the cycle with P packet, whenever receives one group of P packet and just begins the recovery of once decoding.One group of what packet is consulted is by both party determined.
Receiving terminal carries out the error elasticity decoding according to check-node to the data node.After receiving packet P+1, begin to detect in P the packet of receiving previously whether data-bag lost is arranged at every turn, if have and just adopt previously described method, according to FEC Type and FEC Subtype information, determine to call that concrete which FEC processing module is decoded and the data of recovery or partial loss.
After obtaining complete back end, reconsolidate and just obtain a bulk at last, adopt the mode identical, divide and obtain S NALU with transmitting terminal.
Find that in actual applications last example adopts the anti-data-bag lost algorithm based on ERRTP, can under situation about increasing, improve the anti-data-bag lost ability of video code flow greatly less than 17% code word.And compare with RTP payload header structure, only increased by 4 bytes, as seen transmission efficiency there is not influence substantially, obtained significant actual effect.
Having mentioned about another one key technology point of the present invention in front is exactly the realization of unequal loss protection.Be mainly reflected in two aspects, one is to select suitable encoding scheme or parameter according to the multi-medium data of different important level, promptly determines aforementioned FEC type of coding and subtype, and another is exactly to select according to difference network condition constantly.Corresponding these two aspects are called mixing and are used alternatingly various FEC encoding schemes.So-called mix (Hybrid) uses multiple FEC subtype in being meant at one time simultaneously, is mainly used in the data of the different importance of protection; And so-called alternately (Alternation) is meant at different time (under the different network conditions) and uses different FEC subtype.
Therefore for data flow H.264NALU, the front is mentioned, its byte has embodied the significance level of data, therefore transmitting terminal can be assessed the QoS grade according to NRI field in the header of NALU or the type field, and then select the error elasticity encoding scheme, promptly determine FEC the type field and FEC Subtype field.And for network condition, general network transmits all corresponding network condition monitoring mechanism, transmitting terminal can be known the transmission report of receiving terminal feedback according to these mechanism, with this evaluating network transmission status, and then select the error elasticity encoding scheme, promptly determine FEC the type field and FEC Subtype field.
H.264 code stream is based on NALU and transmits or store, and NALU is made up of NAL header and NAL load.In NALU H.264, different N ALU type is recovered the difference that influences of image to decoding.For example, NRI gets Slice or the Slice data band that non-reference picture is deposited in 0 expression among the NALU, can not influence subsequent decoding; Show Slice or the Slice data band of depositing a sequence/picture parameter set or reference picture among the NALU and NRI gets non-0, can have a strong impact on subsequent decoding.
Therefore, when H.264 code stream being carried out the packet protection, can data H.264 be divided into two classes according to the value of NRI or Nal_unit_type: a class is relative important images data (for example Nal_ref_idc equals 1); Another kind of is less important view data (for example Nal_ref_idc equals 0).Then, use redundancy FEC1 sign indicating number big, that anti-packet loss ability is strong to protect to important view data; And less important view data can use the FEC2 sign indicating number that redundancy is less, anti-packet loss ability is more weak to protect.
By this unequal loss protection algorithm, guaranteed the correct recovery of all kinds of important informations under high data-bag lost environment, adopt error concealment and prevent technology such as error code diffusion and the FEC2 sign indicating number is still failed image restored information.FEC1, FEC2 are general method for expressing here, represent any two seed categories.This two seed category can belong to same major types, also can belong to different major types.
Obviously, said method can be generalized to situation more generally, and data are divided into more multiclass according to the value of NAL_unit-type, such as five classes: most important data, inferior significant data, general significant data, more inessential data, significant data least; Also can be divided into 7 classes or more, so, can protect, the corresponding a kind of different subtype of every class data with the FEC subtype of equal number.As long as protective capability is just passable from weak to strong, these subtypes not necessarily belong to same major types.And adopt error concealment and prevent technology such as error code diffusion still failing image restored information after the protection of the strongest FEC sign indicating number of protective capability.
Another situation of unequal loss protection also within the scope of the present invention, exactly can according to the network real time status select different protective capabilities FEC.Come the both sides of notifying communication then by the header of ERRTP, make the data that they can correctly be decoded and recover to lose data.Can be divided into several levels to the situation that the current conveying function that is affected of network descends.Such as Pyatyi: the most serious, inferior seriously, generally serious, more not serious, least serious; Also can be divided into 7 grades or more, so, can protect, every grade of a kind of different subtype of correspondence with the FEC subtype of equal number.As long as protective capability is just passable from weak to strong, these subtypes not necessarily belong to same major types.And adopt error concealment and prevent technology such as error code diffusion still failing image restored information after the protection of the strongest FEC sign indicating number of protective capability.The sensing network situation can realize by existing various QoS monitoring methods.
More complicated application scheme can be used (the communicating pair terminal is all supported) if always have T kind FEC scheme (dissimilar/subtype) also within the scope of the present invention.Which kind of FEC decision adopts, and depend on data importance and network conditions simultaneously.Can adopt the method for a two-dimentional LUT so, as shown in table 3:
The two-dimentional LUT that many kinds of FEC mechanism of table 3 are mixed and are used alternatingly
In the above form, data importance rank and network condition rank are all arranged according to ascending order.Wherein the subscript of FEC is represented with two-dimentional subscript, and the error resilience mechanism FEC in the table (i, j), 0＜i≤U, 0＜j≤V can be in above-mentioned T the FEC scheme any one.
It should be noted that, during the embodiment of foregoing invention describes all with the FEC correcting and eleting codes particularly the Tornado sign indicating number be example, but for other similar error resilience mechanisms particularly the FEC encoding scheme except that the Tornado sign indicating number can be suitable for, do not influence the spirit and scope of the invention.
And in another one embodiment of the present invention, adopted a kind of improved Tornado correcting and eleting codes specially, this improved Tornado correcting and eleting codes only generates the described check-node of one deck for one group of back end, can significantly reduce encoding time delay, satisfies the demand of real time communication.
In real-time video communication, use the protection of FEC code data bag can introduce time delay, the size of time delay is relevant with the size of view data packet.S NALU is divided into one group, and one of them NALU comprises the bit stream data of a Slice.If a two field picture is divided into a Slice, then coding side just has the time delay of S frame, and same decoding end also has the time delay of S frame.The relation of NALU and back end number is shown below:
The addition of S NALU length value equals the size that the back end number is multiplied by each node data bag in the formula.As can be seen when the S value is limited, the value of PackSize * DataNode also can be limited by formula (1), and in addition because the validity that IP network transmits causes the PackSize value can not be too little, so the value of DataNode is limited.
During real-time video is communicated by letter on the IP network, the time-delay T of a two field picture
TotalBe calculated as follows:
T
total＝T
FEC+T
codec+T
trans
T in this formula
FECBe to add the time delay that introduce FEC protection back, T
CodecAnd T
TransBe respectively H.264 codec handling time delay and network transfer delay.Because developing rapidly of Digital Signal Processing and IP network can be supposed T
CodecAnd T
TransCan both requirement of real time:
T
Codec＜=T
Th, T
Trans＜=T
Th, T wherein
Th=1/F
Target
F in the formula
TargetBe decoding target frame rate (but value 10Hz, 30Hz etc.), and establish a two field picture and be divided into a Slice that this up-to-date style (2) can change into:
T
total＜＝S*T
th+2*T
th＝(S+2)*T
th
By last two formulas as can be known, the time-delay T of a two field picture
TotalTime-delay determine by the value of S that substantially and DataNode influences the value of S greatly.Therefore, under the prerequisite that can guarantee the anti-data-bag lost ability of video communication, reduce the time delay that FEC introduces as far as possible, further guarantee the QoS of real-time video communication.
The present invention adopts improved Tornado sign indicating number protection algorithm under the limited situation of DataNode.This improved Tornado method does not adopt the coded system of multistage bigraph, but only uses the coded system of one deck check-node.Compare with original T ornado coded system, coding method after the improvement has improved the flexibility of algorithm greatly, the number of back end and check-node can be provided with arbitrarily, has also reduced the complexity of code decode algorithm, can be used for the anti-data-bag lost of real-time video communication.In addition, under the limited situation of back end, the anti-data-bag lost performance of improving the Tornado sign indicating number does not have to descend substantially.This concrete principle of improved Tornado coding method and detailed step will elaborate later.
Notice how routine ERRTP handles for the information header of not mentioning NALU in the encapsulation of NALU, in the third embodiment of the present invention on the basis of second embodiment, ERRTP similar NALU is handled together and with header comprehensively in the ERRTP header.The difference the most basic with RTP is that in the ERRTP encapsulation process, the header that will have the NALU bag of identical header is comprehensively gone in the header of ERRTP.
NALU header structure had been mentioned in the front, underlined once more here, and NALU information comprises successively:
Account for the F field of 1 bit, be used to indicate described NALU whether to make mistakes;
Account for the NRI field of 2 bits, the importance that is used to indicate described NALU;
Account for the type field of 5 bits, be used to indicate the type of described NALU.
The execution in step of receiving-transmitting sides is as described below.Transmitting terminal is encapsulated in the same ERRTP bag by ERRTP encapsulation format a plurality of NALU back end or check-node that header is identical.According to practical engineering experience, in the ordinary course of things, because H.264 always there is identical this attribute of its corresponding NALU type of adjacent part in bit stream, this hypothesis is always satisfiable.Even can't satisfy in some cases, also can have several countermeasures can handle such situation: first kind can will show NALU accumulation of the same type, after satisfying certain number, be encapsulated among the ERRTP, if the number of the NALU of another kind same type does not reach certain number, the method that adopts RTP to fill, though waste a point bandwidth, but this is insignificant, also have a kind of method to be if the different NALU of type is very many, then can adopt the RTP encapsulation, anyway can identify according to ERRTP at receiving terminal and discern, carry out corresponding processing.
Above-mentioned in described ERRTP encapsulation format, with the identical header that NALU had that it carried comprehensively in the header of this ERRTP bag, and the NALU that is carried removed its header again according to above-mentioned flow processing, divide, encode and encapsulate, be packed in the payload of this ERRTP bag.So how with the NALU head comprehensively in the ERRTP head? to specifically provide two sets of plan below to solve this several problems.
In the ERRTP encapsulation format, NRI field in the NALU header and the type field are filled in the PT field of ERRTP header packet information, and the front is narrated, and this PT field is positioned at back 7 bits of the 2nd byte of ERRTP header packet information.Provided the form of such ERRTP head in Fig. 7, wherein different with RTP places represent with bolded section that some place also can be explained in the back among the figure in addition.
Two other main points are: the first, the V field in the ERRTP packet header is identified as ERRTP, and the front is mentioned; Second, F field in the NALU header is filled in the M field of ERRTP header packet information, this M field is positioned at preceding 1 bit of the 2nd byte of ERRTP header packet information, then judge according to the M field of ERRTP bag whether its NALU that carries makes mistakes at receiving terminal, also just realized the bit function of forbidding of F field.As seen this scheme can be told the recipient of RTP packet by the difference of version, and this Real-time Transport Protocol is ERRTP, thereby in the processing of back, will carry out according to the handling process at the ERRTP agreement.
In this scheme, NALU header byte (8 bits) is replaced 1 bit of sign M field in the former RTP header and 7 bits of PT field totally 8 bits.Concrete replacement order is such as can being like this:
The F bit is replaced the M bit;
2 bits of NRI are replaced the highest 2 bits in 7 bits of PT;
5 bits of Type are replaced minimum 5 bits in 7 bits of PT;
In fact, such alternative is to have it rational.7 bits of PT can freely use originally, and the front is mentioned.The purposes of M field is stipulated as follows in RTP (RFC 3550): certain concrete aspect (Profile) can be stipulated not use the M bit, but it is incorporated into PT, and PT can have 8 bits at most like this, distinguishes 256 kinds of different types.Therefore, replace the M bit with the F bit and meet the RTP regulation fully, can not cause the problem of intercommunication between ERRTP and the traditional RTP.
The encapsulation format of finding out ERRTP of the present invention easily has tangible three advantages: the first, and overhead is few, when a plurality of NALU is especially arranged among RTP, obviously saves and transmits bit number; The second, need not just can differentiate the relative importance of these NALU to the H.264NALU data decode in the RTP packet; The 3rd, need not just can discern owing to other bit drop-out the H.264NALU data decode in the RTP packet and whether can cause this RTP bag to be correctly decoded.
For ins and outs of the present invention are described in further detail, the process prescription that provides an ERRTP encapsulation below and go to encapsulate.After carrying out above-mentioned processing, a plurality of H.264NALU types in same ERRTP packet are identical, the header byte that is them is all identical, so when they divide, encode, are encapsulated in the ERRTP packet, can peel off original header byte, if N NALU arranged like this, can reduce by N byte.When going to encapsulate, exactly NALU is extracted decoding, repartitions the original form that is reduced to from the ERRTP packet, soon this N NALU extracts from the ERRTP packet at their place and decodes out, then in minimum 7 bits of 7 bit-copy of the PT in the ERRTP header in the byte H (8 bit), and the higher bit of H is set to 0 as the F bit.Then the H byte that generates is appended to the foremost of each NALU that extracts, so just reduced each NALU.Certainly if the F field in the ERRTP packet header is 1, illustrate that the NALU in this ERRTP bag makes mistakes, therefore directly abandon and get final product, also the processing time of Jie Shenging.
Provide second kind of solution below, it is identical that this scheme has any with first, promptly also is the NRI in the NALU head and the type field are filled in 7 bits of PT field of ERRTP head.There are 2 points in different places: adopt M field identification ERRTP, a problem of bringing like this is exactly that the F field does not have the place to fill, among this embodiment with F whether two class NALU of set treat respectively, the NALU that makes mistakes for F set still adopts original RTP to transmit, for then adopting ERRTP to pass normally, but ignore this F bit.Detail is as described below.
With M field value is 1 to identify ERRTP bag, and this M field is positioned at preceding 1 bit of the 2nd byte of described ERRTP header packet information.And for the F bit, H.264 stipulating in the agreement: if syntax clash or mistake are arranged, then be 1.When having the bit mistake in Network Recognition this element, it can be made as 1, so that the recipient loses this unit.Be mainly used in and adapt to different types of network environment, the environment that combines such as wire and wireless.Concrete using priciple is: Tong Xin transmitting terminal and receiving terminal do not carry out " writing " operation for this bit carrying out H.264 Code And Decode for video when generally speaking, and decoding end is carried out " reading " operation for this bit.If find F=1, then receiving terminal will abandon this NALU in decode procedure.According to present industry widespread usage situation, carry out " writing " operation for the F bit, mainly be on the gateway between two kinds of heterogeneous networks, to carry out, such as the situation of carrying out code conversion (H.264 H.263 MPEG-4 arrive and wait to H.264).
Therefore, the present invention ignores the F bit, need not with the original H.264 purpose of definition.Thereby making the M field originally be used to fill the F bit to keep, be used for following expansion and carry more information, is exactly to be used to identify the ERRTP bag here.The benefit of doing like this is, need not make amendment for version information V=2, and ERRTP still uses original version V value 2.This also is to have saved present only RTP version information resource.
Yet inevitably, the small probability situation that needs use the F bit may appear in actual applications, in the time of such as NALU grammer mistake, the present invention does following processing for this situation: in the ERRTP encapsulation format, ignore the F field in the described NALU header; But at transmitting terminal,, still adopt RTP to seal dress, only normal NALU is adopted the ERRTP packing for the F field NALU that effectively makes mistakes; Judge then that at receiving terminal this bag still is that this bag is handled by corresponding encapsulation format in RTP bag back for ERRTP.That is to say, when the F bit at some in particular cases, be used for the original H.264 purpose of definition, the situation of the H.264NALU syntax error that exists promptly will be used to express possibility, if intermediate equipment such as gateway for video according to when H.264 agreement is carried out video coding, find that there is syntax error in certain NALU, will carry out encapsulation process separately for this NALU so.
The method flow of concluding above-mentioned ERRTP and RTP alternate treatment is as follows:
Transmitting terminal judges at first whether the F field in the header of at least one NALU is effective, in view of the above it is divided into normal NALU and the NALU that makes mistakes;
By the ERRTP encapsulation format normal NALU is packaged into the ERRTP bag then, and establishes the ERRTP sign; Be packaged into RTP bag by the RTP encapsulation format NALU that will make mistakes;
Receiving terminal judges at first whether the header of the bag that receives establishes the ERRTP sign, and it is divided into ERRTP bag and RTP bag;
Handle the ERRTP bag according to the ERRTP encapsulation format then, seal dress format analysis processing RTP bag according to RTP.
As seen, gateway is for normal N ALU, according to previously described method, for identical H.264NALU (the determining of type by concrete application according to certain rule, mainly stipulate in each ERRTP packet what similar NALU of encapsulation) carry out the ERRTP encapsulation, there is syntax error in case find certain NALU, will adopts conventional RTP encapsulation for this NALU so.This time routine the RTP packet in perhaps just only contain one H.264NALU.
What also need at last to illustrate a bit is, notice the type of the NALU that provides in the table 1 that preamble mentions and the value of corresponding the type field thereof, can find 16 kinds of existing type less thaies, 5 bits that is to say Type can be reduced to 4 fully, and this does not influence existing H.264 transmission, therefore in the ERRTP encapsulation format, when all types of NALU is less than 16 kinds, only low 4 bits with the type field characterize, and the higher bit of Type is called the C field as the expansion reservation bit.Use after this C bit waited until, proceed the function expansion.After bit C kept, the NALU type that provides in the table 1 will be done corresponding modify: totally 16 values, and value 0-12 is identical with table 1, and value 13-15 is for keeping.
Though present certainly NALU type H.264 has only 13 kinds, but H.264 back extended meeting development, may produce more NALU type,, need add that so still the C bit indicates as type with minimum 4 bits in 7 bits of PT if following NALU type is increased to more than 16 kinds.
It should be noted that here comprehensively the largest benefit in the ERRTP header packet information is just with the NALU header, the multimedia transfer equipment can directly be known the relevant information of the NALU that it carried according to the ERRTP header, and implements the H.264 real-time qos policy that transmits of multi-medium data in view of the above.This point can't realize that at existing RTP because for the RTP layer, NALU layer information is unconcerned, also just can't know the header of each NALU in the payload, thereby can't realize qos policy.
On the basis of ERRTP,, adopt the enhancement techniques of SEI carrying QoS report in order to realize the feedback of receiving terminal, describe as seen from preamble, RTCP has born the QoS report mechanism, but it is a kind of general method for reporting in fact, can be used to report QoS, also can be used to report out of Memory.For the specific video communications applications, report but not necessarily only with RTCP.Some the time, if the transmit leg of QoS information can both use more high-rise agreement such as H.264 communicating by letter with the recipient, then can consider fully with the content of H.264 carrying report.The present invention just is based on this starting point, and directly H.264 employing carries the QoS report information, can avoid using extra channel, has realized a kind of " in the band " report mechanism.
Another that transmits the QoS report by upper-layer protocol H.264 is according to being, in present video communication applications, the adaptation measure for network transmits mainly realizes based on terminal, rather than the network intermediate equipment is such as router, and switch or gateway are realized.Therefore underlying protocol is extracted and is not relied in the encapsulation of QoS report, and only the QoS report information that needs terminal can understand carrying in the extraction H.264 just can be realized the QoS monitoring, therefore can not rely on the agreements such as RTCP of bottom.Certainly, by employing " in the band " report mechanism H.264, and do not mean that the application of repelling the RTCP report mechanism, two kinds of mechanism can be selected to use, and also can coexist, and use H.264 can reduce the report flow of RTCP on the contrary.In addition; if adopt H.264 " in the band " report manner; then packet H.264 can be taked the kinds of protect measure; and H.264 packet for carrying QoS report; can think important data; according to the unequal loss protection principle of (Unequal Protection is called for short " UEP "), can adopt high-intensity safeguard measure to it.Thereby can guarantee the correct arrival of report data, improve the reliability of QoS monitoring.
The fourth embodiment of the present invention is carried QoS report based on extended message mechanism H.264 on the basis of the 3rd embodiment, roughly be divided into following three basic steps
At first, each multimedia communication terminal statistics generates the QoS report of H.264 multimedia communication, the content of these reports can be identical with SR, the RR report content of RTCP, can certainly be different, but information such as the service quality of described relevant H.264 media communication and network state are consistent;
Then, terminal is carried these QoS reports with extended message H.264, issue other communication terminals, H.264 extended message mechanism front is mentioned, SEI etc. is typically arranged, of the present invention is exactly SEI message basically, certainly along with later expansion H.264 also can be used other extended message carrying;
Terminal also receives the QoS report that other terminal is sent when sending the QoS report, and in fact each terminal all will be according to these QoS report executing qos policys.
The present invention is with SEI message bearing QoS report, is reported as example with the QoS of existing RTCP, can be directly with the main contents of SR, the RR report of RTCP, as the load of message H.264SEI, thereby carry these information with expanding SEI message.
Based on this thought, in the fourth embodiment of the present invention, define concrete SEI extended message and be specifically designed to carrying QoS report.H.264 stipulate that SEI information leaves among the class NALU, as previously mentioned its Type=6.The present invention deposits SR and the RR report message of similar RTCP in the SEI territory, both guaranteed transmission efficiency, and feedback channel state and decoded information effectively are convenient to the interactive anti-data-bag lost of coding side and decoding end again.Concrete structure as shown in Figure 8, wherein except header is arranged according to the SEI message structure, other QoS report content is all used for reference the form of SR, the RR report of RTCP.
The header that is used to carry the SEI message of QoS report comprises following field:
The 1st byte (byte 0) is load type field (SEI Type), being used to indicate load is corresponding QoS report, in the present embodiment, it in the SEI territory is transmission report (SR) among the similar RTCP that SEI Type=200 represents to leave in, and SEI Type=201 represents that it is to receive report (RR);
2nd, 3 bytes (byte 1,2) are payload length field (SEI Packet-Length), are used to indicate corresponding QoS report length, the identical definition of length field employing in the QoS report of this length and RTCP;
After the 4th byte reaches is the load of SEI message, also promptly is used to fill corresponding QoS report.
QoS report also is divided into transmit leg report and recipient's report, distinguish by the load type field indication, i.e. SEI Type value difference, the particular content of QoS report can with SR, the RR of RTCP report identical, shown in Fig. 2:
Version information field (V) accounts for 2 bits, and this routine value is that binary one 1 is V=3, expression and the difference of version in the past;
Fill field (P), account for 1 bit, be used to indicate whether to have the filling content, identical with RTCP;
Receive number of reports field (RC), account for 5 bits, be used for indicating this QoS report to report and receive the report blocks number;
Transmit leg SSRC field accounts for 32 bits, is used to identify the transmit leg of this service quality report;
For transmit leg report, also comprise the caller information piece here, be used to describe the relevant information of the transmit leg of this report;
Comprise polylith afterwards and receive report blocks, be used to describe from the multimedia statistical information of homology not, every ASSOCIATE STATISTICS index that comprises the identifier and the media stream in source has been described the meaning of various indexs among the RTCP of front;
Comprise specific aspect expansion at last, be used for the reservation function expansion of specific aspect.
As seen, the QoS report content and the RTCP that provide among Fig. 8 are basic identical.After the substance RR of RTCP and SR write the SEI territory, can not need special logic channel to transmit RTCP information, save the part bandwidth cost.In fact, marrow of the present invention is to be with interior carrying with SEI message, and the generation of how adding up as for the QoS report as long as can realize the goal of the invention of QoS monitoring, does not influence the spirit and scope of the invention.
After realizing the QoS report, can carry out multiple qos policy on this basis, such as the cumulative data packet loss field of utilizing RTCP, they can be used for feedback decoding information in two-way video communication (the existing encoder of terminal has decoder again), be convenient to interactive anti-data-bag lost.
In addition, in the QoS report fields such as the delay variation of arrival and transmit leg byte count are arranged, they all can be used for the sensing network state.Wherein, rate control algorithm can guarantee that further coding side speed is near constant according to the information that arrives in the delay variation field; The transmit leg byte count field can be estimated the Mean Speed of load, is convenient to transmitting terminal and resets coder parameters according to network state, comprises resolution of adjusting target frame rate, recovery picture quality and original image or the like.
In order to improve the reliability deficiency that RTCP transmits; after adopting H.264 " in the band " report manner; H.264 packet can be taked the kinds of protect measure; and H.264 packet for carrying QoS report; can think important data; according to the principle of unequal loss protection, can adopt high-intensity safeguard measure to it.Thereby can guarantee the correct arrival of report data.Should further carry such as the SEI that is used to carry the QoS report by NALU; and NALU has header that the significance level of this content can be set as previously mentioned; therefore communication terminal can be provided with the nal_ref_idc field of this NALU according to the reliability requirement that the QoS report transmits; can be made as 1; 2; 3 etc., promptly can be in error elasticity coding according to the grade of this field the different safeguard measures that take varying strength.
Communication terminal can also dynamically be adjusted transmission cycle based on the QoS report of SEI message according to current network state and higher layer applications demand.Under the default situation, the time interval (being reporting period) that RTCP information is write the SEI territory is consistent with suggestion RTCP transmission interval among the RFC3550.Certainly, according to the needs (specific guard method etc.) of application-specific, possible reporting period is just the same with RFC 3550 regulation not necessarily, but can adjust.Determine according to the needs of application-specific reporting period.Such as, a performance that important use is the dynamic estimation network of report data: packet loss postpones shake etc.Frequently detect these data if desired, then will lack reporting period, otherwise can grow reporting period.When network condition is good, can stop report.In addition,, can also mix the QoS report of carry multiple Media Stream, only need to report that at QoS the back add various Media Streams and receive report blocks accordingly and get final product with the QoS report that SEI message not only can transmit video H.264.Such as audio stream etc., as long as in the SR report, increase the concrete report blocks content of SSRC in its source.The front is also mentioned, except adopt SEI be with in after the monitoring, communication terminal can also select existing RTCP to transmit, and also can use among H.264 extended message, the RTCP one or both to transmit carrying QoS report simultaneously.
Adopt after SEI realizes the QoS report relevant from receiving terminal feedback network situation having provided, just realize the adjustment of adaptive protection strategy on this basis easily, comprise multi-stage protection and unequal loss protection.According to the problem that prior art can't self adaptation be adjusted for network communication state, the fifth embodiment of the present invention provides a kind of video transmission method of adding up the adaptive guard of current communication conditions and self adaptation adjustment protection strategy.At first according to the guard method performance impact, provide the different parameters configuration, the different many hierarchical protection strategy of protective capability is set, be used under the different communication situation, being selected to and carry out the high-efficiency reliable protection; Secondly, according to signal intelligence statistics network situation, communication quality, and it is sent back to transmitting terminal at receiving terminal; Adjust according to the communication quality statistical information of beaming back by transmitting terminal at last, select the tactful grade of only protection.
The channel that the key of this scheme also is to add up the method for communication quality and beams back statistical information.Utilize sequence number loss situation H.264NALU can add up information such as packet loss and position thereof, and, be used to carry this statistical information, transmit statistics to transmitting terminal from receiving terminal by defining the expansion SEI message structure of payload part among the NALU.Though such feedback mechanism is not quite similar with the SR/RR form of QoS report, but those of ordinary skill in the art are appreciated that, the cardinal principle of dual mode is identical, just with the content difference of SEI carrying, therefore the description below proposes the difference of the scheme and the scheme that QoS reports of SEI bearer network packet loss no longer specially.
With the Tornado correcting and eleting codes is example, and promptly the coding-decoding method according to aforementioned Tornado correcting and eleting codes comes video stream data is protected.The Tornado correcting and eleting codes needs setup parameter to have: back end number, check-node number, pass the ratio that contracts, the check-node number of plies, be used for the bigraph (bipartite graph)s at different levels of calculation check node.In the video flowing communication process, transmitting terminal is divided into back end with video stream data, produces check-node according to the Tornado coding method then, sends to receiving terminal together; Receiving terminal then carries out error correction according to the Tornado coding/decoding method, obtains video stream data.
Because factor such as the practical IP network bandwidth is often to change and unsettled; therefore fixing protection strategy will bring problems such as poor efficiency or high bit-error; therefore present embodiment has preestablished the protection strategy series of protection different brackets, is respectively applied under different communication credit rating situation and protects video stream data.As seen; the protection strategy of different brackets can adapt to the variation of network communication quality, not only can satisfy the protection requirement under the channel degradation situation, and can suitably turn down protection under signal improvement situation; to reduce overhead, save processing, bandwidth resources.
For given different brackets protection strategy, need to set the Tornado correcting and eleting codes of different parameters.Mainly contain the random distribution of back end number, check-node number and bigraph (bipartite graph) both sides node degree vector according to the parameter of aforementioned affect Tornado correcting and eleting codes protective value; for the sake of simplicity; the Tornado sign indicating number of different abilities; general do not have a unified bigraph (bipartite graph), it is tactful to adopt different back end numbers and check-node number to provide the Tornado correcting and eleting codes protection of different protections.According to Tornado correcting and eleting codes principle, the different pieces of information interstitial content can be determined the Tornado correcting and eleting codes of different code checks or redundancy rate with the check-node number, thereby provides different protections and overhead.
Receiving terminal receives data and carries out the decoding of Tornado correcting and eleting codes and obtains video stream data, adds up according to event of data loss simultaneously, obtains statistical information and characterizes communication quality.
Transmitting terminal need be protected strategy adjustment according to communication quality status, therefore need add up the transmission situation, and the receiving terminal basis H.264 sequence number of the NALU of video flow data is added up the transmission situation.In based on the communication of two-way video H.264, all existing encoder of each terminal of communication system, again decoder arranged.And NALU is a sequence numbering, and promptly the NALU that sends of all transmitting terminals has unified sequence numbering, and therefore, receiving terminal can judge whether that NALU loses according to the sequence number of receiving NALU.If there is the discontinuous just explanation of NALU sequence number to exist NALU to lose, the NALU sequence number of interruption is exactly the sequence number of losing NALU, and its number is exactly the NALU number of losing.Through accumulative total after a while, can calculate the total number of the NALU that this section lose in the period, again all NALU numbers in this time period are carried out normalization, can obtain accumulative total packet loss (Accumulated Lost SliceRate is called for short " ALSR ").Certainly, receiving terminal also can directly send back to transmitting terminal with packet loss information, is added up by transmitting terminal.Adopt the NALU sequence number to add up, can guarantee that not only statistical information is accurately errorless, and directly utilize available data information, do not need extra carrying expense.
Receiving terminal is beamed back transmitting terminal with statistical information and other loss of data information by expansion SEI message.After the receiving terminal statistics obtains about the statistical information that transmits situation, need send back to transmitting terminal, present embodiment has defined expansion SEI message structure, is specifically designed to the transmission situation statistical information that carrying is beamed back from receiving terminal.Receiving terminal writes this information in the expansion SEI message body of special definition after finishing statistics, writes then in the SEI territory of the encoding code stream that this terminal beams back, and beams back transmitting terminal.After transmitting terminal is received this SEI message, can directly learn statistical information, perhaps statistics obtains ALSR, thereby sets up the true perception mechanism of transmitting terminal for network packet loss rate.
SEI message is also carried by the base unit NALU of code stream H.264 as previously mentioned, and each SEI territory comprises one or more SEI message, and SEI message is made up of SEI header and SEI payload.The SEI header comprises two code words: load type and magnitude of load.Wherein the length of load type not necessarily, such as type 0 between 255 the time with a byte representation, when type is represented to 0xFFFE with two byte 0xFF00 between 511 the time 256, and the like, the user can self-defined any multiple load type like this.In existing H.264 standard, type 0 has been defined as information specific in Class1 8 standards, as caching period, image timing etc.The SEI territory of definition can be deposited abundant customized information according to demand in this shows H.264.In the first embodiment of the present invention, a kind of expansion SEI message that is used to carry statistical information of definition in the SEI load type of reserving.
Last transmitting terminal carries out the adjustment of Tornado correcting and eleting codes according to the statistical information of beaming back, and uses the protection strategy of suitable more current transmission situation.Last transmitting terminal will be adjusted the protection strategy according to statistical information, promptly select the protection strategy of suitable grade.Here transmitting terminal also will preestablish the judgment threshold series corresponding to different protection class, sets to enter each other threshold value of level, selects its corresponding grade according to threshold value that ALSR drops on then.The statistics of the transmission situation of Jian Liing, feedback, adjustment mechanism can adapt to network transmission demand accurately and timely thus, improve protective capability.
To the The data of different importance different protection strategy series.The protection requirement of considering critical data and non-critical data is different, in order further to improve fitness, has set two different protection strategy series, is respectively applied for protection critical data and non-critical data.Like this, the data of two kinds of different communication demands get final product independent process, by being fit to the protection selection protection strategy of demand separately, improve system effectiveness.
For example, as protection scheme series, its protective capability grade characterizes with parameter n, l with the Tornado sign indicating number of different brackets, and wherein n represents that data section counts, and l represents the check-node number.(n+l, n) expression is by the definite Tornado sign indicating number protection scheme of parameter n, l with TN.Therefore the protection scheme series corresponding to critical data is: TN
K(n
0+ l
0, n
0), TN
K(n
1+ l
1, n
1) ... .., TN
K(n
L-1+ l
L-1, n
L-1); The same protection scheme series for non-critical data is: TN
NK(n
0+ l
0, n
0), TN
NK(n
1+ l
1, n
1) ... .., TN
NK(n
L-1+ l
L-1, n
L-1).Setting threshold series 0＜G
1, G
2..., G
L-1＜1, promptly be used for judging the selection protection class.Transmitting terminal is when adjusting the protection strategy, according to ALSR and threshold value G
1, G
2..., G
L-1Relation, carry out following operation:
If 0＜AlSR＜G1 then adopts TN
K(n
0+ l
0, n
0) protect for critical data, adopt TN
NK(n
0+ l
0, n
0) protect for non-critical data;
If G
i＜AlSR＜G
I+1, i=1,2 ... .., L-2 then adopts T
NK(n
i+ l
i, n
i) protect for critical data, adopt TN
NK(n
i+ l
i, n
i) protect for non-critical data;
If G
L-1TN is then adopted in＜AlSR＜1
K(n
L-1+ l
L-1, n
L-1) protect for critical data, adopt TN
NK(n
L-1+ l
L-1, n
L-1) protect for non-critical data.
In addition, the obliterated data information that transmitting terminal is also beamed back according to receiving terminal resends these information.Receiving terminal when the NALU information that statistics is lost, the locating information of the picture frame that the NALU institute correspondence that obtains to lose simultaneously comprises, this information comprises the sequence number of place frame and the position in the frame.Receiving terminal sends back to transmitting terminal with locating information, and transmitting terminal can navigate to corresponding video stream data, and resends.In real-time video communication, the oversize video stream data of delaying time has lost value, but under some business demand situation or under certain mechanism, data with certain time-delay still have value, such as in the bigger video communication of buffering range, as long as the video stream data of time-delay still drops in the buffering area, these data just can be used to avoid the interruption of video flowing broadcast.As seen retransmission mechanism has important value for the reliability that improves video flowing communication and service quality.
Except adopting error elasticity protection strategy; the sixth embodiment of the present invention is on the basis of the 5th embodiment; eliminate two aspects from error concealment and error code diffusion; eliminate strategy in conjunction with the diffusion of the error code of receiving terminal error concealment strategy and transmitting terminal, can avoid error code to cause the purpose of diffusion again to realize to reduce the video quality loss that error code brings as far as possible.For error concealment, adopt simple replacement scheme can reach the effect that realizes the loss of compensation error code with the complexity of trying one's best low; Diffusion is eliminated for error code, set up error code feedback information mechanism by H.264 having passage, implement intraframe coding according to feedback, eliminate effect to reach diffusion, and do not increase the additional networks burden, guarantee the robustness of video code flow, also therefore avoid the error code diffusion that causes because of error concealment error code problem.
The basic ideas of this scheme are, by to the statistics of NALU sequence number, find obliterated data information at receiving terminal, as the position of Slice etc., adopt highly effective algorithm that obliterated data is carried out simply substituting to cover the error code loss on the one hand, give transmitting terminal with the error code feedback information on the other hand.By expansion SEI message H.264, set up the error code feedback information passage from the receiving terminal to the transmitting terminal.After transmitting terminal is known error code information, take segmentation to carry out the strategy of intraframe coding one by one immediately, error code Slice segmentation is refreshed, to prevent the error code diffusion.
H.264 in the video communication process, transmitting terminal is encoded to video stream data to be sent, obtains video code flow, encapsulates NALU then and sends receiving terminal to by the grouping message.Receiving terminal receives message and also decodes, and receiving terminal need be judged whether video stream data has and lost this moment, eliminated operation to carry out follow-up error code.Error code elimination flow process roughly is divided into covers, feeds back, spreads three big steps of elimination.
At first, receiving terminal judges whether obliterated data according to NALU sequence number interruption situation, and the information of statistics obliterated data, i.e. error code information.As previously mentioned, NALU is the base unit that transmits of video stream data H.264, and each NALU has unique continuous sequence number.Therefore, whether receiving terminal has interruption according to receiving the NALU sequence number, knows which NALU has lost.Thereby can implement error concealment strategy at obliterated data.Adopt the NALU sequence number to add up, can guarantee that not only statistical information is accurately errorless, and directly utilize available data information, do not need extra carrying expense.
At first receiving terminal is known sequence number by the NALU header that identification receives, discontinuous detection error code by sequence number takes place, learn the video data that the NALU of intercalary delection should carry by front NALU, the loss of data that error code is caused positions, such as the previous NALU carrying of losing NALU is the 1st Slice of N frame, then can infer the position of losing the Slice that NALU carries by the transmission order, should be a back Slice of this frame.
It is synchronous that then receiving terminal need carry out the weight of video information, because H.264 in the continuous transport process of video code flow, receiving terminal and data flow need synchronously, could correctly receive then, in case data flow has after the interruption, receiving terminal need carry out synchronously again, and is synchronous by the weight that finds interruptions next NALU header afterwards to finish decoder.This process, receiving terminal also need to judge middle NALU number and locating information thereof of losing by the sequence number of next NALU.
Afterwards, receiving terminal need carry out error concealment, the NALU of obliterated data is abandoned by whole, therefore this NALU carries whole Slice and loses, the error concealment strategy is exactly to substitute by simple, with the data that time-domain or the adjacent data replacement of spatial domain are lost, cover such as the Slice recovery view data of the former frame correspondence position that adopts obliterated data place frame.
Receiving terminal feeds back to transmitting terminal with it after obtaining error code information.Feedback path of feedback error code informational needs, in order to reduce network burden, to simplify realization mechanism, adopt existing H.264 communication mechanism in the first embodiment of the present invention, definition expansion SEI message, be used to carry error code information and set up feedback, so that transmitting terminal prevents the error code diffusion in conjunction with error code information.In fact, eliminate strategy, just can avoid spreading because of the error code that error concealment strategy that the front receiving terminal is implemented causes in conjunction with the error code diffusion of error code feedback information mechanism and transmitting terminal.
Among the embodiment in front, utilize expansion SEI message H.264 that a kind of feedback information mechanism from the receiving terminal to the transmitting terminal is provided, making transmitting terminal can in time understand which NALU has lost, can in time carry out the efficient error diffusion like this and eliminate, prevent to cause later error code diffusion because of these data of losing.
The benefit of setting up feedback information mechanism in system inside H.264 is the conserve network bandwidth expense, saves the system handles resource and does not influence interoperability.Introduce below and how to define expansion SEI message.SEI message is also carried by the base unit NALU of code stream H.264 as previously mentioned, and each SEI territory comprises one or more SEI message, and SEI message is made up of SEI header and SEI payload.The SEI header comprises two code words: load type and magnitude of load.Wherein the length of load type not necessarily, such as type 0 between 255 the time with a byte representation, when type is represented to 0xFFFE with two byte 0xFF00 between 511 the time 256, and the like, the user can self-defined any multiple load type like this.In existing H.264 standard, type 0 has been defined as information specific in Class1 8 standards, as caching period, image timing etc.The SEI territory of definition can be deposited abundant customized information according to demand in this shows H.264
Then, transmitting terminal promptly begins to carry out error code diffusion elimination according to the error code information of feedback.The error code diffusion removing method of associating error code information, it is good that its effect is eliminated than the error code diffusion of existing nothing feedback.Utilize error code information, such as the position of losing Slice, transmitting terminal can autotelicly take to prevent measure at losing Slice, such as avoiding in the coding afterwards to lose Slice as the reference frame, in the time of can shortening the receiving terminal decoding so as far as possible to the dependence of this Slice.
Because H.264 coding is based on Slice, promptly before and after the data of same Slice of frame have with reference to related, the same Slice data of subsequent frame are by the Slice predictive coding of previous frame, so the error code diffusion also will be limited to same Slice inside.In the second embodiment of the present invention, the strategy that adopts segmentation to carry out intraframe coding one by one, promptly after sending error code, this Slice zone segmentation to later frame is divided into new Slice, such as a division place P macro block as a new Slice, then it is adopted intraframe coding, to eliminate reference or the dependence of this Slice to the Slice that loses previously.Because H.264 the real-time video transfer system adopts the data transfer rate controlling schemes to limit the fluctuation of every frame data in order to guarantee delivery quality, makes and the equilibrium of every frame data amount improves the stability that video transmits.Therefore, the data volume of once carrying out intraframe coding in every frame is that macroblock number can not be too many, otherwise will surpass H.264 data transfer rate control range.
Fig. 9 shows the segmentation principle of the error code diffusion elimination of intraframe coding one by one.After the packet loss mistake that occurs recovering when receiving terminal, detection and feedback error code information are given transmitting terminal, and promptly the interior locating information of the Slice place frame of obliterated data and frame sends back to transmitting terminal by the SEI message of expansion.Transmitting terminal extracts the slice locating information of losing from SEI message, be divided into three Slice such as the every frame among Fig. 9, i.e. Slice# 0, Slice# 1, Slice# 2, and the Slice# 1 of n frame loses in transmission, needs to carry out segmentation intraframe coding one by one afterwards.
At first, in the n frame, coding side is pressed the macro block scanning sequency to Slice# 1, begins to cut apart P macro block from original position and forms new Slice# 3, and the residue macro block still is Slice# 1, and have four Slice this moment, wherein new Slice# 3 carried out intraframe coding.
Then, in the n+1 frame, the Slice# 3 of cutting apart new composition in the previous step sends as Slice# 3 after intraframe coding, and other Slice still encode according to routine.
After this, need to judge among the Slice# 1 whether also remain macro block, cut apart, then return the first step and continues that in next frame Slice# 1 is remained the macro block segmentation and form new frame, implement also transmission of intraframe coding, dispose up to all macro blocks if having or not.
Above each macro block number P that divides should meet the following conditions, big as far as possible, avoiding segmentation times, reduce processing delay, to shorten coverage, but need satisfy aforementioned H.264 data transfer rate control range.Each macro block number of dividing can be different, but last macroblock number of dividing will make all macro blocks of losing among the Slice all dispose.
Such as a frame of video stream data is made up of 240 macro blocks, and per 80 macro blocks of initial division are a Slice, and promptly the 1-80 macro block is Slice# 0, and the 81-160 macro block is Slice# 1, and the 161-240 macro block is Slice#2.And calculate definite suitable segmentation numerical value P according to data transfer rate is one section of 12 macro block.After then losing Slice# 1 in the n frame, 80 macro blocks of Slice# 1 should carry out segmentation intraframe coding one by one, at first in the n+1 frame, select preceding 12 macro blocks to carry out intraframe coding and consist of Slice# 3, Slice# 3 can adopt conventional predictive coding in the n+2 frame like this, and 12 macro blocks following carry out intraframe coding composition Slice# 4 again, last residue is that 8 macro blocks carry out intraframe coding composition Slice#9 when the n+7 frame successively, just finishes the segmentation error code method of diffusion flow process of intraframe coding one by one.
Find the method for employing error concealment of the present invention and error code diffusion elimination associating according to experimental result after, the video image effect that obtains is very good.
In the end also will propose a kind of improved Tornado encoding scheme, this scheme uses this Tornado encoding scheme as error elasticity protection strategy in the seventh embodiment of the present invention.Simply point out the main distinction of this kind Tornado encoding scheme and traditional encoding scheme below.
Adopting the Tornado sign indicating number to carry out in the process of data transmission protection; the Tornado code check node layer that multilayer is set can strengthen data to a certain extent and transmit protective capability; but; the Tornado code check node layer that multilayer is set also can make the operand of Tornado sign indicating number big, thereby makes data pay the long cost of time delay in transmitting the protection process.If can guarantee that data transmit protective capability and do not have under the situation of significantly decline; reduce the number of plies of check-node layer; just can effectively reduce the operand of Tornado sign indicating number, reduce the time delay in the data transfer procedure greatly, transmit protective value-cost ratio thereby seek higher data.Therefore, the seventh embodiment of the present invention is: the correcting and eleting codes that only has one deck check-node layer is set, carries out data according to described correcting and eleting codes and transmit protection.
This Tornado correcting and eleting codes scheme only has one deck check-node layer, removed the middle check-node layer of Tornado sign indicating number, equally, also removed the nonnegotiable demands that produces last one deck check-node in the Tornado sign indicating number according to the Reed-Solomon coding, like this, correcting and eleting codes of the present invention only has one deck back end layer and one deck check-node layer as shown in Figure 10, we can say that correcting and eleting codes of the present invention is a kind of Tornado sign indicating number of designs simplification, is a kind of improved Tornado sign indicating number.
In the back end of the improved Tornado sign indicating number of the present invention size L1, the back end layer in the number n of back end, the check-node layer check-node number L can come according to the actual requirements to determine.As according to the check-node number L that comprises in the back end number n that comprises in back end size L1, the back end layer in data transfer rate, data type such as voice data/video data etc., data protection Capability Requirement, the factor specified data node layer such as maximum network delay that can receive, the check-node layer.
If the Tornado sign indicating number has the middle check-node layer of m layer in the setting prior art, and from m intermediate layer of back end layer to the, the geometric ratio of the interstitial content between the adjacent two layers is passed the factor that contracts and is
The geometric ratio of the interstitial content between final layer and the m layer is passed the factor that contracts
Then total node of Tornado sign indicating number is counted Total in the prior art
NodeFor:
Because Total
Node=n+L, therefore, the setting of L is conditional,
L can not set arbitrarily.Owing to need to guarantee that the node number of every node layer in the Tornado sign indicating number all is an integer, therefore need
And
All be integer, this condition is called implicit integer node said conditions.According to this condition, if the m in the given Tornado sign indicating number and
Just can calculate the condition that n need satisfy, as work as m=3,
Then can calculate n=16k, wherein k is any natural number.Hence one can see that, and the minimum value that n can obtain is 16, and the code check r of Tornado sign indicating number is in the prior art:
And redundancy rate 1-r is:
The improved Tornado sign indicating number of the present invention makes improved Tornado sign indicating number no longer need the condition of above-mentioned implicit integer interstitial content owing to there is not middle check-node layer, and the check-node number L of the check-node layer of the improved Tornado sign indicating number of the present invention is:
The geometric ratio of the back end layer of improved Tornado sign indicating number of the present invention and the interstitial content of check-node layer is passed the factor that contracts
Can be provided with arbitrarily, under the condition of given back end number n, L can set flexibly.
The code check r of the improved Tornado sign indicating number of the present invention is:
The redundancy rate 1-r of the improved Tornado sign indicating number of the present invention is:
The improved Tornado sign indicating number of the present invention can be expressed as TN (n+L, n), as TN (30,20), verification interstitial content L=10 in back end number n=20, the check-node layer in the expression back end layer.At this moment, the present invention improved Tornado sign indicating number
And code check r=2/3=66.7%.
In sum, the present invention is on the basis of comprehensive above-mentioned six kinds of enhancement techniques, with whole H.264/ERRTP transport architecture modular implementation, and mutually combine on a protocol stack, advantage separately not only can be realized, and after strengthening mutually, better reliability and service quality can be embodied.
Those of ordinary skill in the art are appreciated that in the description of above-mentioned seven embodiment, relate to various specific implementation details and parameter selection etc., all are to determine in addition according to concrete application, do not influence the spirit and scope of the invention.
Though pass through with reference to some of the preferred embodiment of the invention, the present invention is illustrated and describes, but those of ordinary skill in the art should be understood that and can do various changes to it in the form and details, and without departing from the spirit and scope of the present invention.
Claims (44)
1. a multimedia communication method is characterized in that, its communication process comprises following steps,
The A transmitting terminal is protected multi-medium data according to error elasticity protection strategy, and sends it to receiving terminal;
The described receiving terminal of B receives described multi-medium data, is occurring under the wrong situation of transmission, recovers or partly recover described multi-medium data by described error elasticity protection strategy.
2. multimedia communication method according to claim 1 is characterized in that its communication process also comprises following steps,
The described receiving terminal statistics of C communication quality generates service quality report, and it is sent back to described transmitting terminal;
In the described steps A, described transmitting terminal is adjusted described error elasticity protection strategy according to described service quality report.
3. multimedia communication method according to claim 2 is characterized in that its communication process also comprises following steps,
The described receiving terminal statistics of D transmits error message, and implementation mistake is covered strategy;
Among the described step C, described receiving terminal also feeds back to described transmitting terminal with described transmission error message;
The described transmitting terminal of E is implemented the error code diffusion according to described transmission error message and is eliminated strategy.
4. multimedia communication method according to claim 3 is characterized in that, described steps A comprises following substep:
The A1 transmitting terminal selects the error elasticity encoding scheme that multi-medium data is carried out forward error correction coding;
Multi-medium data after the described transmitting terminal of A2 is encoded with the encapsulation of error elasticity real time transport protocol, and in described error elasticity real time transport protocol header packet information, carry described error elasticity encoding scheme relevant information, send to receiving terminal then;
Described step B comprises following substep:
The described receiving terminal of B1 goes the error elasticity real time transport protocol bag of receiving to encapsulation, and extracts described forward error correction coding scheme relevant information from described error elasticity real time transport protocol header packet information;
If the losing of error elasticity real time transport protocol bag of back end correspondence taken place in B2 in transport process, so described receiving terminal is according to described error elasticity encoding scheme relevant information, select described decoding FEC scheme to carry out decoding FEC, recover or partly recover the described multi-medium data of losing; If losing of back end do not taken place, then do not need to carry out decoding FEC.
5. multimedia communication method according to claim 4 is characterized in that, the multi-medium data behind the described forward error correction coding is divided into back end and check-node two classes.
6. multimedia communication method according to claim 5 is characterized in that, in described steps A 1, described transmitting terminal is selected described forward error correction coding scheme according to one of following factor at least:
Current network transmission status and service of multimedia data credit rating to be sent, service of multimedia data credit rating wherein to be sent depends on the relative importance of different pieces of information.
7. multimedia communication method according to claim 6 is characterized in that, comprises in the described error elasticity real time transport protocol header packet information:
Error elasticity real time transport protocol identification field is used for indication to be different from real time transport protocol;
The forward error correction coding type field is used to the forward error correction code type of indicating described forward error correction coding scheme to adopt;
The forward error correction coding sub-type field is used to indicate the relative parameters setting of described forward error correction coding scheme;
The data packet length field is used to indicate the length of described forward error correction coding scheme at the node that described multi-medium data is carried out obtain behind the forward error correction coding;
The number of data packets field is used to indicate the number of the described node that this error elasticity real time transport protocol bag carried.
8. multimedia communication method according to claim 7, it is characterized in that, in described steps A 1, described transmitting terminal is divided at least one isometric back end with at least one described H.264 network abstraction layer unit, then it is carried out forward error correction coding, obtain at least one check-node;
In described steps A 2, described transmitting terminal sends described back end and described check-node packet encapsulation at least one described error elasticity real time transport protocol bag;
In described step B1, described receiving terminal goes encapsulation to obtain described back end and described check-node after receiving described error elasticity real time transport protocol bag;
In described step B2, if the back end in the transport process has taken place to be lost, then described receiving terminal carries out recovering based on the recovery or the part of decoding FEC to the described back end of losing according to described check-node, and division obtains described H.264 network abstraction layer unit.
9. multimedia communication method according to claim 8 is characterized in that, before beginning transmission, also comprises step,
Described transmitting terminal and described receiving terminal are consulted to determine: for various described forward error correction code types, the corresponding relation of the relative parameters setting of this kind forward error correction that the value of described forward error correction sub-type field is indicated with it.
10. multimedia communication method according to claim 9, it is characterized in that, described transmitting terminal and described receiving terminal are all set up mapping table according to the corresponding relation of described forward error correction coding sub-type field indication, are used for inquiring about pairing forward error correction coding or decoding FEC processing module according to described forward error correction coding type field and described forward error correction coding sub-type field;
In described steps A 1, described transmitting terminal calls corresponding forward error correction coding processing module and carries out forward error correction coding;
In described step B2, described receiving terminal calls corresponding decoding FEC processing module and carries out decoding FEC.
11. multimedia communication method according to claim 10, it is characterized in that, in described steps A 1, described transmitting terminal is according to the combination of network abstract layer reference identification field in the header of described H.264 network abstraction layer unit and any one or both in the network abstraction layer unit type field, and any other can predefined rule assess the relative importance of corresponding data, thereby determine described service quality rating, and then select described forward error correction coding scheme, determine described forward error correction coding type field and forward error correction coding sub-type field.
12. multimedia communication method according to claim 10, it is characterized in that, in described steps A, described transmitting terminal is estimated described network transmission status according to the transmission report of described receiving terminal feedback, and then select described forward error correction coding scheme, determine described forward error correction coding type field and forward error correction coding sub-type field.
13. multimedia communication method according to claim 12 is characterized in that, the version information field value in the described error elasticity real time transport protocol header packet information is binary value " 11 " or decimal value " 3 ", to be different from real time transport protocol;
Described forward error correction coding type field is positioned at after the tabulation of contribution source identifier, accounts for 4 bits;
Described forward error correction coding sub-type field is positioned at after the described forward error correction coding type field, accounts for 9 bits;
Described data packet length field is positioned at after the described forward error correction coding sub-type field, accounts for 11 bits;
Described number of data packets field is positioned at after the described data packet length field, accounts for 8 bits.
14. multimedia communication method according to claim 8, it is characterized in that, in the described steps A, divide together again, encode after at least one network abstraction layer unit that described transmitting terminal is identical with header is removed its header and be encapsulated into described error elasticity real time transport protocol bag, and with identical header that this network abstraction layer unit had comprehensively in the header of this error elasticity real time transport protocol bag;
Among the described step B, described receiving terminal obtains the header that is carried from the header of the described error elasticity real time transport protocol bag that receives, and add the head of the network abstraction layer unit of having peeled off header that extracts from described error elasticity real time transport protocol bag to, obtain whole network level of abstraction unit; Transmit mistake if exist, then have the decoding FEC of carrying out to recover or part restore data node, and then therefrom extract network abstraction layer unit according to presetting strategy.
15. multimedia communication method according to claim 14, it is characterized in that, in described error elasticity real time transport protocol header, network abstract layer reference identification field in the described network abstraction layer unit header and type field are filled in the payload type field of described error elasticity real time transport protocol header packet information, and this payload type field is positioned at back 7 bits of the 2nd byte of described error elasticity real time transport protocol header packet information.
16. multimedia communication method according to claim 15, it is characterized in that, described error elasticity real time transport protocol identification field is the version information field of described error elasticity real time transport protocol header packet information, and this version information field is positioned at preceding 2 bits of the 1st byte of described error elasticity real time transport protocol header packet information.
17. multimedia communication method according to claim 16, it is characterized in that, in described error elasticity real time transport protocol encapsulation format, the bit field of forbidding in the described network abstraction layer unit header is filled in the tag field of described error elasticity real time transport protocol header packet information, and this tag field is positioned at preceding 1 bit of the 2nd byte of described error elasticity real time transport protocol header packet information;
And in described step B, receiving terminal judges according to the tag field of described error elasticity real time transport protocol bag whether its network abstraction layer unit of carrying makes mistakes.
18. multimedia communication method according to claim 15, it is characterized in that, in described steps A, B, described error elasticity real time transport protocol is designated the tag field value of described error elasticity real time transport protocol header packet information, and this tag field is positioned at preceding 1 bit of the 2nd byte of described error elasticity real time transport protocol header packet information.
19. multimedia communication method according to claim 18 is characterized in that, described steps A comprises following substep:
Described transmitting terminal at first judges to forbid whether bit field is effective in the header of at least one described network abstraction layer unit, in view of the above it is divided into proper network level of abstraction unit and the network abstraction layer unit of makeing mistakes;
By described error elasticity real time transport protocol encapsulation format described proper network level of abstraction unit package is become described error elasticity real time transport protocol bag then, and establish described error elasticity real time transport protocol sign;
By described real time transport protocol encapsulation format the described network abstraction layer unit of makeing mistakes is packaged into described real time transport protocol bag;
Described step B comprises following substep:
Described receiving terminal judges that at first whether the header of the bag that receives establishes described error elasticity real time transport protocol sign, is divided into described error elasticity real time transport protocol bag and described real time transport protocol bag with it;
Handle described error elasticity real time transport protocol bag according to described error elasticity real time transport protocol encapsulation format then, seal the described real time transport protocol bag of dress format analysis processing according to described real time transport protocol.
20. multimedia communication method according to claim 8 is characterized in that, the following substep of described step C,
The described receiving terminal statistics of C1 generates described service quality report;
The described receiving terminal of C2 carries described service quality report with extended message H.264, issues described transmitting terminal.
21. multimedia communication method according to claim 20 is characterized in that, described H.264 extended message is a supplemental enhancement information;
The encapsulation format of described service quality report in described supplemental enhancement information is as follows:
The 1st byte is load type field, and being used to indicate load is the corresponding with service quality report;
2nd, 3 bytes are payload length field, are used to indicate corresponding with service quality report length;
After the 4th byte reaches is load, is used to fill the corresponding with service quality report.
22. multimedia communication method according to claim 21 is characterized in that, described service quality report is divided into transmit leg report and recipient's report, is distinguished by described load type field indication;
When described service quality report was filled in the load of described supplemental enhancement information, the load of described supplemental enhancement information comprised:
The version information field accounts for 2 bits;
Fill field, account for 1 bit, be used to indicate whether to have the filling content;
Receive the number of reports field, account for 5 bits, be used for indicating this service quality report to report and receive the report blocks number;
Transmit leg synchronous source identifier field accounts for 32 bits, is used to identify the transmit leg of this service quality report;
When described service quality report is transmit leg when report, also comprise the caller information piece, be used to describe the relevant information of the transmit leg of this service quality report;
Comprise at least one described reception report blocks, be used to describe from the multimedia statistical information of homology not;
Comprise specific aspect expansion, be used for the reservation function expansion of specific aspect.
23. multimedia communication method according to claim 21 is characterized in that, the described supplemental enhancement information that is used to carry described service quality report is further carried by abstract network layer unit;
The reliability requirement that described communication terminal transmits according to described service quality report is provided with the network abstract layer reference identification of this abstract network layer unit.
24. multimedia communication method according to claim 21 is characterized in that, described communication terminal is dynamically adjusted the statistics generation of described service quality report and the cycle that sends according to current network state and higher layer applications demand.
25. multimedia communication method according to claim 21 is characterized in that, when described communication terminal mixes the service quality report of at least a Media Stream of carrying with described additional enhancing message,
Comprise the corresponding described reception report blocks of at least a Media Stream that is carried in this service quality report.
26. multimedia communication method according to claim 20, it is characterized in that, among the described step C, described receiving terminal is according to the network abstraction layer unit sequence number of the described video stream data that receives, the described network abstraction layer unit number that statistics is lost, generate described service quality report, send back to described transmitting terminal;
In the described steps A, described transmitting terminal calculates described accumulative total packet loss according to described network abstraction layer unit sequence number of losing, and adjusts described error elasticity protection strategy in view of the above.
27. multimedia communication method according to claim 26 is characterized in that, described receiving terminal is according to the service quality report that receives, analytical calculation network condition parameter; Described parameter comprises instant bandwidth, time-delay and shake end to end.
28. multimedia communication method according to claim 27; it is characterized in that; described transmitting terminal is provided with the error elasticity protection strategy series of different brackets, selects to use corresponding described error elasticity protection strategy according to described service quality report in described steps A.
29. multimedia communication method according to claim 28, it is characterized in that among the described step C, described receiving terminal is according to the network abstraction layer unit sequence number of the described video stream data that receives, add up the locating information that obtains losing video stream data, and it is sent back to described transmitting terminal;
In the described steps A, described transmitting terminal resends the described video stream data of losing and gives described receiving terminal according to described locating information of losing video stream data.
30. multimedia communication method according to claim 8, it is characterized in that, in described step e, described transmitting terminal obtains described locating information of losing band according to described transmission error message, carry out segmentation intraframe coding one by one by this being lost band, eliminate strategy to realize described error code diffusion.
31. multimedia communication method according to claim 30 is characterized in that, described segmentation intraframe coding one by one comprises following steps,
E1 is cut apart one group of continuous macro block from described losing the band, form new band, and remaining described macro block still belongs to the described band of losing, and enters step e 2;
E2 carries out intraframe coding to described new band, sends when next frame, and this new band is done conventional coding after this, enters step e 3;
When E3 encodes at next frame, judge whether the described band of losing also comprises untreated macro block, if then return step e 1, otherwise finish intraframe coding.
32. multimedia communication method according to claim 31, it is characterized in that, in described step e 1, the size of each described one group of continuous macro block of separating satisfies: after this was organized continuous macro block and carries out intraframe coding, the data transfer rate of this frame was in data transfer rate control range H.264.
33. multimedia communication method according to claim 8 is characterized in that, described step D comprises following substep,
The described receiving terminal of D1 detects and transmits mistake, and statistics transmits error message;
The described receiving terminal of D2 carries out video information and weighs synchronously after taking place to transmit mistake;
The described receiving terminal of D3 is implemented described error concealment strategy according to described transmission error message.
34. multimedia communication method according to claim 33 is characterized in that, among the described step D1, described receiving terminal detects and adds up the transmission error message according to the discontinuous situation of network abstraction layer unit sequence number.
35. multimedia communication method according to claim 34, it is characterized in that, in described step D1, the locating information that described receiving terminal is lost band according to the interruption situation acquisition of described network abstraction layer unit sequence number, this locating information comprises described frame number and described position of losing band at this frame of losing the band place.
36. multimedia communication method according to claim 35 is characterized in that, described error concealment strategy comprises step: described receiving terminal substitutes this and loses band with described respective strap of losing the former frame of band place frame.
37. any described multimedia communication method in 36 is characterized in that described error elasticity encoding scheme comprises improved " Tornado " correcting and eleting codes according to Claim 8;
Described improved " Tornado " correcting and eleting codes only generates the described check-node of one deck for one group of described back end.
38., it is characterized in that the transmission mistake among the described step B comprises data-bag lost or random bit mistake according to each described multimedia communication method in the claim 2 to 36.
39. a multimedia communication terminal comprises the basic function module that is used to realize multimedia communication, wherein comprises the coding/decoding module that is used to realize the multi-media decoding and encoding function, it is characterized in that, also comprises with lower module:
Error elasticity implements to transmit the control protocol module; be used for to be undertaken by the multi-medium data behind the described coding/decoding module coding transmitting at network side again after the error elasticity protection, the described multi-medium data from network side carried out passing to described coding/decoding module again after the error correction decode.
40. according to the described multimedia communication terminal of claim 39, it is characterized in that, also comprise with lower module:
Guard method and policy conferring module are used to be responsible for carry out error elasticity protection policy conferring between communicating pair, determine the protection strategy set, implement to transmit the control protocol module for described error elasticity and select;
Forward error correction block; be used to realize at least a forward error correction guard method; safeguard the relevant parameter of described forward error correction guard method; wherein said guard method and the described forward error correction block of policy conferring module controls are to realize unequal loss protection and adaptive hierarchical defencive function, and described error elasticity is implemented to transmit the control protocol module and realized error elasticity protection and error correction by calling this forward error correction block.
41. according to the described multimedia communication terminal of claim 40, it is characterized in that, also comprise
The error concealment module is used to realize the error concealment function;
Described coding/decoding module is used to realize the H.264 encoding and decoding of encoding and decoding standard, also is used for the error code diffusion and eliminates function;
Also comprise network condition analytical calculation module, be used for the analytical calculation network condition, and provide information to described error concealment module and described coding/decoding module.
42. according to the described multimedia communication terminal of claim 41, it is characterized in that, also comprise
Replenish enhancing extension of message processing module, be used to realize service quality report and network condition function of reporting, and report is sent to described network condition analytical calculation module.
43., it is characterized in that its transport layer is used to realize supporting the multimedia transmitting function of error elasticity based on described error elasticity real time transport protocol/transmit control protocol in real time according to the described multimedia communication terminal of claim 42;
Its application protocol layer comprises protection mechanism and policy conferring sublayer, is used to realize cascade protection and unequal loss protection function;
Its H.264 the video coding layer comprise and replenish to strengthen extension of message report layer, be used to realize the function of reporting that strengthens extension of message based on replenishing;
H.264, it comprises the forward error correction coding layer in network abstract layer, is used to realize the forward error correction coding function.
44., it is characterized in that the described basic function module that is used to realize multimedia communication comprises one of following or its combination in any according to each described multimedia communication terminal in the claim 39 to 43:
Main control module is used for being responsible for the control of whole terminal;
Subscriber Interface Module SIM is used for the demonstration of the mutual and information of responsible user's input and output;
Network communication module is used for being responsible for and network communicates, and provides lower floor to transmit passage;
Input and output and bottom layer driving module are used for being responsible for driving for hardware device;
Business module is used to realize high-level business;
The communication course control module is used to control communication process;
The application protocol module is used to realize the application protocol function;
Implement to transmit the control protocol module, be used for realizing implementing to transmit the control protocol function;
The network abstract layer module is used to realize the network abstract layer function;
The audio coding decoding module is used to realize the audio coding decoding function.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2006100690163A CN100466725C (en) | 2005-11-03 | 2005-11-03 | Multimedia communication method and terminal thereof |
| PCT/CN2006/002961 WO2007051425A1 (en) | 2005-11-03 | 2006-11-03 | A multimedia communication method and the terminal thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2006100690163A CN100466725C (en) | 2005-11-03 | 2005-11-03 | Multimedia communication method and terminal thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN1863302A true CN1863302A (en) | 2006-11-15 |
| CN100466725C CN100466725C (en) | 2009-03-04 |
Family
ID=37390610
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2006100690163A Expired - Fee Related CN100466725C (en) | 2005-11-03 | 2005-11-03 | Multimedia communication method and terminal thereof |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN100466725C (en) |
| WO (1) | WO2007051425A1 (en) |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101883282A (en) * | 2007-05-17 | 2010-11-10 | 索尼株式会社 | Information processing device and method |
| CN102075312A (en) * | 2011-01-10 | 2011-05-25 | 西安电子科技大学 | Video service quality-based hybrid selective repeat method |
| CN102438002A (en) * | 2011-08-10 | 2012-05-02 | 中山大学深圳研究院 | Video file data transmission method based on ad hoc network |
| CN102647412A (en) * | 2007-05-08 | 2012-08-22 | 高通股份有限公司 | Packet structure for mobile display digital interface |
| CN102956233A (en) * | 2012-10-10 | 2013-03-06 | 深圳广晟信源技术有限公司 | Extension structure of additional data for digital audio coding and corresponding extension device |
| CN101933329B (en) * | 2008-02-05 | 2013-05-29 | 汤姆逊许可公司 | Methods and apparatus for implicit block segmentation in video encoding and decoding |
| CN104272758A (en) * | 2012-02-23 | 2015-01-07 | 爱立信电视公司 | Methods and apparatus for managing network resources used by multimedia streams in a virtual pipe |
| CN105307050A (en) * | 2015-10-26 | 2016-02-03 | 何震宇 | HEVC-based network streaming media application system and method |
| CN105653530A (en) * | 2014-11-12 | 2016-06-08 | 上海交通大学 | High efficiency telescopic multimedia transmission, storage and present method |
| CN105916058A (en) * | 2016-05-05 | 2016-08-31 | 青岛海信宽带多媒体技术有限公司 | Streaming media buffer play method and device and display device |
| CN106921843A (en) * | 2017-01-18 | 2017-07-04 | 苏州科达科技股份有限公司 | Data transmission method and device |
| CN107181783A (en) * | 2016-03-11 | 2017-09-19 | 上汽通用汽车有限公司 | The method and apparatus for transmitting data using Ethernet in vehicle |
| CN107535002A (en) * | 2015-01-07 | 2018-01-02 | 奥兰治 | System for transmitting packet according to multiple access protocol |
| CN108702487A (en) * | 2017-11-20 | 2018-10-23 | 深圳市大疆创新科技有限公司 | The image transfer method and device of unmanned plane |
| CN109756468A (en) * | 2017-11-07 | 2019-05-14 | 中兴通讯股份有限公司 | A kind of restorative procedure of data packet, base station and computer readable storage medium |
| CN109891817A (en) * | 2018-02-08 | 2019-06-14 | Oppo广东移动通信有限公司 | Wireless communications method, terminal and the network equipment |
| CN110139150A (en) * | 2019-04-12 | 2019-08-16 | 北京物资学院 | A kind of method for processing video frequency and device |
| CN110233716A (en) * | 2019-05-31 | 2019-09-13 | 北京文香信息技术有限公司 | A kind of communication interaction method, apparatus, storage medium, terminal device and server |
| CN110740135A (en) * | 2019-10-21 | 2020-01-31 | 湖南新云网科技有限公司 | Same-screen data transmission method, device and system for multimedia classrooms |
| CN110769206A (en) * | 2019-11-19 | 2020-02-07 | 深圳开立生物医疗科技股份有限公司 | Electronic endoscope signal transmission method, device and system and electronic equipment |
| CN111010593A (en) * | 2019-11-08 | 2020-04-14 | 深圳市麦谷科技有限公司 | Method and device for packaging H.265 video data based on FLV format |
| CN111083510A (en) * | 2019-12-18 | 2020-04-28 | 深圳市麦谷科技有限公司 | Method and device for pushing HEVC (high efficiency video coding) video |
| CN111490984A (en) * | 2020-04-03 | 2020-08-04 | 上海宽创国际文化科技股份有限公司 | Network data coding and encryption algorithm thereof |
| CN111629282A (en) * | 2020-04-13 | 2020-09-04 | 北京创享苑科技文化有限公司 | Real-time erasure code coding redundancy dynamic adjustment method |
| CN111629279A (en) * | 2020-04-13 | 2020-09-04 | 北京创享苑科技文化有限公司 | Video data transmission method based on fixed-length format |
| CN111800388A (en) * | 2020-06-09 | 2020-10-20 | 盐城网之易传媒有限公司 | Media information processing method and media information processing device |
| CN112866178A (en) * | 2019-11-27 | 2021-05-28 | 北京沃东天骏信息技术有限公司 | Method and device for audio data transmission |
| CN113873340A (en) * | 2021-09-18 | 2021-12-31 | 恒安嘉新（北京）科技股份公司 | Data processing method, device, equipment, system and storage medium |
| CN113938881A (en) * | 2021-10-18 | 2022-01-14 | 上海华讯网络系统有限公司 | Transmission system and method suitable for internet data |
| CN114615549A (en) * | 2022-05-11 | 2022-06-10 | 北京搜狐新动力信息技术有限公司 | Streaming media seek method, client, storage medium and mobile device |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101800751B (en) * | 2010-03-09 | 2013-07-24 | 上海雅海网络科技有限公司 | Distributed real-time data-coding transmission method |
| CN103167319B (en) * | 2011-12-16 | 2016-06-22 | 中国移动通信集团公司 | The transfer processing method of a kind of Streaming Media, Apparatus and system |
| CN103118241B (en) * | 2012-02-24 | 2016-01-06 | 金三立视频科技（深圳）有限公司 | Based on the mobile video monitor streaming media self-adapting regulation method of 3G network |
| CN113381838A (en) * | 2020-03-09 | 2021-09-10 | 华为技术有限公司 | Data transmission method and communication device |
| CN114070458B (en) * | 2020-08-04 | 2023-07-11 | 成都鼎桥通信技术有限公司 | Data transmission method, device, equipment and storage medium |
| CN112311802B (en) * | 2020-11-05 | 2023-10-27 | 维沃移动通信有限公司 | Information transmission method and information transmission device |
| CN115189810B (en) * | 2022-07-07 | 2024-04-16 | 福州大学 | Low-delay real-time video FEC coding transmission control method |
| CN115844325B (en) * | 2022-11-17 | 2024-10-18 | 天津大学 | Distributed fNIRS brain function imaging system for super-scanning application |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| NL9100285A (en) * | 1991-02-19 | 1992-09-16 | Koninkl Philips Electronics Nv | TRANSMISSION SYSTEM, AND RECEIVER FOR USE IN THE TRANSMISSION SYSTEM. |
| KR100667738B1 (en) * | 2000-03-29 | 2007-01-11 | 삼성전자주식회사 | Apparatus for transmitting/receiving wireless packet and method thereof |
| EP1374429A4 (en) * | 2001-03-05 | 2009-11-11 | Intervideo Inc | Systems and methods for encoding and decoding redundant motion vectors in compressed video bitstreams |
| US7117423B2 (en) * | 2002-04-24 | 2006-10-03 | Georgia Tech Research Corp. | Methods and systems for multiple substream unequal error protection and error concealment |
| WO2004036760A1 (en) * | 2002-10-15 | 2004-04-29 | Koninklijke Philips Electronics N.V. | System and method for providing error recovery for streaming fgs encoded video over an ip network |
-
2005
- 2005-11-03 CN CNB2006100690163A patent/CN100466725C/en not_active Expired - Fee Related
-
2006
- 2006-11-03 WO PCT/CN2006/002961 patent/WO2007051425A1/en active Application Filing
Cited By (47)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102647412A (en) * | 2007-05-08 | 2012-08-22 | 高通股份有限公司 | Packet structure for mobile display digital interface |
| CN101883282A (en) * | 2007-05-17 | 2010-11-10 | 索尼株式会社 | Information processing device and method |
| CN101933329B (en) * | 2008-02-05 | 2013-05-29 | 汤姆逊许可公司 | Methods and apparatus for implicit block segmentation in video encoding and decoding |
| US8953679B2 (en) | 2008-02-05 | 2015-02-10 | Thomson Licensing | Methods and apparatus for implicit block segmentation in video encoding and decoding |
| CN102075312A (en) * | 2011-01-10 | 2011-05-25 | 西安电子科技大学 | Video service quality-based hybrid selective repeat method |
| CN102075312B (en) * | 2011-01-10 | 2013-03-20 | 西安电子科技大学 | Video service quality-based hybrid selective repeat method |
| CN102438002B (en) * | 2011-08-10 | 2016-08-03 | 中山大学深圳研究院 | A kind of based on the video file data transmission under Ad hoc network |
| CN102438002A (en) * | 2011-08-10 | 2012-05-02 | 中山大学深圳研究院 | Video file data transmission method based on ad hoc network |
| CN104272758B (en) * | 2012-02-23 | 2018-11-13 | 爱立信电视公司 | Method and apparatus for managing the Internet resources that media stream in virtual pipe uses |
| CN104272758A (en) * | 2012-02-23 | 2015-01-07 | 爱立信电视公司 | Methods and apparatus for managing network resources used by multimedia streams in a virtual pipe |
| CN102956233A (en) * | 2012-10-10 | 2013-03-06 | 深圳广晟信源技术有限公司 | Extension structure of additional data for digital audio coding and corresponding extension device |
| CN102956233B (en) * | 2012-10-10 | 2015-07-08 | 深圳广晟信源技术有限公司 | Extension structure of additional data for digital audio coding and corresponding extension device |
| CN105653530A (en) * | 2014-11-12 | 2016-06-08 | 上海交通大学 | High efficiency telescopic multimedia transmission, storage and present method |
| CN107535002A (en) * | 2015-01-07 | 2018-01-02 | 奥兰治 | System for transmitting packet according to multiple access protocol |
| CN107535002B (en) * | 2015-01-07 | 2021-09-14 | 奥兰治 | System for transmitting data packets according to a multiple access protocol |
| CN105307050B (en) * | 2015-10-26 | 2018-10-26 | 何震宇 | A kind of network flow-medium application system and method based on HEVC |
| CN105307050A (en) * | 2015-10-26 | 2016-02-03 | 何震宇 | HEVC-based network streaming media application system and method |
| CN107181783A (en) * | 2016-03-11 | 2017-09-19 | 上汽通用汽车有限公司 | The method and apparatus for transmitting data using Ethernet in vehicle |
| CN105916058A (en) * | 2016-05-05 | 2016-08-31 | 青岛海信宽带多媒体技术有限公司 | Streaming media buffer play method and device and display device |
| CN105916058B (en) * | 2016-05-05 | 2019-09-20 | 青岛海信宽带多媒体技术有限公司 | A kind of streaming media buffer playback method, device and display equipment |
| CN106921843A (en) * | 2017-01-18 | 2017-07-04 | 苏州科达科技股份有限公司 | Data transmission method and device |
| CN106921843B (en) * | 2017-01-18 | 2020-06-26 | 苏州科达科技股份有限公司 | Data transmission method and device |
| CN109756468A (en) * | 2017-11-07 | 2019-05-14 | 中兴通讯股份有限公司 | A kind of restorative procedure of data packet, base station and computer readable storage medium |
| CN109756468B (en) * | 2017-11-07 | 2021-08-17 | 中兴通讯股份有限公司 | Data packet repairing method, base station and computer readable storage medium |
| CN108702487A (en) * | 2017-11-20 | 2018-10-23 | 深圳市大疆创新科技有限公司 | The image transfer method and device of unmanned plane |
| CN109891817B (en) * | 2018-02-08 | 2020-09-08 | Oppo广东移动通信有限公司 | Wireless communication method, terminal and network equipment |
| CN109891817A (en) * | 2018-02-08 | 2019-06-14 | Oppo广东移动通信有限公司 | Wireless communications method, terminal and the network equipment |
| US10986659B2 (en) | 2018-02-08 | 2021-04-20 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Wireless communication method, terminal and network device |
| CN110139150A (en) * | 2019-04-12 | 2019-08-16 | 北京物资学院 | A kind of method for processing video frequency and device |
| CN110233716A (en) * | 2019-05-31 | 2019-09-13 | 北京文香信息技术有限公司 | A kind of communication interaction method, apparatus, storage medium, terminal device and server |
| CN110740135A (en) * | 2019-10-21 | 2020-01-31 | 湖南新云网科技有限公司 | Same-screen data transmission method, device and system for multimedia classrooms |
| CN111010593A (en) * | 2019-11-08 | 2020-04-14 | 深圳市麦谷科技有限公司 | Method and device for packaging H.265 video data based on FLV format |
| CN110769206A (en) * | 2019-11-19 | 2020-02-07 | 深圳开立生物医疗科技股份有限公司 | Electronic endoscope signal transmission method, device and system and electronic equipment |
| CN112866178B (en) * | 2019-11-27 | 2023-09-05 | 北京沃东天骏信息技术有限公司 | Method and device for transmitting audio data |
| CN112866178A (en) * | 2019-11-27 | 2021-05-28 | 北京沃东天骏信息技术有限公司 | Method and device for audio data transmission |
| CN111083510A (en) * | 2019-12-18 | 2020-04-28 | 深圳市麦谷科技有限公司 | Method and device for pushing HEVC (high efficiency video coding) video |
| CN111490984A (en) * | 2020-04-03 | 2020-08-04 | 上海宽创国际文化科技股份有限公司 | Network data coding and encryption algorithm thereof |
| CN111490984B (en) * | 2020-04-03 | 2022-03-29 | 上海宽创国际文化科技股份有限公司 | Network data coding and encryption algorithm thereof |
| CN111629282A (en) * | 2020-04-13 | 2020-09-04 | 北京创享苑科技文化有限公司 | Real-time erasure code coding redundancy dynamic adjustment method |
| CN111629279B (en) * | 2020-04-13 | 2021-04-16 | 北京创享苑科技文化有限公司 | Video data transmission method based on fixed-length format |
| CN111629279A (en) * | 2020-04-13 | 2020-09-04 | 北京创享苑科技文化有限公司 | Video data transmission method based on fixed-length format |
| CN111800388A (en) * | 2020-06-09 | 2020-10-20 | 盐城网之易传媒有限公司 | Media information processing method and media information processing device |
| CN113873340A (en) * | 2021-09-18 | 2021-12-31 | 恒安嘉新（北京）科技股份公司 | Data processing method, device, equipment, system and storage medium |
| CN113873340B (en) * | 2021-09-18 | 2024-01-16 | 恒安嘉新（北京）科技股份公司 | Data processing method, device, equipment, system and storage medium |
| CN113938881A (en) * | 2021-10-18 | 2022-01-14 | 上海华讯网络系统有限公司 | Transmission system and method suitable for internet data |
| CN114615549A (en) * | 2022-05-11 | 2022-06-10 | 北京搜狐新动力信息技术有限公司 | Streaming media seek method, client, storage medium and mobile device |
| CN114615549B (en) * | 2022-05-11 | 2022-09-20 | 北京搜狐新动力信息技术有限公司 | Streaming media seek method, client, storage medium and mobile device |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2007051425A1 (en) | 2007-05-10 |
| CN100466725C (en) | 2009-03-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1863302A (en) | Multimedia communication method and terminal thereof | |
| CN1859580A (en) | Multimedia data network realtime transfer method for supporting error elasticity | |
| CN1270496C (en) | Method and system for transmitting and receiving packets | |
| CN1914922A (en) | Content encoding, distribution, and reception method, device, and system, and program | |
| CN1190081C (en) | Method and apparatus for processing, transmitting and receiving dynamic image data | |
| CN1663282A (en) | Code conversion/transmission method and apparatus for movingpicture data, and code conversion/reception method and apparatus for movingpicture data | |
| CN1478355A (en) | Video coding | |
| CN1934865A (en) | Resizing of buffer in encoder and decoder | |
| CN1164058C (en) | Packet transmitter | |
| CN1176551C (en) | Image coding device and transmission system and quantization control method and average throughput algorithm thereof | |
| CN100407726C (en) | Method for real-time transmitting H.264 multimedia data | |
| CN1499794A (en) | Method of processing data pack in third layer of communicating appts. | |
| CN1650561A (en) | Audio data code conversion transmission method and code conversion reception method, device, system, and program | |
| CN1934825A (en) | Conveying parameters for broadcast/multicast sessions via a communication protocol | |
| CN1638321A (en) | Transmission apparatus and method, reception apparatus and method, storage medium, and program | |
| CN1729660A (en) | Packet transmission/reception device | |
| CN1557072A (en) | Data communications method and system using buffer size to calculate transmission rate for congestion control | |
| CN1666532A (en) | Image encoding method and image decoding method | |
| CN1839597A (en) | Quality of experience (QOE) measurement for wireless communication networks | |
| CN1682512A (en) | Method and apparatus for realizing multi-business branch on elastic package loop | |
| CN1863313A (en) | Method for monitoring service quality of H.264 multimedia communication | |
| CN1969528A (en) | Point-to-point repair response mechanism for point-to-multipoint transmission systems | |
| CN1846420A (en) | Transmission of embedded information relating to a quality of service | |
| CN1593065A (en) | Video encoding and decoding of foreground and background wherein picture is divided into slice | |
| CN1841989A (en) | Method and system for distributing high throughput bandwidth in a cable network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20171012 Address after: 221000 Jiangsu city in Xuzhou Province Economic Development Zone No. 15 District Jinshan paradise Building 1 unit 804 room Patentee after: Wang Miaomiao Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee before: Huawei Technologies Co., Ltd. |
|
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20090304 Termination date: 20171103 |