CN101166270A - Multimedia video communication method and system - Google Patents
Multimedia video communication method and system Download PDFInfo
- Publication number
- CN101166270A CN101166270A CN 200610139177 CN200610139177A CN101166270A CN 101166270 A CN101166270 A CN 101166270A CN 200610139177 CN200610139177 CN 200610139177 CN 200610139177 A CN200610139177 A CN 200610139177A CN 101166270 A CN101166270 A CN 101166270A
- Authority
- CN
- China
- Prior art keywords
- video data
- frame
- receiving terminal
- video
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 238000004891 communication Methods 0.000 title claims description 59
- 230000015556 catabolic process Effects 0.000 claims abstract description 61
- 238000006731 degradation reaction Methods 0.000 claims abstract description 61
- 230000005540 biological transmission Effects 0.000 claims abstract description 16
- 230000006870 function Effects 0.000 claims description 58
- 238000012544 monitoring process Methods 0.000 claims description 52
- 230000015572 biosynthetic process Effects 0.000 claims description 32
- 230000003247 decreasing effect Effects 0.000 claims description 25
- 230000006835 compression Effects 0.000 claims description 15
- 238000007906 compression Methods 0.000 claims description 15
- 230000008030 elimination Effects 0.000 claims description 13
- 238000003379 elimination reaction Methods 0.000 claims description 13
- 238000009432 framing Methods 0.000 claims description 8
- 238000005259 measurement Methods 0.000 claims description 7
- 230000000153 supplemental effect Effects 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 abstract description 12
- 238000012546 transfer Methods 0.000 abstract description 6
- 230000000694 effects Effects 0.000 abstract description 4
- 230000000875 corresponding effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000000737 periodic effect Effects 0.000 description 5
- 238000009792 diffusion process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 239000000047 product Substances 0.000 description 4
- 238000005538 encapsulation Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012856 packing Methods 0.000 description 3
- 230000001681 protective effect Effects 0.000 description 3
- 230000011664 signaling Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000006866 deterioration Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 101150034459 Parpbp gene Proteins 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000012467 final product Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- GOLXNESZZPUPJE-UHFFFAOYSA-N spiromesifen Chemical compound CC1=CC(C)=CC(C)=C1C(C(O1)=O)=C(OC(=O)CC(C)(C)C)C11CCCC1 GOLXNESZZPUPJE-UHFFFAOYSA-N 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The method and system for multimedia video comm raises effect and efficiency of mechanism for detecting and reporting transmission quality of H.264 video network. The receiving end transfers info of quality monitored through feedback with in-band mode so as to avoid using additional channel. Only in time when receiving error occurs, the receiving end feed back info of quality monitored so as to reduce overhead of bandwidth as well as provide feedback in real time. Using SEI expanded message with in-band mode to feed back info of quality monitored, the invention is compatible with current H.264 protocol completely. Calculating priority index of video data, and based on degradation degree of channel, the sending end selects N units of video data with highest priority index in current frame or next frame; and then the said video data with highest priority index are encoded in frame and sent to the receiving end to eliminate error at the receiving end.
Description
Technical field
The present invention relates to the communications field, particularly the multimedia video communication technology.
Background technology
(the International Telecommunication UnionTelecommunication Standardization Sector of ITU Telecommunication Standardization Sector, be called for short " ITU-T ") unite the H.264 video compression coding standard that ISO/IEC MPEG International Standards Organization formulates, become the mainstream standard in the multimedia communication at present gradually.A large amount of employing H.264 multimedia realtime communication product (as video conferencing, video telephone, 3G mobile communication terminal) and network flow-medium product is successively come out.Whether support H.264 to have become the key factor of decision product competitiveness in this market segment.Look back developing history H.264, we find, along with 3G (Third Generation) Moblie (The Third Generation is called for short " 3G ") appearance of system and developing rapidly of IP (Internet Protocol) network, video communication just progressively becomes one of main business of communication.ITU-T has formally issued H.264 standard in 2003 after having formulated H.261, H.263, H.263+ waiting video compression standard, it also is the main contents of MPEG-4 the 10th part simultaneously.Formulating H.264, the purpose of standard is to improve more effectively video coding efficient and its suitability to network, therefore can predict, along with H.264 formal promulgation be extensive use of, IP based network and 3G, the multimedia communication of B3G even 4G wireless network must enter a new stage of developing by leaps and bounds.H.264 on IP network, transmit, generally based on User Datagram Protoco (UDP) (User DatagramProtocol, be called for short " UDP ") and real time transport protocol (the RealTime Transfer Protocol on its upper strata, be called for short " RTP "), this standard is that the Internet engineering duty group (Internet Engineering Task Force is called for short " IETF ") is formulated.Real-time Transport Protocol is used very extensive, and the real-time application on the IP network (video, multimedia realtime communications such as audio frequency) substantially all adopts Real-time Transport Protocol to transmit.RTP structurally can both be suitable for for different media data type, but for every kind of concrete more upper-layer protocol, such as H.261, H.263, MPEG-1/-2/-4, MP3 etc., IETF can formulate RTP payload (Payload) the packing agreement at this agreement, the method of specified in more detail RTP encapsulation packing, for this concrete agreement through optimization.For H.264, also having corresponding ietf standard is H.264 Video (being used for the H.264 RTP payload format of video) of RFC 3984:RTP PayloadFormat for.Be the main standard that on IP network, transmits of video code flow H.264 before this standard mesh, use very extensive.In field of video communication, each mainstream vendor's product all is based on RFC's 3984.
H.264 video is the main agreement of following multimedia communication, and the network that following multimedia communication is used mainly is to be the packet-switched network and the wireless network of representative with IP.This two big class network all can't provide good service quality (Quality of Service, be called for short " QoS "), so video transmits on network and will inevitably be subjected to various transmission mistakes and the influence of packet loss, thereby makes the communication quality reduction.We know that a topmost problem is that IP network realizes that " doing the best " (best effort) transmits, and can not guarantee to transmit the QoS of vision signal.Particularly to the H.264 code stream through efficient compressed encoding, problem is more outstanding.Transmission as possible on the IP network can not guarantee the QoS of real-time video communication, is in particular in three aspects: data-bag lost, time delay and delay variation.Wherein, data-bag lost is to recovering the quality influence maximum of video, because H.264 Coding Compression Algorithm is used estimating motion and motion compensation technique, in case there is data-bag lost to exist, not only influence current decoded picture, and can influence the subsequent decoding image, be i.e. error code diffusion.The influence that the error code diffusion couple recovers video quality is very big, has only in conjunction with coding side and decoding end and unites anti-error code, could avoid the error code diffusion fully.
Error elasticity (Error Resilience) be meant that transfer mechanism has that guard against false takes place or the ability that after mistake takes place, can correct with certain ability (mistake intensity can be corrected within the specific limits fully; Surpass certain limit, can only partly correct).In the multimedia communication environment widely in future, whether a kind of video transfer mechanism has error elasticity will be very crucial.
There is multiple error resilience mechanism at present, such as Forward Error Correction (Forward ErrorCorrection, be called for short " FEC "), automatic repeat request (Automatic Retransmission Request is called for short " ARQ "), Error Concealment (error concealment), message source and channel combined coding (JointSource-Channel Coding, be called for short " JSCC "), interweave (Interleaving) eliminates error code diffusion or the like.Transmit on packet networks for video H.264, FEC is a kind of very practical technology, and effect is fine.This method mainly adopts multiple error correction coding to encode for the data that will protect, and essence is to form data redundancy, thereby increases the wrong ability of resisting.Certainly, other error resilience mechanism also has them to have a few separately.But no matter take which kind of delivery quality protection mechanism, the problem that must solve is the QoS situation of awareness network, obtains the report information about delivery quality, such as packet loss, and time-delay, numerical value such as shake.Only know about network delivery quality situation, could select correct guard method.In quality good fully or only be subjected to minimal effect in, just be not suitable for the method that adopts protective capability very strong because the strong cost of protective capability is that expense is big, it is many to lower efficiency; Have only when delivery quality is subjected to relatively having a strong impact on, can adopt the strong method of protective capability.
H.264 standard distinguishing feature is divided into system two independently layers exactly: i.e. video coding layer (VCL) and network adaptation layer (NAL).The former main task is to carry out video data encoding compression efficiently; The latter encapsulates packing according to network characteristic to data, realizes the adaptive of video QOS and network QOS.
Transmit the cooperation agreement that control protocol (RealTime Transfer Control Protocol is called for short " RTCP ") is a Real-time Transport Protocol in real time, also formulate by IETF.RTCP is mainly used in the control and the report of Real-time Transport Protocol.The main contents of report are exactly the information relevant with QoS.The method for reporting that RTCP takes is a periodic report.Promptly periodically all participant communications of control data bags in session (Session is appreciated that into a meeting in the multi-party communication) and RTP packet adopt same distribution mechanisms.Underlying protocol must provide multiplexed (for example use independent udp port number etc.) of data and control data bag.In RFC3550, advise being fixed as 5% for the session bandwidth that RTCP increases.RTCP finishes four functions:
1, basic function-for the real-time multimedia data delivery quality provides feedback report mechanism, this is the organic component of RTP as transport-layer protocols.This feedback function transmits transmit leg report (SR) by RTCP and recipient's report (RR) realizes.
2, RTCP is that a permanent transport layer sign is transmitted in each RTP source, is called canonical name (CNAME).Synchronisation source (Synchronization Rsource is called for short " SSRC ") is identified to be found may change when conflict or program are restarted, so the recipient need follow the tracks of each participant by CNAME.
3, preceding two functions need all participants all to send the RTCP packet, therefore in order to allow RTP can increase the number of participant pari passu, and speed that must control RTCP packet.
4, the 4th is optional function---transmit the least possible control information.
In the communications field, signaling (also can be described as message etc.) has two kinds of load modes, signaling and data is transmitted be called in-band method in same channel, and signaling is independent of to transmit outside the data transmission channel and is called out-band method.
RTCP has used the outer transmission mode of band, and RTCP is other channel with respect to RTP, H.323 waiting when realizing under the concrete protocol frame, need open extra logical channel, thereby consume extra resource.
In addition, RTCP bring the QoS report mechanism can be provided in because adopt the periodic report method, caused the expense of additional networks bandwidth, the highlyest can reach 5%.If deterioration (Congestion) appears in network, cause transmitting QoS and descend, the added flow of RTCP generation will make that problem worsens more so.Though rtcp protocol is just recognized this problem in exploitation, thereby an optional function is provided: require to transmit the least possible control information.But, if transmit very few, levels of precision that will Impact Report, this is a contradiction.
Summary of the invention
In view of this, main purpose of the present invention is to provide a kind of multimedia video communication method and system, to improve the H.264 detection of video network delivery quality and the effect and the efficient of report mechanism.
For achieving the above object, the invention provides a kind of multimedia video communication method, comprise following steps:
Receiving terminal feeds back the delivery quality monitoring information of paid-in video data to transmitting terminal with in-band method;
Described transmitting terminal is adjusted the compression of video data encoded attributes that needs send to described receiving terminal according to the described delivery quality monitoring information of band internal feedback.
Wherein, described transmitting terminal is carried on described delivery quality monitoring information in the supplemental enhancement information SEI extended message, feeds back to described transmitting terminal with in-band method.
In this external described method, it is one of following that the delivery quality monitoring information of described SEI extended message carrying comprises at least:
Make mistakes frame number, error macro block sequence number, real time transport protocol RTP loss rate, instant refresh frame refreshes sign.
In this external described method, described RTP loss rate obtains by following steps:
When receiving the RTP bag, write the correct or fault sign of expression to the formation that is used for adding up according to the result who receives;
When described queue full, will represent in this formation that the fault sign number obtains described RTP loss rate divided by the length of formation.
In this external described method, the length of the described formation that is used to add up is calculated in the following manner and is provided with:
The length of the average received code check of the length=measurement period of formation * receiving terminal/each RTP bag.
In this external described method, when described receiving terminal monitors the video data generation mistake of receiving, feed back the delivery quality monitoring information to described transmitting terminal with in-band method;
Described transmitting terminal is adjusted the compression of video data encoded attributes that need send to described receiving terminal in the following manner:
The compression of video data encoded attributes that adjustment will send to described receiving terminal is the compressed encoding attribute with wrong elimination ability.
In this external described method, described receiving terminal sends to comprise in the delivery quality monitoring information of described transmitting terminal and is used to locate the sign that transmits the video data of makeing mistakes and the information of indicating channel degradation;
Described transmitting terminal is determined describedly to can be used to carry out the wrong video data of eliminating to what described receiving terminal sent by following steps:
Described transmitting terminal is according to the channel degradation degree, in current video frame or back to back next frame of video, select the video data of N the highest unit of priority index to carry out intraframe coding, the video data after the intraframe coding is sent to described receiving terminal as the described video data that is used for carrying out wrong elimination.
In this external described method, determine the size of N in the following manner according to the channel degradation degree:
Formerly set m thresholding β
1To β
m, m+1 positive integer N
1To N
M+1, wherein m is a positive integer, β
i＜β
I+1, 1≤i≤m-1;
If the index of expression channel degradation degree is less than β
1, then the N value is N
1
If the index of expression channel degradation degree is more than or equal to β
iAnd less than β
I+1, 1≤i≤m-1, then N value N
I+1
If the index of expression channel degradation degree is more than or equal to β
m, N value N then
M+1
Wherein, the big more expression channel of desired value of expression channel degradation degree is poor more.
In this external described method, described channel degradation degree is indicated with the RTP loss rate;
Described video data of makeing mistakes be designated frame of video sequence number and/or macro block sequence number;
Described unit is a macro block.
In this external described method, described priority index one of meets the following conditions or its combination in any:
The increasing function of the motion severe degree that described priority index is a macro block;
Described priority index is the decreasing function of macro block for the influence degree of other macro block in the subsequent video frame;
Described priority index is the decreasing function that is used to adjudicate the cost function value of predictive coding type;
The decreasing function of described priority index to be macro block be refreshed in each frame of predetermined number before the number of times of coded system in the framing.
In this external described method, described priority index is one or more the weighted linear combination in following four functions, and the weight coefficient of each function correspondence is a nonnegative number:
Increasing function about the motion severe degree of macro block;
One about the decreasing function of macro block for the influence degree of other macro block in the subsequent video frame;
Decreasing function about the cost function value that is used to adjudicate the predictive coding type;
A decreasing function that in each frame of predetermined number before, is refreshed the number of times of coded system in the framing about macro block.
In this external described method, if described video data of makeing mistakes is the inter prediction encoding frame, then described transmitting terminal is according to the RTP loss rate, from the Error Location, chooses N the highest macro block of priority index at every turn and sends to described receiving terminal after intra-encoded.
In this external described method, if described video data of makeing mistakes is an intracoded frame, and the RTP loss rate is less than first thresholding that formerly is provided with, and then described receiving terminal suspends decoding immediately, resends instant refresh frame by the described transmitting terminal of described SEI extended message request;
Described transmitting terminal in time refreshes coding with frame of video, then this instant refresh frame is used for carrying out the wrong video data of eliminating and sends to described receiving terminal as described.
In this external described method, if described video data of makeing mistakes is an intracoded frame, and the RTP loss rate forbids then that greater than first thresholding that formerly is provided with described receiving terminal resends instant refresh frame because of this reception makes mistakes to described transmitting terminal request.
The present invention also provides a kind of multimedia video communication system, and receiving terminal comprises: the feedback unit that feeds back the delivery quality monitoring information of paid-in video data with in-band method to transmitting terminal;
Transmitting terminal comprises: receive with in-band method described receiving terminal feedback the delivery quality monitoring information the unit and adjust the unit of the compression of video data encoded attributes that needs send to this receiving terminal according to the delivery quality monitoring information of feedback.
Wherein, described transmitting terminal is carried on described delivery quality monitoring information in the supplemental enhancement information SEI extended message, feeds back to described transmitting terminal with in-band method;
It is one of following that the delivery quality monitoring information of described SEI extended message carrying comprises at least:
Make mistakes frame number, error macro block sequence number, RTP loss rate, instant refresh frame refreshes sign.
In this external described system, described receiving terminal also comprises the unit that calculates the RTP loss rate, and this unit comprises:
The formation that is used to add up;
When receiving the RTP bag, write the module of representing correct or fault sign to described formation according to the result who receives;
When described queue full, expression fault sign number in this formation is obtained the module of described RTP loss rate divided by the length of formation;
Wherein, the length of the average received code check of the length=measurement period of described formation * receiving terminal/each RTP bag.
In this external described system, described receiving terminal also comprises the monitoring video data and receives wrong unit, if this unit monitors the reception mistake, then indicates described feedback unit to carry out feedback operation;
Described transmitting terminal is adjusted the video data that need send to described receiving terminal in the following manner:
In the video data that sends to described receiving terminal, add and be used for the video data that this receiving terminal carries out wrong elimination.
In this external described system, described receiving terminal sends to and comprises the sign that transmits the video data of makeing mistakes and the information of indicating channel degradation in the delivery quality monitoring information of described transmitting terminal;
Described transmitting terminal also comprises:
Be used to calculate the unit of the priority index of video data;
According to the channel degradation degree, by calculating for the priority index of the video data of a plurality of units in current video frame or the back to back next frame of video, in current video frame or back to back next frame of video, select the video data of N the highest unit of priority index, the unit that notice is used for video compression coding carries out intraframe coding for the video data of this N unit, with the unit of the video data that carries out wrong elimination as receiving terminal.
In this external described system, described receiving terminal also comprises: the type judging unit of judging the video data frame predictive coding type of makeing mistakes; With judge that the RTP loss rate is whether less than the degradation judging unit of first thresholding that formerly is provided with;
If described type judging unit judges that the video data of makeing mistakes is an intracoded frame, and described degradation judging unit judges that the RTP loss rate less than first thresholding, then is provided with the sign that requirement refreshes immediately in the SEI extended message request that sends to described transmitting terminal;
If described type judging unit judges that the video data of makeing mistakes is an intracoded frame, and described degradation judging unit judgement RTP loss rate forbids then that less than first thresholding described feedback unit sends described SEI extended message because of this reception makes mistakes;
If described type judging unit judges that the video data of makeing mistakes is the inter prediction encoding frame, then send described SEI extended message to described transmitting terminal, report described video data delivery quality monitoring information.
The present invention also provides error concealment methods in a kind of multimedia video communication, comprises following steps:
The sign of the video data that receiving terminal is made mistakes transmission and the information of indicating channel degradation send to transmitting terminal;
Described transmitting terminal is according to the channel degradation degree, send to described receiving terminal after the video data of N the unit that selection priority index is the highest in current video frame or back to back next frame of video carries out intraframe coding, described receiving terminal uses the intra-coded video data of a described N unit to carry out mistake and eliminates.
Wherein, described channel degradation degree is indicated with the RTP loss rate;
Described video data of makeing mistakes be designated frame of video sequence number and/or macro block sequence number;
Described unit is a macro block.
In this external described method, described priority index one of meets the following conditions or its combination in any:
The increasing function of the motion severe degree that described priority index is a macro block;
Described priority index is the decreasing function of macro block for the influence degree of other macro block in the subsequent video frame;
Described priority index is the decreasing function that is used to adjudicate the cost function value of predictive coding type;
The decreasing function of described priority index to be macro block be refreshed in each frame of predetermined number before the number of times of coded system in the framing.
The present invention also provides wrong elimination system in a kind of multimedia video communication, and receiving terminal comprises: the unit that is used for the information of the sign of the video data of loading error occurring and indicating channel degradation is sent to transmitting terminal; With use video data to carry out the wrong unit of eliminating from transmitting terminal;
Transmitting terminal comprises:
Be used to calculate the unit of the priority of video data;
Reception is from the unit of receiving terminal information; With
According to the channel degradation degree, after carrying out intraframe coding, the video data of N the unit that selection priority is the highest in the video data of makeing mistakes sends to the unit of described receiving terminal.
By finding that relatively the main distinction of technical scheme of the present invention and prior art is that receiving terminal with respect to feeding back with out-band method in the prior art, has been avoided the use of extra channel with in-band method feedback delivery quality monitoring information.
Receiving terminal is feedback delivery quality monitoring information when reception makes mistakes only, with respect to the mode of periodic feedback in the prior art, has reduced the expense of bandwidth, has improved the real-time of feedback.
Use the SEI extended message with in-band method feedback delivery quality monitoring information, can compatible fully existing H.264 agreement.
Add up the RTP loss rate by the mode of formation, can calculate the RTP loss rate easily, and comparatively detailed RTP packet loss situation can be provided.
Transmitting terminal calculates the priority index of video data, and according to the channel degradation degree, in present frame or next frame, send to receiving terminal after beginning to select the video data of N the highest unit of priority index intra-encoded from the position of makeing mistakes, the mistake that is used for receiving terminal is eliminated.The mode that obtains N according to the channel degradation degree has multiple, for example can mark off the interval of a plurality of channel degradation degree by a plurality of thresholdings, there is the different N of a correspondence in each interval, N should be less when the channel degradation degree was higher in principle, preferably can also take into account in the practical operation and consider wrong elimination ability and extra bandwidth expense.Because use the data of intraframe coding less when the channel degradation degree is higher, and the data volume that intraframe coding produces is more, so the transmission environment is further worsened.Because what pass is the highest partial video data of priority index, so the help maximum of the receiving terminal mistake being eliminated when transmitting same data volume.
Proposed the method for definite priority index, having comprised: the priority index is the increasing function of the motion severe degree of macro block; The priority index is the decreasing function of macro block for the influence degree of other macro block in the subsequent video frame; The priority index is the decreasing function that is used to adjudicate the cost function value of predictive coding type; The decreasing function of priority index to be macro block be refreshed in each frame of predetermined number before the number of times of coded system in the framing.Thereby strengthened the operability that the priority index is calculated.
If the video data of makeing mistakes is an intracoded frame, and the RTP loss rate is greater than first thresholding that formerly is provided with, and then receiving terminal no longer sends instant refresh frame request.The RTP loss rate is excessive to show that current channel packet loss is serious, even transmitting terminal resends instant refresh frame, this instant refresh frame is still probably destroyed in transport process, causes receiving terminal to continue to require to refresh instant refresh frame in a short time.Because instant refresh frame data volume is big, not only so repeats to refresh instant refresh frame and can effectively not improve receiving terminal picture quality, over-burden also may to cause channel, causes channel degradation more and more serious.By not sending instant refresh frame request, prevented the further deterioration of channel.
Description of drawings
Fig. 1 is the two-way video system configuration schematic diagram according to first embodiment of the invention;
Fig. 2 is the structural representation of SEI message;
Fig. 3 eliminates schematic diagram according to the H.264 video delivery quality monitoring report and the interaction error of first embodiment of the invention;
Fig. 4 is the multimedia video communication method flow chart of receiving terminal in the first embodiment of the invention;
Fig. 5 is the VDTQI formation schematic diagram that is used to calculate the RTP loss rate in the first embodiment of the invention;
Fig. 6 is an example of SEI extended message in the first embodiment of the invention;
Fig. 7 is the multimedia video communication method flow chart of transmitting terminal in the first embodiment of the invention;
Fig. 8 is the multimedia video communication system configuration schematic diagram according to four embodiment of the invention.
Embodiment
For making the purpose, technical solutions and advantages of the present invention clearer, the present invention is described in further detail below in conjunction with accompanying drawing.
First execution mode of the present invention relates to multimedia video communication method, comprising the mode of H.264 video delivery quality monitoring report with based on mutual error concealment methods.Typically be applied to H.264 two-way video system, as visual telephone etc.The general configuration of two-way video system as shown in Figure 1.Under the two-way video communication pattern, all existing encoder of each terminal of communication system has decoder again.Based on this characteristic, receiving terminal can be charged to information in supplemental enhancement information (Supplemental Enhancement Information is called for short " the SEI ") territory of this terminal encoding code stream, mails to far-end.So neither take special feedback channel, can accelerate the speed of feedback information again.
The receiving terminal decoder sends to the local terminal encoder with information (as the error messages etc. of decoding), these information are written as the extended message into SEI at encoder, carry out network abstract layer (NetworkAbstraction Layer afterwards, abbreviation " NAL ") encapsulation, add the NAL header, be sent to the decoder of transmitting terminal then, the transmitting terminal decoder is understood the SEI extended message, needs according to circumstances, can give the local terminal encoder with SEI extended message context notification, encoder is made corresponding action according to SEI extended message content.
H.264, the multiple mechanism that can carry out extension of message that provides is provided, and that the present invention uses is SEI.H.264 in SEI is defined, the data representation of SEI zone is independent with video data encoder, provides in the description of the using method of SEI NAL in agreement H.264.H.264 the base unit of code stream is network abstraction layer unit (Network Abstraction Layer Unit, be called for short " NALU "), NALU can carry various H.264 data types, such as video sequence parameter (Sequence parameters), image parameter (Picture parameters), Slice data (being concrete view data), and SEI message data.SEI is used to transmit various message, supports extension of message.Therefore be used to be transmitted as specific purpose and self-defining message in the SEI territory, and can not influence based on the compatibility of video communication system H.264.The NALU of carrying SEI message is called SEI NALU.A SEI NALU contains one or more SEI message.Each SEI message contains some variablees, mainly is payloadType and payloadSize, and these variablees have indicated the type and size of message load.At Annex H.264 D.8, the syntax and the meaning of one's words of some H.264 SEI message commonly used have been defined in D.9.
The structure of SEI message as shown in Figure 2, each SEI message is made up of SEI header and SEI payload.The SEI header comprises two code words: a code word provides the type of load in the SEI message, and another code word provides the size of load.When load type is represented to 0xFE with a byte 0x00 between 255 the time 0, when type is represented to 0xFFFE with two byte 0xFF00 between 511 the time 256, when type greater than 511 the time method for expressing by that analogy, the user can self-defined any multiple load type like this.Wherein type 0 has been defined as information specific such as caching period, image timing etc. in Class1 8 standards.The SEI territory of definition can be deposited abundant customized information according to demand in this shows H.264.For the H.264 decoder of not supporting to resolve these customized informations, can abandon the data in the SEI territory automatically.Therefore, charging to useful self-defined information in the SEI territory can not influence based on the compatibility of video communication system H.264.
Fig. 3 illustrates the schematic diagram that H.264 monitoring of video delivery quality is reported and interaction error is eliminated in the present embodiment.In order to be illustrated more clearly in, the flow process that need carry out receiving terminal and transmitting terminal respectively is illustrated below.
The handling process of receiving terminal as shown in Figure 4, in step 401, receiving terminal decoder receiving video data bag, and real-time statistics represents the index of channel degradation degree, in the present embodiment with the index of RTP loss rate as the channel degradation degree.Wherein, the RTP loss rate can be calculated according to following mode.
Set up transfer of data monitoring formation in advance.This formation is a round-robin queue, write full data after, empty again, use for next measurement period.Measurement period (perhaps being called the timing statistics window) is t
FQ, unit is second.
Queue length
Wherein,
Be the average received code check of receiving terminal, unit is bps (generally should be consistent with the encoder bit rate of transmitting terminal, therefore also can use the code check value of coding control approx), t
FQ* R
RevPromptly at t
FQInterior code stream length, the L of obtaining
RTPBe the length (unit is a bit) of a RTP bag.
For video compression data,, then in formation, write 0, otherwise write 1 if receiving terminal correctly receives a RTP bag with the transmission of RTP packet form.As shown in Figure 5.
When queue full, the number C of " 1 " in formation this moment of beginning real-time statistics
1, and calculate RTP loss rate R
Loss:
The formation content has write down current network transmission packet loss situation in real time, and prediction also directly reflects the mass change of receiving terminal decoded picture.R
LossBe data delivery quality coefficient VDTQI.VDTQI is big more, illustrates that current network conditions is poor more.
Certainly reflect that H.264 the parameter of video data delivery quality is not only one of VDTQI, out of Memory can also be arranged, such as: the picture number of makeing mistakes, error macro block sequence number etc.The index that VDTQI just portrays the video data delivery quality from this dimension of RTP packet loss degree in fact.If RTP packet loss, must cause wherein the losing of H.264 video data of encapsulation, so and the picture number of makeing mistakes, these information of error macro block sequence number are relevant and corresponding, and both portray the delivery quality of video data respectively the back from image aspect and macro block aspect.
After this enter step 402, receiving terminal judges whether to find that video packets of data receives mistake when video packets of data is decoded, if then enter step 403, otherwise return step 401.
In step 403, receiving terminal is further judged the type of coding of video data in the packet of makeing mistakes, if inter prediction encoding frame (as the P frame) then enters step 404, if intracoded frame (as the I frame) then enters step 405.H.264 Ding Yi image encoding type has IDR, P, B, four kinds of PB.Wherein the IDR type is the intraframe coding type, and P and category-B type are the inter prediction encoding types.When intraframe coding, refresh (IDR) image immediately and can make the previous reference picture cache invalidate, no longer with reference to the image before the IDR image, thereby the IDR image has very strong heavy net synchronization capability during afterwards picture decoding.In the serious channel of some packet losses and error code, the mode that can take irregularly to transmit the IDR image further improves error-resilient performance H.264.
In step 404, after this receiving terminal returns step 401 to transmitting terminal feedback SEI extended message.Present embodiment with in-band method feedback delivery quality monitoring information, with respect to feeding back with out-band method in the prior art, has been avoided the use of extra channel by the SEI extended message.Because only feedback delivery quality monitoring information when reception makes mistakes with respect to the mode of periodic feedback in the prior art, has reduced the expense of bandwidth, improved the real-time of feedback.
The form of SEI extended message has comprised SEI type, SEI loaded length, the frame number of makeing mistakes, error macro block sequence number, has represented the data delivery quality coefficient (VDTQI) and the IDR frame of RTP loss rate to refresh sign.SEI type wherein, the SEI loaded length is collectively referred to as the SEI header, and remaining information is called the SEI payload.
Picture number is 260 if receiving terminal decoder discovery decoding makes mistakes, the error macro block sequence number is 56 in this frame, data delivery quality coefficient is 5, the I frame data are not lost, SEI type value is defined as 0x0A, expression SEI message is the error message of feedback decoding end, and the SEI payload length is 5 bytes, and the SEI extended message of then writing as shown in Figure 6.Latter two byte is that the IDR frame refreshes sign.If these two bytes are 0x00, then showing does not need transmitting terminal to refresh immediately, if be 0x01, then showing needs transmitting terminal to refresh immediately.In step 404,, do not need to refresh immediately because just the inter prediction encoding frame is made mistakes.
In step 405, the data delivery quality coefficient (VDTQI) that receiving terminal judges whether to represent the RTP loss rate resends IDR frame because of this reception makes mistakes to the transmitting terminal request if then forbid receiving terminal, otherwise enters step 406 greater than the first thresholding α.If receiving terminal find the I frame data have lose and current VDTQI greater than the thresholding of setting, show that then current channel packet loss is serious, even transmitting terminal resends the IDR frame, this IDR frame is still probably destroyed in transmission course, causes receiving terminal to continue to require to refresh the IDR frame in a short time.Because IDR frame data amount is big, not only so repeats to refresh the IDR frame and can effectively not improve receiving terminal picture quality, over-burden also may to cause channel, causes channel congestion more and more serious.Therefore, in this case, receiving terminal no longer sends the request of IDR frame, after VDTQI reduces, more just at that time situation take corresponding measure.
In step 406, receiving terminal suspends decoding immediately, writes the SEI extended message, and with IDR frame in the SEI extended message refresh the sign be changed to 0x01, send this SEI extended message.After this get back to step 401.
The handling process of transmitting terminal as shown in Figure 7.
In step 701, transmitting terminal carries out sending behind the compressed encoding to video data, the priority index of while sequence of calculation image, and according to priority index sorts from large to small and backs up.In the present embodiment, the priority index is the preferential index of macro block updating (Macroblock Refreshment Priority Index is called for short " MRPI "), and the present invention does not repel other priority index.
The relevant factor of determining MRPI comprises:
(1) the motion severe degree of macro block can be measured with the norm size (geometric meaning is exactly the length of vector) of motion vector.This numerical value is then big more, and MRPI should be high more.
(2) this macro block is for the influence degree of other macro block in the subsequent video frame.In a follow-up N frame, directly rely on or depend on indirectly the macro block number of this macro block.This numerical value is then big more, and MRPI should be high more.
(3) H.264 in original judgement carry out the principle that intraframe coding still is an inter prediction encoding.Be generally each possible predictive coding mode and assess its COST (cost) functional value respectively, select the formal coded system of a kind of conduct of COST functional value minimum.The COST functional value is then big more, and MRPI should be low more.
Is (4) whether this macro block refreshed coded system in the framing in preceding some frames? if have, then the value of current MRPI should be low, if do not have, then the numerical value of MRPI should be high.Refresh if having, then refreshing frequency is many more, and current MRPI numerical value is low more.
Therefore, should be the function of the quantized value of above factor for MRPI as variable.Should be the increasing function of factor (1), the increasing function of factor (2), the decreasing function of factor (3), the decreasing function of factor (4).
Certainly it is also conceivable that other factors.Therefore, the form of MRPI can have a variety of, as long as satisfy above condition.
The present invention does not limit concrete MRPI form.Be a kind of concrete form of MRPI below:
With v
1, v
2, v
3, v
4(wherein each symbology vector promptly can comprise a plurality of scalar variables) expression factor (1), (2), (3), (4) corresponding quantitative variable.So
MRPI(v
1，v
2，v
3，v
4)＝λ
1f
1(v
1)+λ
2f
2(v
2)+λ
3f
3(v
3)+λ
4f
4(v
4)(3)
λ wherein
1λ
2λ
3λ
4It is non-negative weight.
f
1(v
1) be variable v
1Increasing function; For v
1Be the situation of vector, i.e. f
1(v
1) be v
1The increasing function of each component;
f
2(v
2) be variable v
2Increasing function; For v
2Be the situation of vector, i.e. f
2(v
2) be v
2The increasing function of each component;
f
3(v
3) be variable v
3Decreasing function; For v
3Be the situation of vector, i.e. f
3(v
3) be v
3The decreasing function of each component;
f
4(v
4) be variable v
4Decreasing function; For v
4Be the situation of vector, i.e. f
4(v
4) be v
4The decreasing function of each component.
After this enter step 702, transmitting terminal judges whether to receive the SEI extended message that carries quality-monitoring information, if then the reception mistake has taken place in explanation, enters step 703, otherwise gets back to step 701.
In step 703, transmitting terminal judges according to the value that IDR frame in the SEI extended message refreshes sign whether receiving terminal requires to refresh the IDR frame, if then enter step 707, otherwise enters step 704.
In step 704, transmitting terminal judges that the data delivery quality coefficient (VDTQI) of expression RTP loss rate in the SEI extended message whether greater than the second thresholding β, if then enter step 705, otherwise enters step 706.
In step 705, transmitting terminal, is chosen the maximum macro block of N1 MRPI value and is carried out intraframe coding and transmission in current video frame or back to back next frame of video from the Error Location.When receiving VDTQI information,, carry out this strategy at present frame if also be able to do in time; If have little time, be over such as the big odd encoder of major part of present frame, remaining macro block N inadequately is individual, just handles at next frame.After this get back to step 701.
In step 706, transmitting terminal, is chosen the maximum macro block of N2 MRPI value and is carried out intraframe coding and transmission in current video frame or back to back next frame of video from the Error Location, after this gets back to step 701.Wherein N2 is greater than N1.
Data delivery quality coefficient (VDTQI) has in fact been represented the channel degradation degree, step 704 is according to the channel degradation degree to the basic thought of step 706, send to receiving terminal after selecting N the highest macro block of priority index intra-encoded, the mistake that is used for receiving terminal is eliminated.The mode that obtains N according to the channel degradation degree has multiple, and step 704 to 706 is a kind of comparatively simple modes, and N should be less when the channel degradation degree was higher in principle, preferably can also take into account in the practical operation and consider wrong elimination ability and extra bandwidth expense.Because use the data of intraframe coding less when the channel degradation degree is higher, and the data volume that intraframe coding produces is more, so the transmission environment is further worsened.Because what pass is the highest partial video data of priority index, so the help maximum of the receiving terminal mistake being eliminated when transmitting same data volume.
In step 707, according to the requirement of receiving terminal, transmitting terminal carries out IDR frame coding, and sends the IDR frame.After this get back to step 701.
Also relate to multimedia video communication method in second embodiment of the invention, second execution mode and first execution mode are roughly the same, difference only is, in the step 704 to 706 of first execution mode, transmitting terminal is only selected N1 or N2 the maximum macro block of MRPI value to be used for mistake according to thresholding β to eliminate, and second execution mode selects to be used for the number of macro block of the MRPI value maximum of mistake elimination according to a plurality of thresholdings.Specifically, can set 3 thresholding β 1, β 2 and β 3, β 1＜β 2＜β 3.If VDTQI is less than β 1 then choose the maximum macro block of N1 MRPI value and carry out intraframe coding and send, if VDTQI more than or equal to β 1 less than β 2 then choose the maximum macro block of N2 MRPI value and carry out intraframe coding and send, if less than β 3 then choose the maximum macro block of N3 MRPI value and carry out intraframe coding and send, VDTQI chooses the maximum macro block of N4 MRPI value greater than 3 of β and carries out also transmission of intraframe coding VDTQI more than or equal to β 2.With respect to first execution mode, more thresholding is the situation of adaptive channel better.
Also relate to multimedia video communication method in third embodiment of the invention, the 3rd execution mode and first execution mode are roughly the same, difference only is, first execution mode transmits the delivery quality monitoring information by the SEI extended message, and the 3rd execution mode uses rtcp protocol to transmit the delivery quality monitoring information.May not contain with RTP loss rate in the RTCP message, but carry concrete packet loss information, but can calculate the RTP loss rate or other can represent the index of channel degradation degree at transmitting terminal by these concrete packet loss information equally.
The 3rd execution mode is revised with respect to first execution mode and is: in step 401 and step 406, receiving terminal changes into by RTCP message and transmits the delivery quality monitoring information; In step 402, not to make mistakes when receiving packet by receiving terminal to trigger the feedback that transmits the delivery quality monitoring information, but first error logging is got off, trigger to transmit sending in the lump when the delivery quality monitoring information feeds back by periodic mechanism (as clock signal).In step 702, transmitting terminal changes reception RTCP message into and gets final product.
From the 3rd execution mode as can be seen, the error concealment methods that the present invention proposes can be applied to the mode of various feedbacks, is not limited to in-band method feedback transmission delivery quality monitoring information.
In addition, can also use SEI and RTCP to feed back simultaneously, the content of feedback can have the division of labor certainly.
The 4th execution mode of the present invention relates to a kind of multimedia video communication system, as shown in Figure 8, comprises sending ending equipment (being called for short " transmitting terminal ") and receiving device (abbreviation " receiving terminal ").
Receiving terminal further comprises:
Decoding unit is used to decode from the H.264 video data that sends and monitors the reception mistake of video data, and these video datas are carried in the RTP bag and are transferred to receiving terminal;
RTP loss rate computing unit is used to calculate the Loss Rate of the RTP bag that decoding unit receives, and this index can the indicating channel degradation.This unit further comprises:
The formation that is used to add up;
When receiving the RTP bag, write the module of representing correct or fault sign to formation according to the result who receives;
When queue full, expression fault sign number in this formation is obtained the module of RTP loss rate divided by the length of formation;
Wherein, the length of the average received code check of the length=measurement period of formation * receiving terminal/each RTP bag.
The type of makeing mistakes judging unit is used to judge that decoding unit receives the predictive coding type of the video data frame of makeing mistakes.Two kinds of judged results are arranged, and a kind of is intraframe coding,, another kind is an inter prediction encoding.If inter prediction encoding is directly to notify the feedback message generation unit to generate the SEI extended message, if intraframe coding then indicates the degradation judging unit to feed back according to the needs that judge whether of channel degradation degree, if judged result is to need feedback, then notify the feedback message generation unit to generate the SEI extended message.
The degradation judging unit, be used for obtaining the RTP loss rate from RTP loss rate computing unit, and judge that whether the RTP loss rate is less than first thresholding that formerly is provided with, judged result is outputed to the feedback message generation unit, and notice feedback message generation unit generates the SEI extended message according to judged result.
The feedback message generation unit is used to generate the SEI extended message that carries the delivery quality monitoring information, feeds back to transmitting terminal with in-band method by coding unit.The delivery quality monitoring information of SEI extended message carrying comprises: the frame number of makeing mistakes, error macro block sequence number, RTP loss rate and IDR frame refresh sign.Wherein the RTP loss rate is used to indicate the channel degradation degree.
The feedback message generation unit generates different SEI extended message according to the different judged results of make mistakes type judging unit and degradation judging unit, specifically:
The type judging unit judges that the video data of makeing mistakes is the inter prediction encoding frame if make mistakes, and then generates the SEI extended message of carrying the delivery quality monitoring information, and the sign that requires the IDR frame to refresh wherein is not set;
The type judging unit judges that the video data of makeing mistakes is an intracoded frame if make mistakes, and the degradation judging unit judges that the RTP loss rate is less than first thresholding, then generate the SEI extended message of carrying the delivery quality monitoring information, the sign that requires the IDR frame to refresh wherein is set;
The type judging unit judges that the video data of makeing mistakes is an intracoded frame if make mistakes, and degradation judging unit judgement RTP loss rate does not then generate and send the SEI extended message because of this reception makes mistakes greater than first thresholding.
Transmitting terminal further comprises:
Decoding unit is used for receiving the delivery quality monitoring information that receiving terminal feeds back with in-band method, specifically receives and resolves the SEI extended message from receiving terminal, and the delivery quality monitoring information is offered selected cell.Comprise the sign that transmits the video data make mistakes and the information of indicating channel degradation in the delivery quality monitoring information
Priority calculation unit is used to calculate the priority index of video data, and result calculated is sorted and keeps.Usually can be unit calculating priority level index with the macro block, this priority index can be MRPI, sees the associated description in first execution mode for details.
Selected cell, channel degradation degree in the delivery quality monitoring information that provides according to decoding unit, in current video frame or back to back next frame of video, select the video data of N the highest unit of priority index, and notify coding unit selection result.
Coding unit is used for compressed encoding and sends video data, if selected cell has indication, then carries out intraframe coding with the video data to N unit of selected cell indication.
The various unit or the module that it is pointed out that among the present invention to be mentioned all are logics, can merge or further split, and just belong to scope of the present invention as long as can finish function corresponding.For example the feedback message generation unit and the coding unit of receiving terminal can be merged into feedback unit, the function of feedback unit is the intersection of feedback message generation unit and coding unit.Each unit can also can be realized on different physical entities respectively in realization on the same physical entity, even a unit can be realized with being total to by a plurality of physical entities in distributed mode.
Though pass through with reference to some of the preferred embodiment of the invention, the present invention is illustrated and describes, but those of ordinary skill in the art should be understood that and can do various changes to it in the form and details, and without departing from the spirit and scope of the present invention.
Claims (24)
1. a multimedia video communication method is characterized in that, comprises following steps:
Receiving terminal feeds back the delivery quality monitoring information of paid-in video data to transmitting terminal with in-band method;
Described transmitting terminal is adjusted the compression of video data encoded attributes that needs send to described receiving terminal according to the described delivery quality monitoring information of band internal feedback.
2. multimedia video communication method according to claim 1 is characterized in that, described transmitting terminal is carried on described delivery quality monitoring information in the supplemental enhancement information SEI extended message, feeds back to described transmitting terminal with in-band method.
3. multimedia video communication method according to claim 2 is characterized in that, it is one of following that the delivery quality monitoring information of described SEI extended message carrying comprises at least:
Make mistakes frame number, error macro block sequence number, real time transport protocol RTP loss rate, instant refresh frame refreshes sign.
4. multimedia video communication method according to claim 3 is characterized in that, described RTP loss rate obtains by following steps:
When receiving the RTP bag, write the correct or fault sign of expression to the formation that is used for adding up according to the result who receives;
When described queue full, will represent in this formation that the fault sign number obtains described RTP loss rate divided by the length of formation.
5. multimedia video communication method according to claim 4 is characterized in that, the length of the described formation that is used to add up is calculated in the following manner and is provided with:
The length of the average received code check of the length=measurement period of formation * receiving terminal/each RTP bag.
6. according to each described multimedia video communication method in the claim 1 to 4, it is characterized in that, when described receiving terminal monitors the video data generation mistake of receiving, feed back the delivery quality monitoring information to described transmitting terminal with in-band method;
Described transmitting terminal is adjusted the compression of video data encoded attributes that need send to described receiving terminal in the following manner:
The compression of video data encoded attributes that adjustment will send to described receiving terminal is the compressed encoding attribute with wrong elimination ability.
7. multimedia video communication method according to claim 6 is characterized in that, described receiving terminal sends to comprise in the delivery quality monitoring information of described transmitting terminal and is used to locate the sign that transmits the video data of makeing mistakes and the information of indicating channel degradation;
Described transmitting terminal is determined describedly to can be used to carry out the wrong video data of eliminating to what described receiving terminal sent by following steps:
Described transmitting terminal is according to the channel degradation degree, in current video frame or back to back next frame of video, select the video data of N the highest unit of priority index to carry out intraframe coding, the video data after the intraframe coding is sent to described receiving terminal as the described video data that is used for carrying out wrong elimination.
8. multimedia video communication method according to claim 7 is characterized in that, determines the size of N in the following manner according to the channel degradation degree:
Formerly set m thresholding β
1To β
m, m+1 positive integer N
1To N
M+1, wherein m is a positive integer, β
i＜β
I+1, 1≤i≤m-1;
If the index of expression channel degradation degree is less than β
1, then the N value is N
1
If the index of expression channel degradation degree is more than or equal to β
iAnd less than β
I+1, 1≤i≤m-1, then N value N
I+1
If the index of expression channel degradation degree is more than or equal to β
m, N value N then
M+1
Wherein, the big more expression channel of desired value of expression channel degradation degree is poor more.
9. multimedia video communication method according to claim 7 is characterized in that, described channel degradation degree is indicated with the RTP loss rate;
Described video data of makeing mistakes be designated frame of video sequence number and/or macro block sequence number;
Described unit is a macro block.
10. multimedia video communication method according to claim 9 is characterized in that, described priority index one of meets the following conditions or its combination in any:
The increasing function of the motion severe degree that described priority index is a macro block;
Described priority index is the decreasing function of macro block for the influence degree of other macro block in the subsequent video frame;
Described priority index is the decreasing function that is used to adjudicate the cost function value of predictive coding type;
The decreasing function of described priority index to be macro block be refreshed in each frame of predetermined number before the number of times of coded system in the framing.
11. multimedia video communication method according to claim 10 is characterized in that, described priority index is one or more the weighted linear combination in following four functions, and the weight coefficient of each function correspondence is a nonnegative number:
Increasing function about the motion severe degree of macro block;
One about the decreasing function of macro block for the influence degree of other macro block in the subsequent video frame;
Decreasing function about the cost function value that is used to adjudicate the predictive coding type;
A decreasing function that in each frame of predetermined number before, is refreshed the number of times of coded system in the framing about macro block.
12. multimedia video communication method according to claim 9, it is characterized in that, if described video data of makeing mistakes is the inter prediction encoding frame, then described transmitting terminal is according to the RTP loss rate, from the Error Location, choose N the highest macro block of priority index at every turn and send to described receiving terminal after intra-encoded.
13. multimedia video communication method according to claim 9, it is characterized in that, if described video data of makeing mistakes is an intracoded frame, and the RTP loss rate is less than first thresholding that formerly is provided with, then described receiving terminal suspends decoding immediately, resends instant refresh frame by the described transmitting terminal of described SEI extended message request;
Described transmitting terminal in time refreshes coding with frame of video, then this instant refresh frame is used for carrying out the wrong video data of eliminating and sends to described receiving terminal as described.
14. multimedia video communication method according to claim 9, it is characterized in that, if described video data of makeing mistakes is an intracoded frame, and the RTP loss rate forbids then that greater than first thresholding that formerly is provided with described receiving terminal resends instant refresh frame because of this reception makes mistakes to described transmitting terminal request.
15. a multimedia video communication system is characterized in that receiving terminal comprises: the feedback unit that feeds back the delivery quality monitoring information of paid-in video data with in-band method to transmitting terminal;
Transmitting terminal comprises: receive with in-band method described receiving terminal feedback the delivery quality monitoring information the unit and adjust the unit of the compression of video data encoded attributes that needs send to this receiving terminal according to the delivery quality monitoring information of feedback.
16. multimedia video communication according to claim 15 system is characterized in that described transmitting terminal is carried on described delivery quality monitoring information in the supplemental enhancement information SEI extended message, feeds back to described transmitting terminal with in-band method;
It is one of following that the delivery quality monitoring information of described SEI extended message carrying comprises at least:
Make mistakes frame number, error macro block sequence number, RTP loss rate, instant refresh frame refreshes sign.
17. multimedia video communication according to claim 16 system is characterized in that, described receiving terminal also comprises the unit that calculates the RTP loss rate, and this unit comprises:
The formation that is used to add up;
When receiving the RTP bag, write the module of representing correct or fault sign to described formation according to the result who receives;
When described queue full, expression fault sign number in this formation is obtained the module of described RTP loss rate divided by the length of formation;
Wherein, the length of the average received code check of the length=measurement period of described formation * receiving terminal/each RTP bag.
18. multimedia video communication according to claim 15 system is characterized in that, described receiving terminal also comprises the monitoring video data and receives wrong unit, if this unit monitors the reception mistake, then indicates described feedback unit to carry out feedback operation;
Described transmitting terminal is adjusted the video data that need send to described receiving terminal in the following manner:
In the video data that sends to described receiving terminal, add and be used for the video data that this receiving terminal carries out wrong elimination.
19. multimedia video communication according to claim 15 system is characterized in that, described receiving terminal sends to and comprises the sign that transmits the video data of makeing mistakes and the information of indicating channel degradation in the delivery quality monitoring information of described transmitting terminal;
Described transmitting terminal also comprises:
Be used to calculate the unit of the priority index of video data;
According to the channel degradation degree, by calculating for the priority index of the video data of a plurality of units in current video frame or the back to back next frame of video, in current video frame or back to back next frame of video, select the video data of N the highest unit of priority index, the unit that notice is used for video compression coding carries out intraframe coding for the video data of this N unit, with the unit of the video data that carries out wrong elimination as receiving terminal.
20. multimedia video communication according to claim 16 system is characterized in that described receiving terminal also comprises: the type judging unit of judging the video data frame predictive coding type of makeing mistakes; With judge that the RTP loss rate is whether less than the degradation judging unit of first thresholding that formerly is provided with;
If described type judging unit judges that the video data of makeing mistakes is an intracoded frame, and described degradation judging unit judges that the RTP loss rate less than first thresholding, then is provided with the sign that requirement refreshes immediately in the SEI extended message request that sends to described transmitting terminal;
If described type judging unit judges that the video data of makeing mistakes is an intracoded frame, and described degradation judging unit judgement RTP loss rate forbids then that less than first thresholding described feedback unit sends described SEI extended message because of this reception makes mistakes;
If described type judging unit judges that the video data of makeing mistakes is the inter prediction encoding frame, then send described SEI extended message to described transmitting terminal, report described video data delivery quality monitoring information.
21. error concealment methods in the multimedia video communication is characterized in that, comprises following steps:
The sign of the video data that receiving terminal is made mistakes transmission and the information of indicating channel degradation send to transmitting terminal;
Described transmitting terminal is according to the channel degradation degree, send to described receiving terminal after the video data of N the unit that selection priority index is the highest in current video frame or back to back next frame of video carries out intraframe coding, described receiving terminal uses the intra-coded video data of a described N unit to carry out mistake and eliminates.
22. error concealment methods in the multimedia video communication according to claim 21 is characterized in that, described channel degradation degree is indicated with the RTP loss rate;
Described video data of makeing mistakes be designated frame of video sequence number and/or macro block sequence number;
Described unit is a macro block.
23. error concealment methods in the multimedia video communication according to claim 22 is characterized in that, described priority index one of meets the following conditions or its combination in any:
The increasing function of the motion severe degree that described priority index is a macro block;
Described priority index is the decreasing function of macro block for the influence degree of other macro block in the subsequent video frame;
Described priority index is the decreasing function that is used to adjudicate the cost function value of predictive coding type;
The decreasing function of described priority index to be macro block be refreshed in each frame of predetermined number before the number of times of coded system in the framing.
24. wrong elimination system in the multimedia video communication is characterized in that receiving terminal comprises: the unit that is used for the information of the sign of the video data of loading error occurring and indicating channel degradation is sent to transmitting terminal; With use video data to carry out the wrong unit of eliminating from transmitting terminal;
Transmitting terminal comprises:
Be used to calculate the unit of the priority of video data;
Reception is from the unit of receiving terminal information; With
According to the channel degradation degree, after carrying out intraframe coding, the video data of N the unit that selection priority is the highest in the video data of makeing mistakes sends to the unit of described receiving terminal.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200610139177 CN100558167C (en) | 2006-10-16 | 2006-10-16 | Multimedia video communication method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200610139177 CN100558167C (en) | 2006-10-16 | 2006-10-16 | Multimedia video communication method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101166270A true CN101166270A (en) | 2008-04-23 |
| CN100558167C CN100558167C (en) | 2009-11-04 |
Family
ID=39334705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 200610139177 Expired - Fee Related CN100558167C (en) | 2006-10-16 | 2006-10-16 | Multimedia video communication method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100558167C (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101651790A (en) * | 2008-08-15 | 2010-02-17 | 康佳集团股份有限公司 | TV, and device and method for indicating TV signal intensity |
| CN102025993A (en) * | 2010-12-17 | 2011-04-20 | 深圳中兴力维技术有限公司 | Video transmission method and system based on H.264 |
| CN102316360A (en) * | 2010-07-09 | 2012-01-11 | 华为终端有限公司 | Video refreshing method, device and system |
| CN102474661A (en) * | 2009-08-06 | 2012-05-23 | 高通股份有限公司 | Encapsulating three-dimensional video data in accordance with transport protocols |
| CN102594778A (en) * | 2011-01-11 | 2012-07-18 | 中兴通讯股份有限公司 | Realizing method and system for stream media feedback |
| CN103119895A (en) * | 2010-09-16 | 2013-05-22 | 德国电信股份有限公司 | Method of and system for measuring quality of audio and video bit stream transmissions over a transmission chain |
| CN103621015A (en) * | 2011-06-01 | 2014-03-05 | 锐迪讯有限公司 | Systems, methods, and media for identifying degraded video call links |
| CN104113722A (en) * | 2014-06-19 | 2014-10-22 | 南京熊猫电子股份有限公司 | Wireless video conference transmission method |
| CN106961627A (en) * | 2017-03-24 | 2017-07-18 | 西安理工大学 | A kind of method for improving real-time video play quality |
| CN106998328A (en) * | 2017-03-30 | 2017-08-01 | 北京奇艺世纪科技有限公司 | A kind of video transmission method and device |
| CN107872675A (en) * | 2016-09-26 | 2018-04-03 | 联芯科技有限公司 | The restorative procedure of video data based on H264 and the data terminal of transmission |
| WO2018121775A1 (en) * | 2016-12-30 | 2018-07-05 | SZ DJI Technology Co., Ltd. | System and methods for feedback-based data transmission |
| US10924745B2 (en) | 2016-12-30 | 2021-02-16 | SZ DJI Technology Co., Ltd. | Image processing method and device |
| CN112995214A (en) * | 2021-04-26 | 2021-06-18 | 安心智能(武汉)信息技术有限公司 | Real-time video transmission system, method and computer readable storage medium |
| US11070732B2 (en) | 2016-12-30 | 2021-07-20 | SZ DJI Technology Co., Ltd. | Method for image processing, device, unmanned aerial vehicle, and receiver |
| CN113612584A (en) * | 2017-09-25 | 2021-11-05 | 杜比实验室特许公司 | System and method for optimizing load of multipath data transmission |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI411307B (en) * | 2010-08-11 | 2013-10-01 | Univ Hungkuang | Video playback fluency priority link source channel coding system |
-
2006
- 2006-10-16 CN CN 200610139177 patent/CN100558167C/en not_active Expired - Fee Related
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101651790A (en) * | 2008-08-15 | 2010-02-17 | 康佳集团股份有限公司 | TV, and device and method for indicating TV signal intensity |
| US8878912B2 (en) | 2009-08-06 | 2014-11-04 | Qualcomm Incorporated | Encapsulating three-dimensional video data in accordance with transport protocols |
| CN102474661A (en) * | 2009-08-06 | 2012-05-23 | 高通股份有限公司 | Encapsulating three-dimensional video data in accordance with transport protocols |
| CN102474661B (en) * | 2009-08-06 | 2015-11-25 | 高通股份有限公司 | According to delivery protocol encapsulation three dimensional video data |
| US9131279B2 (en) | 2009-08-06 | 2015-09-08 | Qualcomm Incorporated | Preparing video data in accordance with a wireless display protocol |
| CN102316360B (en) * | 2010-07-09 | 2015-11-25 | 华为终端有限公司 | Video refreshing method, Apparatus and system |
| CN102316360A (en) * | 2010-07-09 | 2012-01-11 | 华为终端有限公司 | Video refreshing method, device and system |
| WO2012003808A1 (en) * | 2010-07-09 | 2012-01-12 | 华为终端有限公司 | Method, device, and system for video refresh |
| CN103119895A (en) * | 2010-09-16 | 2013-05-22 | 德国电信股份有限公司 | Method of and system for measuring quality of audio and video bit stream transmissions over a transmission chain |
| CN102025993B (en) * | 2010-12-17 | 2014-01-08 | 深圳中兴力维技术有限公司 | Video transmission method and system based on H.264 |
| CN102025993A (en) * | 2010-12-17 | 2011-04-20 | 深圳中兴力维技术有限公司 | Video transmission method and system based on H.264 |
| WO2012094996A1 (en) * | 2011-01-11 | 2012-07-19 | 中兴通讯股份有限公司 | Method, system and device for streaming media feedback |
| CN102594778A (en) * | 2011-01-11 | 2012-07-18 | 中兴通讯股份有限公司 | Realizing method and system for stream media feedback |
| CN103621015B (en) * | 2011-06-01 | 2017-03-22 | 阿瓦亚公司 | System, method, and device for identifying degraded video call links |
| CN103621015A (en) * | 2011-06-01 | 2014-03-05 | 锐迪讯有限公司 | Systems, methods, and media for identifying degraded video call links |
| CN104113722A (en) * | 2014-06-19 | 2014-10-22 | 南京熊猫电子股份有限公司 | Wireless video conference transmission method |
| CN107872675A (en) * | 2016-09-26 | 2018-04-03 | 联芯科技有限公司 | The restorative procedure of video data based on H264 and the data terminal of transmission |
| CN107872675B (en) * | 2016-09-26 | 2020-06-16 | 联芯科技有限公司 | H.264-based video data repair method and transmission data terminal |
| WO2018121775A1 (en) * | 2016-12-30 | 2018-07-05 | SZ DJI Technology Co., Ltd. | System and methods for feedback-based data transmission |
| CN109983775A (en) * | 2016-12-30 | 2019-07-05 | 深圳市大疆创新科技有限公司 | The system and method sent for the data based on feedback |
| US10911750B2 (en) | 2016-12-30 | 2021-02-02 | SZ DJI Technology Co., Ltd. | System and methods for feedback-based data transmission |
| US10924745B2 (en) | 2016-12-30 | 2021-02-16 | SZ DJI Technology Co., Ltd. | Image processing method and device |
| US11070732B2 (en) | 2016-12-30 | 2021-07-20 | SZ DJI Technology Co., Ltd. | Method for image processing, device, unmanned aerial vehicle, and receiver |
| CN106961627B (en) * | 2017-03-24 | 2019-07-02 | 北京金风易通科技有限公司 | A method of improving real-time video play quality |
| CN106961627A (en) * | 2017-03-24 | 2017-07-18 | 西安理工大学 | A kind of method for improving real-time video play quality |
| CN106998328A (en) * | 2017-03-30 | 2017-08-01 | 北京奇艺世纪科技有限公司 | A kind of video transmission method and device |
| CN106998328B (en) * | 2017-03-30 | 2019-12-13 | 北京奇艺世纪科技有限公司 | Video transmission method and device |
| CN113612584A (en) * | 2017-09-25 | 2021-11-05 | 杜比实验室特许公司 | System and method for optimizing load of multipath data transmission |
| CN112995214A (en) * | 2021-04-26 | 2021-06-18 | 安心智能(武汉)信息技术有限公司 | Real-time video transmission system, method and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN100558167C (en) | 2009-11-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100558167C (en) | Multimedia video communication method and system | |
| CN100456834C (en) | Method for monitoring service quality of H.264 multimedia communication | |
| CN100466725C (en) | Multimedia communication method and terminal thereof | |
| CN100459717C (en) | Error elimination of compressed video transmission based on H.264 | |
| US8015474B2 (en) | Adaptive forward error correction | |
| CN101636983B (en) | Reducing effects of packet loss in video transmissions | |
| CN1801944B (en) | Method and device for coding and decoding video | |
| CN1934865B (en) | Method and device for resizing of buffer in encoder and decoder | |
| CN101371312B (en) | For the system and method for the error resilience in video communication system and Stochastic accessing | |
| US8127040B2 (en) | Signaling buffer parameters indicative of receiver buffer architecture | |
| WO2006105713A1 (en) | Video transmission protection method based on h.264 | |
| CN101611551A (en) | The improvement system and method that is used for the error resilience of video communication system | |
| KR20100005132A (en) | Feedback based scalable video coding | |
| CN101562615A (en) | Transmission method for MPEG-4 code based multimedia data stream self-adapting network bandwidth | |
| US8750373B2 (en) | Delay aware rate control in the context of hierarchical P picture coding | |
| WO2006027661A1 (en) | System and method for using redundant representations in streaming applications | |
| WO2009154656A1 (en) | Network abstraction layer (nal)-aware multiplexer with feedback | |
| CN102316360B (en) | Video refreshing method, Apparatus and system | |
| JP2005033556A (en) | Data transmitter, data transmitting method, data receiver, data receiving method | |
| JP2002064472A (en) | Communication system, transmitter, and method of preventing transmission error | |
| Bahl et al. | H. 263 based video codec for real-time visual communications over wireless radio networks | |
| Liu et al. | RTP/AVPF compliant feedback for error resilient video coding in conversational applications | |
| CN115396733A (en) | Video frame transmission method, device, equipment, storage medium and product | |
| KR20070068742A (en) | A hybrid scalable encoding/decoding method of moving pictures, which includes parity layers as ones of video layers and terminal used therein | |
| Nafaa et al. | A bandwidth-efficient application level framing protocol for H. 264 video multicast over wireless LANs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20160504 Address after: California, USA Patentee after: SNAPTRACK, Inc. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20091104 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |