CN103024374A

CN103024374A - Transmission of video data

Info

Publication number: CN103024374A
Application number: CN2012104032873A
Authority: CN
Inventors: P.卡尔松; A.杰弗里莫夫; S.萨布林; D.赵
Original assignee: Skype Ltd Ireland
Current assignee: Skype Ltd Ireland
Priority date: 2011-10-20
Filing date: 2012-10-22
Publication date: 2013-04-03

Abstract

A method of transmitting video data includes at an encoder encoding the video data as a plurality of frames, including intermediate frames, each of which is encoded based on at least one reference frame and at least some of which are encoded based on multiple reference frames; at the encoder maintaining for each frame a current list of reference frames; and transmitting the plurality of intermediate frames, each intermediate frame being transmitted in associate with a current list of reference frames for that frame.

Description

The transmission of video data

Technical field

The present invention relates to the transmission of video data.

Background technology

Because the transmission of video data needs high bit rate, so known have various dissimilar compressions to reduce the quantity of the required bit of transduction activity image.When compressed video data, between the resolution of the quantity of the bit that need to transmit by transmission channel and live image and accuracy, have compromise.

Video image transmits in frame, and each frame comprises one group of for example macro block of 8 * 8.Macro block for example can be 16 * 16 block of pixels.In order to generate the image of disappearance, should there be ideally all frames according to particular order.

The known compress technique that is used for the transmission video data is to use so-called reference frame.

When the piece of compressed video data, frame (intra frame, I-frame) in the cataloged procedure delta frame.Frame is the compressed version of frame in the frame, and it only need not just can be decompressed with reference to other frame by the information in the use I-frame self.They are called as key frame sometimes.The frame of another type also is generated, and is called as inter-frame (inter frame) here, and it is generated by the prediction interframe encode based on reference frame.Reference frame can be the frame of front, and perhaps it can be the different frame much earlier or later in a succession of frame.

Reference frame can be inter-frame oneself, maybe can be frame in the frame.

In the method for video coding in early days, one type inter-frame (being called as the P frame) is the frame of based single front normally.A kind of dissimilar inter-frame be based on one more early with a more late frame (such frame is called as the B frame in the MPEG2 standard).

Nearer video encoding standard allows to generate any specific inter-frame with a plurality of reference frames.H.264/AVC standard is a kind of such standard.This each macro block that gives the particular frame of video encoder for being encoded is selected the option of particular reference frame.Usually, best frame is the frame of front, but has wherein extra reference frame can improve the situation of compression efficiency and/or video quality.H.264 standard allows nearly that 16 reference frames exist jointly.According to standard H.264, encoder is all preserved the reference frame lists that comprises short-term and long term reference frame.The picture buffer DPB of decoding is used in and holds reference frame on the decoder, is used for being used during decoding by decoder.Long term reference frame (LTR) the more than one frame that is used for encoding, and short-term reference frame (STR) is used for the single frame of only encoding usually.Yet for a plurality of reference frames, STR can be by the frame of several subsequently codings with for referencial use.Specific frame can use the mixing of LTR and STR.

Although the use of a plurality of reference frames can improve compression efficiency and/or video quality, may cause difficulty, because no longer supposing encoder, decoder when generating inter-frame, may use the agreement of what kind.

Reference frame lists is managed by storage management control operation order (MMCO order), and it is short-term reference and long term reference that described order is made frame flag by encoder, and removes short-term and long-term frame from reference listing.In case order is generated at encoder, it just is sent to decoder with the frame of its impact on transmission channel.Therefore, decoder can access similarly MMCO order and assessment how based on be stored on the decoder previous information and by the new information that the MMCO order the provides described frame of decoding.

Difficulty cause be because: if the MMCO order is lost during the transmission, then decoder no longer has the information corresponding with the information of this frame that is used at encoder encoding, and owing to the failure of decoder because of this reason, in fact so that bit stream is invalid.

Summary of the invention

According to one aspect of the present invention, a kind of method that transmits video data is provided, described method comprises:

Be a plurality of frames with video data encoding on encoder, described a plurality of frames comprise reference frame and intermediate frame (intermediate frame), and at least some in them are encoded based on a plurality of reference frames;

On encoder, preserve the current list of reference frame for each frame; With

Transmit described a plurality of frame, each frame is transmitted explicitly with the current list for the reference frame of this frame.

In the present context, intermediate frame is from reference frame coding (for example, generation or prediction) frame.Should be pointed out that reference frame itself can be at intermediate frame front generation or prediction.Term " reference frame " refers to be used for generating or predicting the frame of other (centre) frame.

Preferably, the frame number of identifying each frame is transmitted with described frame, so that in the mapping that can keep on the decoder between described frame number and the reference listing.

Another aspect of the present invention provides a kind of method of a succession of frame that represents video data of decoding, and described frame comprises reference frame and intermediate frame, and each in the described intermediate frame is encoded based at least one reference frame, and described method comprises:

Be received in explicitly on the encoder the current list of the reference frame of preserving for this frame with each intermediate frame;

Each intermediate frame of decoding in the following manner, i.e. described decoding are to carry out with reference to the reference frame that is mentioned in the current list for described frame.

Another aspect of the present invention provides a kind of encoder, it comprises: being used for video data encoding is the device of a plurality of frames, described a plurality of frame comprises intermediate frame, in the described intermediate frame each is encoded based at least one reference frame, and in the described intermediate frame at least some are encoded based on a plurality of reference frames; Be used for preserving for each intermediate frame the device of the current list of reference frame; And the device that is used for transmitting described a plurality of intermediate frames, each intermediate frame is transmitted explicitly with the current list for the reference frame of this frame.

Another aspect of the present invention provides a kind of computer program, it comprises program code devices, when being carried out by processor, program code devices implements following steps: be a plurality of frames with video data encoding, described a plurality of frame comprises intermediate frame, in the described intermediate frame each is encoded based at least one reference frame, and in the described intermediate frame at least some are encoded based on a plurality of reference frames; Preserve the current list of reference frame for each intermediate frame; And transmitting described a plurality of intermediate frame, each intermediate frame is transmitted explicitly with the current list for the reference frame of this frame.

Another aspect of the present invention provides a kind of decoder, be used for a succession of frame that decoding represents video data, described frame comprises intermediate frame, in the described intermediate frame each is encoded based at least one reference frame, and described decoder comprises: the device that is used for being received in explicitly with each intermediate frame the current list of the reference frame of preserving for this intermediate frame on the encoder; And the decoding device of the intermediate frame that can operate to decode, wherein said decoding device can be operated to decode in the following manner in the described intermediate frame at least some, i.e. described decoding is to carry out with reference to the reference frame of mentioning in the current list for described intermediate frame.

The method of this decoding can comprise based on the order that receives with video data safeguards the decoding buffer, and wherein said decoding buffer identification is used for the reference frame of decoding intermediate frame.

The method of this decoding can also comprise the step that the detection frame not yet is received; And use for the current list of a upper frame that receives identify for the decoding at least one intermediate frame subsequently reference frame.

This decoder can comprise holding the thesaurus for the current list of the reference frame of each intermediate frame.

This decoder can comprise the decoded picture buffering device, its identification is used for the reference frame of decoding intermediate frame, wherein said decoding device in the situation of the LOF that receives, can operate to use on the current list of frame of a reception remove to identify reference frame at least one intermediate frame subsequently of decoding.

The method of this coding can comprise the frame number that transmits this frame of identification with each intermediate frame.

This encoder can comprise that described frame number is transmitted with each frame for the device of identifying each frame by frame number.

The computer program that should be used for coding can be arranged to the frame number that transmits this frame of identification with each intermediate frame when being performed.

This decoder method can comprise the frame number that receives this frame of identification with each intermediate frame, and keeps the mapping between described frame number and reference listing.

This decoder can comprise for maintaining the frame number that receives with each frame and the device of the mapping between the checked inner index number of video data.

In order to understand better the present invention and to show how the present invention is implemented, referring now to following accompanying drawing.

Description of drawings

Fig. 1 is the schematic diagram that is shown in two user terminals of communicating by letter in the communication system;

Fig. 2 A is the schematic block diagram of encoder;

Fig. 2 B is the schematic block diagram of decoder;

Fig. 3 a-3e illustrates a kind of example scenario of the grouping that is dropped; With

Fig. 4 a-4e illustrates the another kind of example scenario of the grouping that is dropped.

Embodiment

Fig. 1 illustrates first user terminal UE 1 with the form of illustrating and is connected to packet-based communication system 2, such as internet or other packet-based network.The present invention is useful in the context based on the communication system of VoIP, is such as Skype based on the communication system of VoIP ^TM, wherein video data is transmitted in the communication event that also can carry calling.

The second user terminal UE2 also is connected to network 2.Supposition user terminal UE1 is just serving as the source of the video data that consumes for received terminal UE 2 in Fig. 1.User terminal can be the form of the suitable equipment in any source that can serve as video data, is mobile or very.

In a nonrestrictive embodiment, the first and second user terminals have all been installed communication client, communication client is carried out and is set up the function of communication event by network 2, and provides encoder to be used for passing through the video flowing of network 2 transmission to be respectively applied to Code And Decode in the communication event of being set up by communication client.

Video data adopts the form of bit stream 20, the series of frames that the form that bit stream 20 comprises dividing into groups transmits.Described frame comprises (I) frame in interframe (P) frame and the frame.As mentioned, inter-frame comprises the data that represent the difference between this frame and the one or more reference frame.Frame (key frame) is the frame of the difference between the interior pixel of representative frame in the frame, and therefore it can be decoded with reference to other frame.When coding, frame can be marked as short-term with reference to (STR) or long term reference (LTR), as what determined by encoder.

Decoder on the receiving terminal need to be stored STR and LTR in order to use during decoding, guarantee that simultaneously LTR is not by overwrite unexpectedly.

Fig. 2 A is the schematic diagram of the operation on the encoder that uses in the user terminal of the type discussed in the above.Encoder 4 has processor 6 and memory 8.Encoder receive to comprise macro block a succession of frame form video data 1(for example, come the video camera that operates on the comfortable user terminal), processor with described macroblock coding framing with by network 2 transmission.The encoder operation compression algorithm generates the series of frames for transmission, comprises P frame and I frame.Each frame is associated with frame number.Encoder is preserved reference listing 10 in memory 8.Reference frame lists 10 comprises short-term (STR) and long-term (LTR) reference frame.In standard H.264, stipulated maximum 16.Reference listing on encoder is by using storage management control operation (MMCO) order to be managed.Following table 1 is the tabulation of MMCO order, comprises six different MMCO orders and stops sign.

For each I frame, reference listing is orderly group for the reference frame of this frame of coding.

Table 1

0	Stop sign, last in the MMCO tabulation
		1	From reference listing, remove a short-term reference frame (being defined as the difference with the present frame numbering)
2	From reference listing, remove a LTR frame
		3	A short-term reference frame (being defined as the difference with the present frame numbering) is labeled as the LTR frame
4	The maximum quantity of regulation LTR frame.Yet these buffers not yet are filled.
		5	Remove all reference frames
6	Be LTR-X with current frame flag

Such as what obviously find out from top table 1, storage management control operation order allows short-term with reference to being inserted into (MMCO-3) reference listing and removing (MMCO-1) from reference listing.In addition, long term reference frame can be inserted into (MMCO-6) reference listing and remove (MMCO-2) from reference listing.LTR is assigned with specific station location marker, for example LTR-0, LTR-1.

Reference listing can be eliminated by MMCO-5 or the mechanism by instantaneous decoder refresh (IDR) frame.Such frame is removed the content of reference frame lists immediately.Sign (Long_Term_Reference_Flag(long term reference sign)) whether assigned I DR frame should be marked as long term reference frame.LTR is different from the STR frame, because the STR frame can pass through sliding window process (back description) by overwrite in buffer, however the maintenance of LTR frame, until it is removed clearly.

Fig. 2 illustrates the output of encoder with the form of a series of groupings, and each grouping represents a frame.For following discussion, suppose that the frame of N series is at first encoded (having shown N-1 and N wherein in Fig. 2 A), follow thereafter the frame (having shown K-1 and K wherein) of K series.Frame N is marked as the long term reference for N series, and frame K is marked as long term reference subsequently.Generated and unmarked frame for " be not used in reference to " is assumed to be and serves as short-term reference frame by encoder.Described frame is sent to decoder, and this decoder comprises decoded picture buffering device DPB.Long term reference frame can be placed on based on its station location marker among the first buffer positions LTR-0 or the second buffer positions LTR-1.A frame can not be present in two buffers simultaneously.

In existing system, MMCO order is sent out with their the P frame that is associated, if like this so that the P LOF, the MMCO that then is associated orders and also loses.Though frame itself can be for example by not belonging to the application's scope (concealment) technology of hiding that is known in the art be resumed, can be so that there be undefined (undefined) situation in losing of MMCO order and therefore causes the failure of decoder for decoder.

According to embodiments of the invention, video flowing 20 comprises reference listing.Each intermediate frame (I-frame) is sent out with encode its current list 10 of reference frame of frame number and being used for.Prefix N, K of being associated with each frame etc. are carried in tabulation 10.

Encoder generates the tabulation of the reference frame that is used by present frame.In addition, it also reports the frame number of present frame.This is so that frame number can be used identical frame index with reference listing.Frame number and reference listing all are passed to decoder, as the supplementary for each frame.Decoder receives the frame number for each frame, and can therefore be created in the mapping between frame number and the frame interior index.

Should be pointed out that aspect this that H264 standard provides parameter f rame_num(frame number), it is the frame interior index in the bit stream.Yet existing encoder only can determine to assign a small amount of bit for it, for example to 16, like this so that it will circulate very fast.Because it is many that long term reference frame can keep for a long time in DPB, so this index number is for being inadequate for the purpose of mapping reference frame in the buffer.

Therefore in addition, frame_num is reset at key frame, and using may have ambiguity from the frame_num in the feedback information of receiver, in the situation of especially very long at feedback delay and shake.

Importantly, the index that is used in frame number and reference listing must be identical, since therefore encoder generating reference tabulation, it also should number to identify frame by delta frame, so that can keep between the content of reference listing and buffer synchronously.

Fig. 2 B is the schematic block diagram of the function on the diagram decoder.Decoder for example can be positioned on the second user terminal UE2, and is arranged to receive the video flowing 20 that is transmitted from user terminal UE1.To recognize easily that two user terminals can have encoder.

Decoder comprises decoded picture buffering device DPB40 and decoding function 42, and decoding function 42 operation comes the frame of decoding and receiving based on the content of decoded picture buffering device 40 in video flowing 20, as below in greater detail.The content of the receiver stage 44 control video flowings of decoder is come the frame that is provided for decoding for decoder stage 42, and is provided for the decoded picture buffering device is remained up-to-date MMCO order, again as being described in greater detail below.In addition, according to embodiments of the invention, receiver stage 44 holds the current list 10 for the current frame that receives in memory 46.

Fig. 3 a illustrates typical sight in the decoding side to 3e, and wherein decoder is receiving a series of frame that the encoder by Fig. 2 sends.In each level of decode procedure, the left-hand side has shown the grouping that enters and the state of decoding front decoding buffer DPB.Show decoded stream of packets at right-hand side, together with the state of the decoding buffer DPB after the decoder stage.The decoding buffer namely operates on the basis of first in first out according to the sliding window process operation.Yet, be labeled as the frame of long term reference and without undergoing the sliding window process and be retained in the buffer.

According to Fig. 3 a, grouping N arrives, and has enclosed LT REF_UPDATE 0 order.This frame is placed in the buffer, and because groove (slot) free time is arranged, so frame N-1 is retained, and this long term reference frame N also is placed in the buffer, at position LTR0 place.

Fig. 3 b shows the arrival of grouping K-2, and it does not enclose the MMCO order.Before reception, buffer comprises former frame K-3 and long term reference frame N.The frame K-2 that enters releases frame K-3, but long term reference frame N is retained.The maximum quantity of " dead slot " (that is, the size of buffer) is determined by parameter (for example, the max_num_ref_frame in the H264 standard).In the example in front, buffer sizes is configured to 2.

In Fig. 3 c, similarly process is applied to frame K-1 subsequently.In Fig. 3 d, the next frame that is transmitted by encoder is frame K, but Fig. 3 d illustrates the situation that this frame is dropped during the transmission.In this situation, frame K has enclosed LT REF_UPDATE 0 order, and its plan allows frame K replace frame N as the next long term reference at the LTR0 place.Decoder recognizes that frame K is dropped, and the attempt regenerate it with hiding process, be marked as K(Con in order to provide) frame.Yet it does not also know losing of MMCO order, does not therefore replace this long term reference frame N.The version of dotted line illustrates the thing that the decoding buffer should be held now, and the solid line version illustrates the thing that in fact it hold.

One receives next frame K+1, and this frame is just expected the reference frame as it with frame K according to the frame reference listing of setting up at encoder, and it expects that frame K is housed inside the LTR0 place now.In fact, be N at this frame that holds with reference to the place, so decoder will be undefined and failed, perhaps decoded frame K+1 improperly.And, because in the decoding buffer, do not have anything to go to hold hiding frame K, when finishing, the decoder stage that the frame K+1 that enters shows fully replaces it in Fig. 3 e.

In an embodiment of the present invention, this problem is overcome by be transmitted in the current reference frame lists 10 of setting up on the encoder with each frame.Therefore in the situation of disappearance frame (K among Fig. 3 d), decoder can recognize that frame just lacks, and generates the version of hiding in known manner.The more important thing is that encoder should be guaranteed when this frame has been pushed out buffer not with reference to this frame.

Fig. 4 a illustrates the other exemplary sight of the impact of lost packets to 4e.In this situation, the packet frames sequence that is produced by encoder is P0, P1, P2 etc., and wherein each grouping represents the frame of reference numeral.In the decoder stage of Fig. 4 a representative, the frame P1 that enters is moved in the decoded picture buffering device, and the frame of front is pushed downwards one in buffer.

Next frame P2 has MMCO order LT REF_UPDATE 0, if be received, this order general so that described frame be stored in the last remaining empty position of buffer, as shown at the right-hand side of Fig. 4 b.That is, according to the H264 standard, LTR is stored in ending place of reference listing, but other realization is possible.Yet, if grouping does not receive, be undefined in decoded decoder stage, until that it becomes is settled.

In a realization of encoder, the impact of decode procedure is as shown in the dotted line of the left-hand side of Fig. 4 c.That is, the version of hiding of frame P2 is generated by decoder, and its basis at sliding window is placed on the top of buffer.When next frame P3 was received, the frame that is denoted as P3* was generated, and its version of hiding that will use frame 2 is as the short-term reference, and did not know that frame P2 should be long term reference.

Here it is, and what transmit is orderly why favourable reason with External Reference tabulation.In the macro block of coding itself, reference frame is only identified by their positions in described tabulation, and the reference of identifying ambiguously them is STR or LTR.

In this example, owing to lose, reference frame P2 and P1 have the position of exchange, and reference key will point to wrong frame.

And, when next frame P4 is received (it comprises the renewal reference command by chance in this situation), because the long-term position LT0 that has not distributed in buffer, so buffer is full now, therefore (in the H264 standard) decode procedure is undefined and failed at that point.This is illustrated by the question mark in the dotted line version of the dexter buffer of Fig. 4 d.

In an embodiment of the present invention, this problem can be solved by be transmitted in the present frame reference listing that generates on the encoder with each frame.Then this operates rightly existing LTR groove is substituted into P4 from P2 allowing to have the frame P4 subsequently that upgrades reference command.In this case, the position that relies on the frame of disappearance to occupy in reference listing will be known that it is contemplated to where to be positioned at.This position is provided by the reference listing that transmits.Yet if there is not idle frame groove in buffer, decoder removes the oldest STR from buffer.If there is not STR, then it removes the oldest LTR.

If have the frame P2 of MMCO update command be received, but the frame P4 with MMCO update command is not received, and then causes different problems.In this case, when frame P4 failed to come true (materialise), buffer had the scene in the solid line of left-hand side of Fig. 4 d.In this case, the version of hiding of P4 is generated P4(Con), and be placed on and replace P3 in the buffer, P3 replaces P1 on the basis of sliding window.

When the frame P5 when subsequently was received, picture buffer was full, and the long-term position LTR1 that has not distributed.In order to create this position, the MMCO that invests frame P5 has the order that removes short-term frame_num1 (P1).This frame is because for the applied sliding windows of the frame 4 of losing recover not exist, so the decoder failure.

In an embodiment of the present invention, this problem can be solved by be transmitted in the present frame reference listing that generates on the encoder with each frame.Therefore, in this case, rely on the position of frame P4 in the reference listing that transmits of disappearance, will know that it is (intended) P1 of expection.Therefore identical P5 can be decoded based on the version of hiding of P4, and then will correctly be in the LTR1 place in the buffer, is used for the decoding of back.

Therefore, have in video flowing in the situation of LOF, the reference listing of transmission can be by decoding function 42 access.LOF can need not to use reference listing and be detected, and for example in the H264 standard, the frame_num syntactic element is transmitted in the H264 bit stream, and therefore can be detected by the gap in the sequence of frame_num.

When losing when being detected, reference listing is used by decoder and solves owing to lose the undefined decoder situation (for example, as describing in front) that occurs, so that the behavior of the decoder of improvement during loss situation.For example, in Fig. 4 C, the order of the tabulation of the frame among the DPB may have ambiguity owing to losing, but the reference that externally transmits mapping (can access it from memory 46 in this case) will alleviate this problem.

Reference listing 10 can be generated at encoder during cataloged procedure discussed above.Alternatively, it can be generated by the separate modular of encoder bit stream outer, that transmit coding.

WhenWhen early stage system compared, described embodiments of the invention provided improved robustness.The communication enabled of the tabulation of the reference frame from the encoder to the decoder is the long-term recovery logic on reference frame management and the Erasure channel flexibly.In any case, in the codec when bottom was not context when ideally being designed to Erasure channel, it was useful especially.

Claims

1. method that transmits video data comprises:

Be a plurality of frames with video data encoding on encoder, described a plurality of frames comprise intermediate frame, and each in the described intermediate frame is encoded based at least one reference frame, and in them at least some are encoded based on a plurality of reference frames;

On encoder, preserve the current list of reference frame for each intermediate frame; With

Transmit described a plurality of intermediate frame, each intermediate frame is transmitted explicitly with the current list for the reference frame of this frame.

2. encoder comprises:

Being used for video data encoding is the device of a plurality of frames, and described a plurality of frames comprise intermediate frame, and each in the described intermediate frame is encoded based at least one reference frame, and in them at least some are encoded based on a plurality of reference frames;

Be used for preserving for each intermediate frame the device of the current list of reference frame; With

Be used for transmitting the device of described a plurality of intermediate frames, each intermediate frame is transmitted explicitly with the current list for the reference frame of this frame.

3. according to the method for claim 1, or according to the encoder of claim 2, wherein at least one key frame is generated and transmits as the version of the compression of source frame of video, and described key frame consists of reference frame.

4. according to claim 1,2 or 3 method or encoder, wherein current intermediate frame is based on the reference frame of the front in (ⅰ) a series of frame or (ⅱ) reference frame subsequently in a series of frame and being encoded.

5. according to method or the encoder of claim 1 or 2, wherein each intermediate frame is by using the prediction interframe encode based on described at least one reference frame to be generated.

6. according to method or the encoder of arbitrary aforementioned claim, comprise: at least one in the described reference frame in will tabulating on encoder is labeled as long term reference frame, indicating thus described reference frame to be stored until experience update command, wherein is that long term reference frame comprises that identification is used for the buffer positions of described long term reference frame with frame flag; And/or in the described reference frame in will tabulating at least one be labeled as short-term reference frame, indicates thus described reference frame not experiencing in the situation of update command by overwrite.

7. according to method or the encoder of claim 6, wherein the step of mark comprises: the storage management order is appended to the frame that is labeled, and the state of this frame is indicated in described storage management order, and described order is transmitted with this frame.

8. according to the method for claim 1 or 2, wherein at least one intermediate frame and/or at least one key frame are identified in the tabulation of reference frame.

9. according to method or the encoder of arbitrary aforementioned claim, wherein said tabulation comprises orderly group of reference frame, and each reference frame has the position in described orderly group.

10. computer program, it comprises program code devices, implements following steps when described program code devices is carried out by processor:

Be a plurality of frames with video data encoding, described a plurality of frames comprise intermediate frame, and each in the described intermediate frame is encoded based at least one reference frame, and in them at least some are encoded based on a plurality of reference frames;

Preserve the current list of reference frame for each intermediate frame; With