WO2021262388A1

WO2021262388A1 - Video encoding

Info

Publication number: WO2021262388A1
Application number: PCT/US2021/034668
Authority: WO
Inventors: Bin Li; Tom W. Holcomb; Mei-Hsuan Lu; Yan Lu; Ming-Chieh Lee
Original assignee: Microsoft Technology Licensing, Llc
Priority date: 2020-06-22
Filing date: 2021-05-28
Publication date: 2021-12-30
Also published as: CN113905241A

Abstract

In accordance with implementations of the subject matter as described herein, there is provided a solution for video encoding. According to the solution, control information from a decoder is received at an encoder, the control information indicating that the decoder fails to receive a first frame in a video stream transmitted from the encoder. At least one long-term reference frame (LTR) for encoding a second frame not yet encoded after the first frame in the video stream is determined from long-term reference frames stored in a reference frame buffer, where the control information indicates that the at least one long-term reference frame is received by the decoder. Then, the second frame is encoded based on the at least one long-term reference frame. The solution enables codecs that do not support LTR syntax to perform error recovery by using LTR(s) in the event of frame loss, thereby improving decoding robustness.

Description

VIDEO ENCODING

BACKGROUND

[0001] Long-Term Reference Frame (LTR) is an important tool in video encoding for improving encoding efficiency and decoding robustness when frame loss occurs. In real time video transmission, LTR can be used for error recovery. However, some video codecs have limited support for LTR.

SUMMARY

[0002] In accordance with implementations of the subj ect matter as described herein, there is provided a solution for video encoding. According to the solution, control information from a decoder is received at an encoder, the control information indicating that the decoder fails to receive a first frame in a video stream transmitted from the encoder. At least one long-term reference frame (LTR) for encoding a second frame not yet encoded after the first frame in the video stream is determined from long-term reference frames stored in a reference frame buffer, where the control information indicates that the at least one long term reference frame is received by the decoder. Then, the second frame is encoded based on the at least one long-term reference frame. The solution enables codecs that do not support LTR syntax to perform error recovery by using LTR(s) in the event of frame loss, thereby improving decoding robustness.

[0003] The Summary is to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the subject matter as described herein, nor is it intended to be used to limit the scope of the subject matter as described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Fig. 1 A is a schematic diagram illustrating an example reference frame structure in real-time video transmission;

[0005] Fig. IB is a schematic diagram illustrating a further example reference frame structure in real-time video transmission;

[0006] Fig. 2 is a block diagram illustrating an example system for real-time video transmission in accordance with implementations of the subject matter as described herein; [0007] Fig. 3 is a flowchart illustrating an example method for video encoding in accordance with implementations of the subject matter as described herein; and [0008] Fig. 4 is a block diagram of an electronic device that can implement a plurality of implementations of the subject matter as described herein.

[0009] Throughout the drawings, the same or similar reference symbols refer to the same or similar elements.

DETAILED DESCRIPTION OF EMBODIMENTS

[0010] The subject matter as described herein will now be described with reference to several example implementations. It would be appreciated that description of those implementations is merely for the purpose of enabling those skilled in the art to better understand and further implement the subject matter as described herein, without suggesting any limitation to the scope disclosed herein.

[0011] As used herein, the term “includes” and its variants are to be read as open-ended terms that mean “includes, but is not limited to.” The term “based on” is to be read as “based at least in part on.” The terms “an implementation” and “one implementation” are to be read as “at least one implementation.” The term “another implementation” is to be read as “at least one other implementation.” The term “first,” “second” or the like may represent different or the same objects. Other definitions, either explicit or implicit, may be included below.

[0012] Fig. 1 A is a schematic diagram illustrating an example reference frame structure 100 in real-time video transmission. Fig. 1A shows consecutive frames 110, 120, 130 and 140 in a video stream, where the frame 110 acts as a reference frame of the frame 120, the frame 120 acts as a reference frame of the frame 130, and the frame 130 acts as a reference frame of the frame 140. The reference frames of the frame 110 may be one or more frames preceding the frame 110 (not shown in Fig. 1A). In real-time video transmission, for example, an encoder may encode the frames 110, 120, 130 and 140 according to the reference frame structure 100 as shown in Fig. 1A. The encoded video stream may be transmitted to a decoder. Accordingly, the decoder decodes the frames 110, 120, 130 and 140 according to the reference frame structure 100 as shown in Fig. 1A. In the context, the reference frame that is served by a neighboring frame of the current frame, as shown in Fig. 1A, is also referred to as a “Short-Term Reference Frame (STR).” STR is typically reserved shortly in the codec. For example, when decoding a next frame of the current frame, the reference frame used by the codec will be updated accordingly.

[0013] In the case where no frame loss occurs during video streaming, the reference frame structure 100 as shown in Fig. 1 A can achieve a relatively high video encoding performance. However, in the case where frame loss occurs during video stream transmission, if the codec still follows such reference frame structure, the decoder will not be able to perform decoding normally. For example, if the frame 130 is not received by the decoder, the decoder cannot predict the frame 140 based on the frame 130. At this time, for example, the decoder may require the encoder to retransmit the frame 130, which will introduce extra delay. For another example, the encoder may perform intra-frame prediction encoding on the frame 140, which may, however, reduce the encoding efficiency of the frame 140. In order to prevent introduction of extra delay or obvious reduction of the encoding efficiency, the encoder may encode the frame 140 using an LTR as the reference frame of the frame 140. Contrary to the STR, an LTR refers to a reference frame reserved relatively long. An LTR may be a specific frame that appears before the current frame, not necessarily the immediately previous frame of the current frame.

[0014] Fig. IB is a schematic diagram illustrating a further example reference frame structure 105 in real-time video transmission. As shown in Fig. IB, for example, the frame 130 is missing during video transmission. In order to ensure that the decoder can decode the subsequent frame 140 of the frame 130 correctly, the encoder may encode the frame 140 using an LTR 150. The LTR 150 may be a specific frame that is received by the decoder. In this way, the decoder may predict the frame 140 using the LTR 150.

[0015] As described above, in real-time video transmission, LTR can be used for error recovery to improve decoding robustness in the event of frame loss. However, some video encoders have limited support for LTR. For example, the syntax of the AVI codec does not support the function of LTR.

[0016] In accordance with implementations of the subj ect matter as described herein, there is provided a solution for video encoding. According to the solution, control information from a decoder is received at an encoder, the control information indicating that the decoder fails to receive a first frame in a video stream transmitted from the encoder. At least one long-term reference frame (LTR) for encoding a second frame not yet encoded after the first frame in the video stream is determined from long-term reference frames stored in a reference frame buffer, where the control information indicates that the at least one long term reference frame is received by the decoder. Then, the second frame is encoded based on the at least one long-term reference frame. The solution enables codecs that do not support LTR syntax to perform error recovery by using LTR(s) in the event of frame loss, thereby improving decoding robustness.

[0017] In the following, an AVI codec will be used as an example to describe in detail various example implementations of the solution. It would be appreciated that this is provided merely for illustration, without suggesting any limitation as to the scope of the subject matter as described herein. In addition to the AVI codec, the implementations of the subject matter as described herein are also applicable to other codecs. [0018] Fig. 2 is a block diagram of an example system 200 for real-time video transmission in accordance with implementations of the subject matter as described herein. As shown in Fig. 2, the system 200 includes a video encoding device 210 and a video decoding device 220

[0019] In some implementations, the video encoding device 210 may include an encoding controller 211 and a video encoder 212, and the video decoding device 220 may include a decoding controller 221 and a video decoder 222. The encoding controller 211 may receive control information from the decoding controller 221 and generate, based on the received control information, respective control commands for controlling operations of the video encoder 212. The encoding controller 221 may send the respective generated control commands to the video encoder 212 and receive the encoded video stream from the video encoder 212. The encoding controller 211 may transmit the encoded video stream to the decoding controller 221. The decoding controller 221 may forward the received video stream to the video decoder 222 for decoding. In addition, the decoding controller 221 may generate, based on the received video stream, control information to be transmitted to the encoding controller 211. The control information transmitted from the decoding controller 221 to the encoding controller 211 may indicate reception states of frames in the video stream, for example, received successfully or lost.

[0020] In some implementations, the commands sent from the encoding controller 211 to the video encoder 212 may include, but are not limited to: encoding a current frame, marking the current frame as an LTR, using the LTR as a reference frame to encode the current frame, invalidating an STR, invalidating an unused LTR, or the like. The video encoder 212 may perform respective operations responsive to the received commands. In the following description, an AVI encoder will be taken as an example of the video encoder 212.

[0021] Typically, there is a reference frame buffer at the AVI encoder, which can store a predetermined number of reference frames. For example, in the legacy solution, the predetermined number is 8. Typically, the reference frames stored in the reference buffer are all STRs. That is, as the encoding proceeds, the reference frames stored in the reference frame buffer will be updated accordingly. The AVI encoder may send the encoded video stream to the AVI decoder, including information of the used reference frames (for example, preceding reference frames based on which each frame is encoded). A reference frame buffer of the same size is also maintained at the AVI decoder. The AVI decoder may decode the video stream based on the information of the reference frames included in the received video stream, and store the decoded reference frames in the reference frame buffer, for use in decoding subsequent frames. As the decoding proceeds, the reference frames stored in the reference frame buffer will be updated.

[0022] In some implementations, in order to enable the AVI codec to support the LTR, the reference frame buffer at the AVI encoder may be divided into two portions, namely one portion for storing STRs and the other portion for storing LTRs. Suppose that the reference frame buffer stores at most 8 reference frames with indices 0-8. In the reference frame buffer, for example, the portion for storing STRs may store at most 3 STRs with indices 0- 2; and the portion for storing LTRs may store at most 5 LTRs with indices 3-7. It would be appreciated that the number of the reference frames is provided merely as an example, and the respective numbers of STRs and LTRs stored in a reference frame buffer in different implementations may be different than those in the above example. The portion for storing STRs in a reference frame buffer may be updated along with the progress of encoding according to the legacy solution. The storage, use and invalidation of the LTR will be further illustrated in detail below.

[0023] In some implementations, the encoding controller 211 may send a command for marking the current frame as an LTR to the video encoder 212. The command, for example, may indicate an index LTR index of the current frame that will be used as an LTR in the reference frame buffer. The index LTR index may be an absolute index in the reference frame buffer, namely LTR index e [0, 7], or may be a relative index in the LTR portion, for example, LTR index e [0, 4] Responsive to the command, the video encoder 212 may modify a syntax element refresh frame flags of the AVI encoder to identify the current frame as an LTR. For example, if LTR index is an absolute index in the reference frame buffer, refresh frame flags may be modified as follows: refresh frame flags |= (1 « LTR index). If LTR index is a relative index in the LTR portion of the reference frame buffer, refresh frame flags may be modified as follows; refresh frame flags |= (1 « (LTR index + N)), where N represents the number of STRs that the reference frame buffer can store therein, for example, N=3. By setting the syntax element refresh frame flags, the current frame can be stored as an LTR into the reference frame buffer.

[0024] In some implementations, the encoding controller 211 may send to the video encoder 212 a command for using at least one LTR as the reference frame of the current frame. The command, for example, may indicate an index of the at least one LTR in the reference frame buffer. In some implementations, when at least one LTR is used for error recovery, the video encoder 212 may set the syntax element error resilient mode in the AVI encoder to 1 to improve decoding robustness. [0025] In some implementations, the command sent from the encoding controller 211 to the video encoder 212 may indicate that an LTR is to be used as the reference frame of the current frame, and the absolute index of the LTR in the reference frame buffer is LTR index, for example, LTR index e [0, 7] Responsive to the command, the video encoder 212 may set each element in the syntax element ref_frame_idx[i] (for example, i e [0, 7]) of the AVI encoder as LTR index. Alternatively, the command sent from the encoding controller 211 to the video encoder 212 may indicate that an LTR is to be used as the reference frame of the current frame, and the relative index of the LTR in the LTR portion is LTR index, for example, LTR index e [0, 4] Responsive to the command, the video encoder 212 may set each element in the syntax element ref_frame_idx[i] (for example, i e [0, 7]) of the AVI encoder to LTR_index+ N, where N represents the number of STRs that the reference frame buffer can store therein, for example, N=3.

[0026] In some other implementations, the command sent from the encoding controller 211 to the video encoder 212 may indicate that a plurality of LTRs are to be used as the reference frames of the current frame, and may indicate indices of these LTRs. For example, suppose that the command indicates two LTRs will be used as the reference frames of the current frame, and the absolute indices of the two LTRs in the reference frame buffer are LTR indexO and LTR indexl, respectively. For example, LTR indexO e [0, 7] and LTR indexl e [0, 7] Responsive to the command, the video encoder 212 may set each element in the syntax element ref_frame_idx[i] (for example, i e [0, 7]) of the AVI encoder to LTR indexO or LTR indexl. Alternatively, suppose that the command indicates that the two LTRs will be used as the reference frames of the current frame, and the relative indices of the two LTRs in the LTR portion are LTR indexO and LTR indexl, respectively. For example, LTR indexO e [0, 4] and LTR indexl e [0, 4] Responsive to the command, the video encoder 212 may set each element in the syntax element ref_frame_idx[i] (for example, i e [0, 7]) of the AVI encoder to LTR_index0+N or LTR_indexl+N, where N represents the number of STRs that the reference frame buffer can store therein, for example, N=3.

[0027] In some implementations, the encoding controller 211 may send a command for encoding the current frame to the video encoder 212. Responsive to the command, the video encoder 212 may encode the current frame using a reference frame indicated by the syntax element ref_frame_idx[i] (for example, i e [0, 7]).

[0028] The syntax element ref_frame_idx[i] is used to set the reference frame to indicate the index of the reference frame in the reference frame buffer for the video encoder 212 to encode the current frame, which is also referred to herein as a “first syntax element”. The syntax element refresh frame flags is used to update the reference frame buffer to store the reference frame in the reference frame buffer or to invalidate the reference frame originally stored in the reference frame buffer, which is also referred to herein as a “second syntax element”.

[0029] In some implementations, for example, when an LTR is used as the reference frame, the encoder 210 may invalidate unreliable reference frames stored in the reference frame buffer to ensure that subsequent frames of the current frame can be decoded normally by the decoder 220.

[0030] In some implementation, for example, when an LTR is used as the reference frame, the encoding controller 211 may send to the video encoder 212 a command for invalidating an STR in the reference frame buffer. The command, for example, may indicate the index STR index of the STR in the reference frame buffer, where STR index e [0, N) and N represents the number of STRs that the reference frame buffer can store therein, for example, N=3. Responsive to the command, the video encoder 212 may modify the syntax element refresh frame flags of the AVI encoder, to replace the STR in the reference frame buffer with the current frame and thus invalidate the STR. F or example, refresh frame flags may be modified as follows: refresh frame flags |= (1 « STR index). In some implementations, for example, when an LTR is used as the reference frame, the encoding controller 211 may send to the video encoder 212 a command for invalidating all of STRs in the reference frame buffer. Responsive to the command, the video encoder 212 may modify the syntax element refresh frame flags of the AVI encoder, to replace all of the STRs in the reference frame buffer with the current frame and thus invalidate all of the STRs. [0031] In some implementations, for example, when an LTR is used as a reference frame, the encoding controller 211 may send to the video encoder 212 a command for invalidating an unused LTR in the reference frame buffer. The command, for example, may indicate a decisive index LTR index of the LTR in the reference frame buffer, where LTR index e [N, 8) and N represents the number of STRs that the reference frame buffer can store therein, for example, N=3. Responsive to the command, the video encoder 212 may modify the syntax element refresh frame flags of the AVI encoder, to replace the LTR with the current frame in the reference frame buffer and thus invalidate the LTR. For example, refresh frame flags may be modified as follows: refresh frame flags |= (1 « LTR index). In some implementations, for example, when an LTR is used as a reference frame, the encoding controller 211 may send to the video encoder 212 a command for invalidating all of unused LTRs in the reference frame buffer. Responsive to the command, the video encoder 212 may modify the syntax element refresh frame flags of the AVI encoder to replace all of the unused LTRs in the reference frame buffer with the current frame and thus invalidate these LTRs.

[0032] Although the encoding controller 211 and the video encoder 212 in Fig. 2 are shown as different modules in the video encoding device 210, it would be appreciated that this is provided merely for illustration, without suggesting any limitation as to the scope of the subject matter as described herein. In some implementations, the functionalities of the encoding controller 211 and the video encoder 212 may be integrated into a same module. Likewise, in some implementations, the functionalities of the decoding controller 221 and the video decoder 222 may be integrated into a same module. In the following, for further simplicity, the video encoding device 210 may also be referred to as “encoding device” or “encoder”, and the video encoding device 220 may also be referred to as “decoding device” or “decoder” below.

[0033] Fig. 3 is a flowchart of an example method 300 for video encoding in accordance with implementations of the subject matter as described herein. For example, the method 300 may be implemented by the encoder 210. It would be appreciated that the method 300 may include additional acts not shown and/or may omit some acts as shown, and the scope of the subject matter as described herein is not limited in the aspect.

[0034] As shown in Fig. 3, at block 310, the encoder 210 receives control information from the decoder 220, where the control information indicates that the decoder 220 fails to receive a first frame in a video stream transmitted from the encoder 210. At block 320, in response to the control information, the encoder 210 determines, from LTRs stored in the reference frame buffer, at least one LTR for encoding a second frame (for example, the current frame) not encoded yet after the first frame in the video stream, where the control information indicates that the at least one long-term reference frame is received by the decoder 220. Then, at block 330, the encoder 210 encodes the second frame based on the at least one LTR.

[0035] In some implementations, the reference frame buffer at the encoder 210 may be used to store at most a predetermined number of LTRs. For example, the predetermined number may be 5 or other numerical value. The encoder 210 may maintain a first array markedLTR[NUM] having a length equal to the predetermined number, where the NUM may be 5 or other numerical value, and the first array is provided for recording whether an LTR is stored at a respective position in the reference frame buffer. The encoder 210 may maintain a second array confirmedLTR[NUM] having a length equal to the predetermined number, to record whether the decoder 220 has received a respective LTR. Elements in the two arrays may be initialized as FALSE.

[0036] In some implementations, if the encoder 210 determines that the number of LTRs stored in the reference frame buffer does not exceed a predetermined number, i.e., there is an element of FALSE in the first array, the encoder 210 may mark a certain frame in the video stream as an LTR. In order to mark the frame as an LTR, the encoder 210 may determine an index to be allocated to the frame, which may indicate a position in the reference frame buffer where the frame is to be stored. Specifically, the encoder 210 may search the first array for a first element of FALSE and set the element as TRUE. The array index of the element may correspond to the index to be allocated to the frame. For example, the array index of the element may be a relative index LTR index to be allocated to the frame, for example, LTR index e [0, 4] In some implementations, as stated previously, the encoder 210 may set a syntax element refresh frame flags for refreshing the reference frame buffer in the AVI encoder by using the determined index LTR index, so as to identify the frame as an LTR. By setting the syntax element refresh frame flags, the frame can be stored as an LTR in the reference frame buffer.

[0037] In some implementations, if the control information from the decoder 220 indicates that a certain marked LTR is received by the decoder 220 successfully, the encoder 210 may set a respective element in the second array as TRUE, to indicate that the LTR is received successfully by the decoder 220. If the control information from the decoder 220 indicates that a certain marked LTR is not received by the decoder 220, the encoder 210 may set a respective element in the second array as FALSE, to indicate that the LTR is lost.

[0038] In some implementations, if the control information from the decoder 220 indicates that the decoder 220 fails to receive a first frame in the video stream transmitted from the encoder 210, regardless of whether the first frame is an LTR or not, the encoder 210 may perform error recovery using the LTRs stored in the reference frame buffer.

[0039] In some implementations, the encoder 210 may search the second array for at least one element of TRUE. The array index of the at least one element corresponds to an index of the at least one LTR in the reference frame buffer which is received by the decoder 220. The encoder 210 may determine the at least one LTR as a reference frame for encoding a second frame (for example, the current frame) after the first frame, and may encode the second frame based on the at least one LTR.

[0040] In some implementations, as described above, in order to encode the second frame based on the at least one LTR, the encoder 210 may set the syntax element ref_frame_idx[i] (for example, i e [0, 7]) in the AVI encoder with the index of the at least one LTR. The syntax element ref_frame_idx[i] is used for setting a reference frame in the AVI encoder. For example, the encoder 210 may set each element in the syntax element ref_frame_idx[i] as an index of one of the at least one LTR. As such, the encoder 210 can encode the second frame using the at least one LTR as the reference frame.

[0041] In some implementations, in the case where the at least one LTR is used as a reference frame for encoding the second frame, in order to ensure that a subsequent frame of the second frame can be decoded normally by the decoder, the encoder 210 may invalidate unreliable reference frames stored in the reference frame buffer.

[0042] In some implementations, in the case where the at least one LTR is used as the reference frame for encoding the second frame, the encoder 210 may invalidate an STR stored in the reference frame buffer. As described above, in order to invalidate the STR stored in the reference frame buffer, the encoder 210 may set the syntax element refresh frame flags for refreshing the reference frame buffer in the AVI encoder using the index of the STR in the reference frame buffer, so as to replace the STR stored in the reference frame buffer with the second frame.

[0043] Alternatively or in addition, in some implementations, in the case where the at least one LTR is used as a reference frame for encoding the second frame, the encoder 210 may invalidate other LTRs stored in the reference frame buffer. As described above, in order to invalidate the other LTRs stored in the reference frame buffer, the encoder 210 may set the syntax element refresh frame flags for refreshing the reference frame buffer in the AVI encoder using indices of the other LTRs in the reference frame buffer, so as to replace the other LTRs stored in the reference frame buffer with the second frame.

[0044] In this way, the AVI encoder can implement error recovery by using an LTR in the event of frame loss. Since the LTR used by the AVI encoder is the one acknowledged by the AVI decoder as being received, it is ensured that the AVI decoder can successfully decode a frame encoded based on the LTR, thereby improving decoding robustness in the event of frame loss. Since information of the reference frames (regardless of STRs or LTRs) used by the AVI encoder is sent to the decoder along with the encoded video stream, the decoder can obtain the information of the reference frames used by the AVI encoder from the received video stream and decode the video stream accordingly. That is, the AVI decoder can be implemented according to the legacy solution, without any changes.

[0045] Although an AVI codec is used as an example above to describe various implementations of the subject matter as described herein, it would be appreciated that this is provided merely for illustration, without suggesting any limitation to the scope of the subj ect matter as described herein. The implementations of the subj ect matter as described herein may also be applied to other codecs that do not support LTR syntax. As such, those codecs can perform error recovery by using LTR(s) in the event of frame loss, thereby improving decoding robustness.

[0046] Fig. 4 is a block diagram illustrating an electronic device 400 that can implement multiple implementations of the subject matter as described herein. It would be appreciated that the electronic device 400 as shown in Fig. 4 is provided merely as an example, which cannot constitute any limitation to the functionalities and scope of the subject matter as described herein. For example, the video encoding device 210 and/or video decoding device 220 as shown in Fig. 2 can be implemented by the electronic device 400. As shown, the electronic device 400 includes an electronic device 400 in the form of general computing device. The components of the electronic device 400 may include, but are not limited to, one or more processors or processing units 410, a memory 420, a storage device 430, one or more communication units 440, one or more input devices 450 and one or more output devices 460.

[0047] In some implementations, the computing device 400 can be implemented as various user terminals or service terminals. The service terminal may be any server, large-scale computing device, and the like, as provided by various service providers. The user terminal, for example, may be any type of mobile terminal, fixed terminal or portable terminal, such as mobile telephone, station, unit, device, multimedia computer, multimedia tablet, Internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio/video player, digital camera/video camera, positioning device, TV receiver, radio broadcast receiver, E- book device, gaming device or any combinations thereof, including accessories and peripherals of these devices or any combinations thereof. It would be anticipated that the electronic device 400 can support any type of interface for a user (such as a “wearable” circuit).

[0048] The processing unit 410 may be any physical or virtual processor and can perform various processing based on programs stored in the memory 420. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capacity of the electronic device 400. The processing unit 410 can also be referred to as a central processing unit (CPU), microprocessor, controller or microcontroller.

[0049] The electronic device 400 typically includes a plurality of computer storage media, which may be any available media accessible by the electronic device 400, including, but not limited to, volatile and non-volatile media, and removable and non-removable media. The memory 420 may be a volatile memory (for example, a register, cache, Random Access Memory (RAM)), non-volatile memory (for example, a Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory), or any combination thereof.

[0050] The storage device 430 may be any removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, disk or any other medium, which can be used for storing information and/or data and accessed in the electronic device 400. The computing electronic device 400 may further include additional removable/non removable, volatile/non-volatile memory media. Although not shown in Fig. 4, a disk drive may be provided for reading or writing a removable and non-volatile disk and a disc drive may be provided for reading or writing a removable non-volatile disc. In these cases, each drive may be connected to the bus (not shown) via one or more data medium interfaces. [0051] The communication unit 440 communicates with a further computing device via communication media. In addition, functionalities of components in the electronic device 400 can be implemented by a single computing cluster or multiple computing machines connected communicatively for communication. Therefore, the electronic device 400 can be operated in a networking environment using a logical link with one or more other servers, network personal computers (PCs) or another general network node.

[0052] The input device 450 may include one or more input devices, such as a mouse, keyboard, tracking ball, voice-input device, and the like. The output device 460 may include one or more output devices, such as a display, loudspeaker, printer and the like. As required, the electronic device 400 can also communicate via the communication unit 440 with one or more external devices (not shown) such as a storage device, display device and the like, one or more devices that enable users to interact with the electronic device 400, or any devices that enable the electronic device 400 to communicate with one or more other computing devices (for example, a network card, modem, and the like). Such communication can be performed via an input/output (I/O) interface (not shown).

[0053] In some implementations, besides integrated on a single device, some or all components in the electronic device 400 can be provided in the form of cloud computing architecture. In the cloud computing architecture, these components can be arranged remotely, and can operate together to implement the functionalities described herein. In some implementations, the cloud computing provides computing, software, data access and storage services, without a necessity of letting terminal users know the physical location or configuration of the system or hardware that provides these services. In various implementations, the cloud computing provides services via a wide area network (for example, Internet) using an appropriate protocol. For example, a cloud computing provider provides applications via a wide area network, and they are accessible via a web browser or any other computing component. Software or components and respective data of the cloud computing architecture can be stored on a server at a remote location. Computing resources in a cloud computing environment can be merged at a remote data center location or can be dispersed. Cloud computing infrastructures can provide services through a shared data center even though they are presented as a single access point for users. Hence, the cloud computing architecture can be used to provide components and functionalities as described herein from the service provider at the remote location. Alternatively, they may be provided from regular servers, or they may be mounted on a client device directly or in other manners.

[0054] The electronic device 400 may be used to implement video encoding in multiple implementations of the subject matter as described herein. The memory 420 may include a video encoding module 422 having one or more program instructions, which can be accessed and operated by the processing unit 410 to implement functionalities of various implementations as described herein.

[0055] Some example implementations of the subject matter described herein are listed below.

[0056] In a first aspect, the subject matter as described herein provides a computer- implemented method. The method comprises: receiving, at an encoder, control information from a decoder, the control information indicating that the decoder fails to receive a first frame in a video stream transmitted from the encoder; determining, from long term reference frames stored in a reference frame buffer, at least one long-term reference frame for encoding a second frame not yet encoded after the first frame in the video stream, the control information indicating that the at least one long-term reference frame is received by the decoder; and encoding the second frame based on the at least one long-term reference frame.

[0057] In some implementations, the encoder comprises an AVI encoder, and encoding the second frame based on the at least one long-term reference frame comprises: determining an index of the at least one long-term reference frame in the reference frame buffer; and performing AVI encoding on the second frame based on the at least one long term frame by setting a first syntax element with the index, the first syntax element being used to set a reference frame in the AVI encoder.

[0058] In some implementations, the first syntax element is used to set a plurality of reference frames for encoding the second frame, and setting the first syntax element with the index comprises: setting the first syntax element with the index, such that each of the plurality of reference frames is set as one of the at least one long-term reference frame. [0059] In some implementations, the method further comprises: in response to the at least one long-term reference frame being used in the encoding, invalidating a short-term reference frame stored in the reference frame buffer.

[0060] In some implementations, the encoder comprises an AVI encoder, and invalidating the short-term reference frame stored in the reference frame buffer comprises: determining an index of the short-term reference frame in the reference frame buffer; and replacing the short-term reference frame stored in the reference frame buffer with the second frame by setting a second syntax element with the index, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

[0061] In some implementations, the method further comprises: in response to the at least one long-term reference frame being used in the encoding, invalidating other long-term reference frames stored in the reference frame buffer.

[0062] In some implementations, the encoder comprises an AVI encoder, and invalidating the other long-term reference frames stored in the reference frame buffer comprises: determining indices of the other long-term reference frames in the reference frame buffer; and replacing the other long-term reference frames stored in the reference frame buffer with the second frame by setting a second syntax element with the indices, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

[0063] In some implementations, the reference frame buffer stores at most a predetermined number of long-term reference frames, and the method further comprises: in accordance with a determination that the number of long-term reference frames stored in the reference frame buffer does not exceed the predetermined number, identifying at least one frame in the video stream as a long-term reference frame; and storing the identified at least one frame into the reference frame buffer.

[0064] In some implementations, the encoder comprises an AVI encoder, and identifying the at least one frame in the video stream as a long-term reference frame comprises: determining an index to be allocated to the at least one frame, the index indicating a position in the reference frame buffer where the at least one frame is to be stored; and identifying the at least one frame as a long-term reference frame by setting a second syntax element with the index, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

[0065] In a second aspect, the subject matter as described herein provides an electronic device. The electronic device comprises: a processing unit; and a memory coupled to the processing unit and having instructions stored thereon, the instructions when executed by the processing unit causing the electronic device to perform acts comprising: receiving control information from a decoder, the control information indicating that the decoder fails to receive a first frame in a video stream transmitted from the electronic device; determining, from long-term reference frames stored in a reference frame buffer, at least one long-term reference frame for encoding a second frame not yet encoded after the first frame in the video stream, the control information indicating that the at least one long-term reference frame is received by the decoder; and encoding the second frame based on the at least one long-term reference frame.

[0066] In some implementations, the encoder comprises an AVI encoder, and encoding the second frame based on the at least one long-term reference frame comprises: determining an index of the at least one long-term reference frame in the reference frame buffer; and performing AVI encoding on the second frame based on the at least one long term frame by setting a first syntax element with the index, the first syntax element being used to set a reference frame in the AVI encoder.

[0067] In some implementations, the first syntax element is used to set a plurality of reference frames for encoding the second frame, and setting the first syntax element with the index comprises: setting the first syntax element with the index, such that each of the plurality of reference frames is set as one of the at least one long-term reference frame. [0068] In some implementations, the acts further comprise: in response to the at least one long-term reference frame being used in the encoding, invalidating a short-term reference frame stored in the reference frame buffer.

[0069] In some implementations, the electronic device comprises an AVI encoder, and invalidating the short-term reference frame stored in the reference frame buffer comprises: determining an index of the short-term reference frame in the reference frame buffer; and replacing the short-term reference frame stored in the reference frame buffer with the second frame by setting a second syntax element with the index, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

[0070] In some implementations, the acts further comprise: in response to the at least one long-term reference frame being used in the encoding, invalidating other long-term reference frames stored in the reference frame buffer.

[0071] In some implementations, the electronic device comprises an AVI encoder, and invalidating the other long-term reference frames stored in the reference frame buffer comprises: determining indices of the other long-term reference frames in the reference frame buffer; and replacing the other long-term reference frames stored in the reference frame buffer with the second frame by setting a second syntax element with the indices, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

[0072] In some implementations, the reference frame buffer is used to store at most a predetermined number of long-term reference frames, and the acts further comprise: in accordance with a determination that the number of long-term reference frames stored in the reference frame buffer does not exceed the predetermined number, identifying at least one frame in the video stream as a long-term reference frame; and storing the identified at least one frame into the reference frame buffer.

[0073] In some implementations, the electronic device comprises an AVI encoder, and identifying the at least one frame in the video stream as a long-term reference frame comprises: determining an index to be allocated to the at least one frame, the index indicating a position in the reference frame buffer where the at least one frame is to be stored; and identifying the at least one frame as a long-term reference frame by setting a second syntax element with the index, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

[0074] In a third aspect, the subject matter as described herein provides a computer program product, which is tangibly stored in a computer storage medium and includes machine executable instructions which, when executed by a device, cause the device to perform the method in accordance with the first aspect.

[0075] In a fourth aspect, the subj ect matter described herein provides a computer readable medium which stores thereon machine executable instructions which, when executed by a device, cause the device to perform the method in accordance with the first aspect.

[0076] The functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[0077] Program code for carrying out methods of the subject matter as described herein may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowcharts and/or block diagrams to be implemented. The program code may execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0078] In the context of this disclosure, a machine readable medium may be any tangible medium that may contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine readable medium may be a machine readable signal medium or a machine readable storage medium. A machine readable medium may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0079] Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the subject matter described herein, but rather as descriptions of features that may be specific to particular implementations. Certain features that are described in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable sub combination.

[0080] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A computer-implemented method, comprising: receiving, at an encoder, control information from a decoder, the control information indicating that the decoder fails to receive a first frame in a video stream transmitted from the encoder; determining, from long-term reference frames stored in a reference frame buffer, at least one long-term reference frame for encoding a second frame not yet encoded after the first frame in the video stream, the control information indicating that the at least one long term reference frame is received by the decoder; and encoding the second frame based on the at least one long-term reference frame.

2. The method of claim 1, wherein the encoder comprises an AVI encoder, and encoding the second frame based on the at least one long-term reference frame comprises: determining an index of the at least one long-term reference frame in the reference frame buffer; and performing AVI encoding on the second frame based on the at least one long-term frame by setting a first syntax element with the index, the first syntax element being used to set a reference frame in the AVI encoder.

3. The method of claim 2, wherein the first syntax element is used to set a plurality of reference frames for encoding the second frame, and setting the first syntax element with the index comprises: setting the first syntax element with the index, such that each of the plurality of reference frames is set as one of the at least one long-term reference frame.

4. The method of claim 1, further comprising: in response to the at least one long-term reference frame being used in the encoding, invalidating a short-term reference frame stored in the reference frame buffer.

5. The method of claim 4, wherein the encoder comprises an AVI encoder, and invalidating the short-term reference frame stored in the reference frame buffer comprises: determining an index of the short-term reference frame in the reference frame buffer; and replacing the short-term reference frame stored in the reference frame buffer with the second frame by setting a second syntax element with the index, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

6. The method of claim 1, further comprising: in response to the at least one long-term reference frame being used in the encoding, invalidating other long-term reference frames stored in the reference frame buffer.

7. The method of claim 6, wherein the encoder comprises an AVI encoder, and invalidating the other long-term reference frames stored in the reference frame buffer comprises: determining indices of the other long-term reference frames in the reference frame buffer; and replacing the other long-term reference frames stored in the reference frame buffer with the second frame by setting a second syntax element with the indices, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

8. The method of claim 1, wherein the reference frame buffer stores at most a predetermined number of long-term reference frames, and the method further comprises: in accordance with a determination that the number of long-term reference frames stored in the reference frame buffer does not exceed the predetermined number, identifying at least one frame in the video stream as a long-term reference frame; and storing the identified at least one frame into the reference frame buffer.

9. The method of claim 8, wherein the encoder comprises an AVI encoder, and identifying the at least one frame in the video stream as a long-term reference frame comprises: determining an index to be allocated to the at least one frame, the index indicating a position in the reference frame buffer where the at least one frame is to be stored; and identifying the at least one frame as a long-term reference frame by setting a second syntax element with the index, the second syntax element being used for refreshing the reference frame buffer in the AVI encoder.

10. An electronic device, comprising: a processing unit; and a memory coupled to the processing unit and having instructions stored thereon, the instructions when executed by the processing unit causing the electronic device to perform acts comprising: receiving control information from a decoder, the control information indicating that the decoder fails to receive a first frame in a video stream transmitted from the electronic device; determining, from long-term reference frames stored in a reference frame buffer, at least one long-term reference frame for encoding a second frame not yet encoded after the first frame in the video stream, the control information indicating that the at least one long-term reference frame is received by the decoder; and encoding the second frame based on the at least one long-term reference frame.

11. The electronic device of claim 10, wherein the electronic device comprises an AVI encoder, and encoding the second frame based on the at least one long-term reference frame comprises: determining an index of the at least one long-term reference frame in the reference frame buffer; and performing AVI encoding on the second frame based on the at least one long-term frame by setting a first syntax element with the index, the first syntax element being used to set a reference frame in the AVI encoder.

12. The electronic device of claim 11, wherein the first syntax element is used to set a plurality of reference frames for encoding the second frame, and setting the first syntax element with the index comprises: setting the first syntax element with the index, such that each of the plurality of reference frames is set as one of the at least one long-term reference frame.

13. The electronic device of claim 10, wherein the acts further comprise: in response to the at least one long-term reference frame being used in the encoding, invalidating a short-term reference frame stored in the reference frame buffer.

14. The electronic device of claim 10, wherein the acts further comprise: in response to the at least one long-term reference frame being used in the encoding, invalidating other long-term reference frames stored in the reference frame buffer.

15. A computer program product stored tangibly in a computer storage medium and including machine-executable instructions which, when executed by a device, cause the device to perform acts comprising: receiving control information from a decoder, the control information indicating that the decoder fails to receive a first frame in a video stream transmitted from the device; determining, from long-term reference frames stored in a reference frame buffer, at least one long-term reference frame for encoding a second frame not yet encoded after the first frame in the video stream, the control information indicating that the at least one long term reference frame is received by the decoder; and encoding the second frame based on the at least one long-term reference frame.