JP6059219B2 - Latency reduction in video encoding and decoding - Google Patents

Description

  The present invention relates to latency reduction in video encoding and decoding.

  Engineers use compression (also called source coding or source encoding) to reduce the bit rate of digital video. Compression reduces the cost of storing and transmitting video information by converting the information to a lower bit rate format. Decompression (also called decoding) reconstructs a version of the original information from a compressed format. A “codec” is an encoder / decoder system.

  Various video codec standards have been adopted over the past 20 years. Such a standard is H.264. 261, H.H. 262 (MPEG-2 or ISO / IEC 13818-2), H.264. 263 and H264 (AVC or ISO / IEC 14496-10) standards, as well as MPEG-1 (ISO / IEC 11172-2), MPEG-4 Visual (ISO / IEC 14496-2) and SMPTE 421M standards. More recently, the HEVC standard is being formulated. Video codec standards typically specify syntax options for the encoded video bitstream and describe in detail the parameters in the bitstream when specific functions are used during encoding and decoding. In many cases, video codec standards also provide details regarding the decoding operations that should be performed to achieve the correct results when the decoder decodes.

  The basic goal of compression is to provide good rate distortion performance. Thus, at a particular bit rate, the encoder tries to provide the highest quality video. Alternatively, with a certain level of quality / fidelity to the original video, the encoder attempts to provide the lowest bit rate encoded video. In practice, depending on the usage scenario, the coding time, coding complexity, coding resources, decoding time, decoding complexity, decoding resources, overall delay, and / or smoothness of playback may also be Affects decisions made during encoding and decoding.

  For example, video playback from a storage device, video playback from encoded data streamed over a network connection, and video (from one bit rate to another, or from one standard to another) Consider usage scenarios such as transcoding. On the encoder side, such an application allows off-line coding that is totally time independent. Thus, the encoder can increase the encoding time and resources used during encoding to find the most efficient way to compress video, thereby improving rate distortion performance. If a small amount of delay is acceptable at the decoder side, the encoder can further improve the rate distortion performance, for example, by taking advantage of inter-picture dependency from the picture farther ahead of the sequence.

  Also consider usage scenarios such as remote desktop conferencing, surveillance video, video telephony, and other real-time communication scenarios. Such applications are time dependent. The short latency between recording the input picture and playing the output picture is a major factor in performance. When an encoding / decoding tool applied to non-real time communication is applied to a real time communication scenario, the overall latency is often unacceptably long. The delay introduced by these tools during encoding and decoding can improve the performance of normal video playback, but they ruin real-time communication.

  In summary, techniques and tools that reduce latency in video encoding and decoding are presented. This technique and tool reduces latency and improves responsiveness in real-time communications. For example, techniques and tools may indicate frame reordering latency constraints by limiting latency due to video frame reordering, and with one or more syntax elements associated with the encoded data of the video frame. To reduce the overall waiting time.

  In accordance with one aspect of the techniques and tools described herein, a tool such as a video encoder, a real-time communication tool with a video encoder, or other tool can be used for latency constraints (eg, between multiple frames of a video sequence). One or a plurality of syntax elements indicating a frame rearrangement waiting time constraint that coincides with the inter-frame dependency are set. The tool outputs syntax elements, thereby making it easier and faster to determine when the reconstructed frame is ready for output in terms of the output order of the frames.

  In accordance with another aspect of the techniques and tools described herein, a tool such as a video decoder, a real-time communication tool with a video decoder, or other tool can provide latency constraints (e.g., frame reorder latency One or more syntax elements indicating (constraints) are received and analyzed. The tool also receives encoded data for multiple frames of the video sequence. At least a portion of the encoded data is decoded to reconstruct one of the frames. The tool can determine constraints based on the syntax elements, and then use latency constraints to determine when the reconstructed frame is ready for output (in terms of output order). The tool outputs a reconstruction frame.

  The foregoing and other objects, features and advantages of the present invention will become more apparent from the following detailed description, which proceeds with reference to the accompanying figures.

FIG. 1 is an illustration of an example computing system in which some described embodiments may be implemented. FIG. 3 is an illustration of an example network environment in which some described embodiments may be implemented. FIG. 3 is an illustration of an example network environment in which some described embodiments may be implemented. FIG. 3 is a related exemplary encoder system diagram in which some described embodiments may be implemented. FIG. 3 is a related exemplary decoder system diagram in which some described embodiments may be implemented. FIG. 5 is a diagram illustrating a frame encoding order and an output order in an exemplary series. FIG. 5 is a diagram illustrating a frame encoding order and an output order in an exemplary series. FIG. 5 is a diagram illustrating a frame encoding order and an output order in an exemplary series. FIG. 5 is a diagram illustrating a frame encoding order and an output order in an exemplary series. FIG. 5 is a diagram illustrating a frame encoding order and an output order in an exemplary series. 6 is a flowchart illustrating an exemplary technique for setting and outputting one or more syntax elements indicating latency constraints. 6 is a flowchart illustrating an example technique for decoding with reduced latency.

  The detailed description presents techniques and tools that reduce latency in video encoding and decoding. This technique and tool helps reduce latency and improves responsiveness in real-time communications.

  In video encoding / decoding scenarios, a certain delay between when an input video frame is received and when the frame is played is inevitable. The frame is encoded by the encoder, delivered to the decoder, and decoded by the decoder. And a certain amount of latency is caused by practical limits on coding resources, decoding resources and / or network bandwidth. However, other waiting times can be avoided. For example, latency may be introduced by encoders and decoders to improve rate distortion performance (eg, to take advantage of interframe dependencies from pictures that are farther forward in the sequence). Such latency can be reduced, although it may be detrimental in terms of rate distortion performance, processor utilization, or smoothness of playback.

  With the techniques and tools described herein, latency is reduced by suppressing latency (thus limiting the time range of inter-frame dependencies) and by showing latency constraints to the decoder. The For example, the waiting time constraint is a frame rearrangement waiting time constraint. Alternatively, the latency constraint is a constraint in terms of seconds, milliseconds, or another amount of time. The decoder can then determine latency constraints and use the constraints when determining which frames are ready to output. Thus, delay can be reduced for remote desktop conferencing, video telephony, surveillance video, webcam video, and other real-time communication applications.

  Some of the innovations described here are H.264 and / or HEVC standards specific syntax elements and operations will be described. These innovations can be implemented in other standards or formats.

  More generally, various alternatives to the examples described herein are possible. The specific technique described with reference to the flowchart can be changed by changing the order of the steps shown in the flowchart to divide, repeat, or omit the specific steps. Various aspects of reducing video encoding and decoding latency can be used in combination or separately. Different embodiments use one or more of the described techniques and tools. Some of the techniques and tools described herein solve one or more of the problems described in the background. Usually a given technology / tool does not solve all these problems.

<I. Exemplary Computing System>
FIG. 1 illustrates a general example of a suitable computing system (100) in which some described techniques and tools may be implemented. Since the techniques and tools can be implemented in a variety of general purpose or special purpose computing systems, the computing system (100) does not suggest any limitation as to the scope of use or functionality.

  Referring to FIG. 1, the computing system (100) includes one or more processing units (110, 115) and memory (120, 125). In FIG. 1, this most basic configuration (130) is surrounded by a dashed line. The processing units (110, 115) execute computer-executable instructions. The processing unit may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC) processor, or any other type of processor. In a multiprocessing system, multiple processing units execute computer-executable instructions to increase processing power. For example, FIG. 1 also shows a graphics processing unit or coprocessing unit (115) along with a central processing unit (110). Tangible memory (120, 125) can be volatile memory (eg, registers, cache, RAM), non-volatile memory (eg, ROM, EEPROM, flash memory, etc.), or a combination of the two, accessible by the processing unit. There may be. The memory (120, 125) stores software (180) that implements one or more innovations that reduce the latency of video encoding and decoding in the form of computer-executable instructions suitable for execution by the processing unit. .

  The computer system may have additional functions. For example, the computing system (100) includes a storage device (140), one or more input devices (150), one or more output devices (160), and one or more communication connections (170). An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system (100). Typically, operating system software (not shown) provides an operating environment for other software executing on the computing system (100) and coordinates the activities of the components of the computing system (100).

  The tangible storage device (140) may be removable or non-removable and can be used to store information in a magnetic disk, magnetic tape or cassette, CD-ROM, DVD, or non-transitory manner and It includes any other medium accessible within computing system (100). The storage device (140) stores instructions for software (180) that implements one or more innovations that reduce video encoding and decoding latency.

  The input device (150) may be a touch input device such as a keyboard, mouse, pen or trackball, a voice input device, a scanning device, or another device that provides input to the computing system (100). For video encoding, the input device (150) is a camera, video card, TV tuner card, or similar device that accepts video input in analog or digital form, or a CD-ROM that reads video samples into the computing system (100). Alternatively, it may be a CD-RW. The output device (160) may be a display, printer, speaker, CD writer, or another device that provides output from the computing system (100).

  Communication connection (170) allows communication with other computing entities via a communication medium. The communication medium conveys information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. The modulated data signal is a signal that has one or more of its characteristics set or may be changed to encode information in the signal. By way of example, and not limitation, communication media can use electrical, optical, RF or other carriers.

  Techniques and tools may be described in the general concept of computer-readable media. Computer readable media can be any available tangible media that can be accessed within a computing environment. By way of example, and not limitation, in conjunction with computing system (100), computer-readable media includes memory (120, 125), storage device (140), and any combination thereof.

  Techniques and tools may be described in the general notion of computer-executable instructions, such as those included in program modules, that are executed on a computing system on a real or virtual processor of interest. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functions of the program modules may be combined or divided among the program modules as required in various embodiments. Computer-executable instructions for program modules may be executed within a local or distributed computing system.

  The terms “system” and “apparatus” are used interchangeably herein. Unless otherwise noted in context, no terms imply any limitation as to the type of computing system or computing device. In general, a computing system or computing device is local or distributed and may include any combination of special purpose hardware and / or general purpose hardware and software that performs the functions described herein.

  For presentation purposes, the detailed description uses terms such as “determine” and “use” to describe computer operations within the computing system. These terms are high-level abstractions of operations performed by a computer and should not be confused with operations performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.

<II. Exemplary network environment>
2A and 2B illustrate an exemplary network environment (201, 202) having a video encoder (220) and a video decoder (270). The encoder (220) and decoder (270) are connected via the network (250) using an appropriate communication protocol. The network (250) may include the Internet or other computer network.

  In the network environment (201) shown in FIG. 2A, each real-time communication (RTC) tool (210) includes both an encoder (220) and a decoder (270) for bidirectional communication. A given encoder (220), together with a corresponding decoder (270) that accepts encoded data from the encoder (220), SMPTE 421M standard, ISO-IEC 14496-10 standard (also known as H.264 or AVC), HEVC Output can be generated according to a standard, another standard, or proprietary format. Two-way communication may be part of a video conference, video phone, or other two-party communication scenario. The 2A network environment (201) has two real-time communication tools (210), but alternatively the network environment (201) may have more than two real-time communication tools (210) participating in multi-party communication. Is possible.

  The real-time communication tool (210) manages encoding by the encoder (220). FIG. 3 shows an exemplary encoder system (300) that may be included in the real-time communication tool (210). Alternatively, the real-time communication tool (210) uses another encoder system. The real-time communication tool (210) also manages the decoding by the decoder (270). FIG. 4 shows an exemplary decoder system (400) that may be included in the real-time communication tool (210). Alternatively, the real-time communication tool (210) uses another decoder system.

  In the network environment (202) shown in FIG. 2B, the encoding tool (212) has an encoder (220) that encodes video for delivery to a plurality of playback tools (214) including a decoder (270). One-way communication may be provided for video surveillance systems, webcam surveillance systems, remote desktop conference presentations, or other scenarios where video is transmitted from one location to one or more other locations. Although the network environment (202) of FIG. 2B has two playback tools (214), the network environment (202) may have more or fewer playback tools (214). In general, the playback tool (214) communicates with the encoding tool (212) to determine the video stream of the playback tool (214) to be received. The playback tool (214) receives this stream, buffers the received encoded data for an appropriate period, and begins decoding and playback.

  FIG. 3 shows an exemplary encoder system (300) that may be included in the encoding tool (212). Alternatively, the encoding tool (212) uses another encoder system. The encoding tool (212) may also have server-side control logic that manages connections with one or more playback tools (214). FIG. 4 shows an exemplary decoder system (400) that may be included in the playback tool (214). Alternatively, the playback tool (214) uses another decoder system. The playback tool (214) may also have client-side control logic that manages the connection with the encoding tool (212).

  In some cases, the use of syntax elements to indicate latency (eg, frame reordering latency) is specific to a particular standard or format. For example, the encoded data may indicate latency constraints as part of the syntax of the base encoded video bitstream defined according to the standard or format, or as defined media metadata associated with the encoded data. Multiple syntax elements may be included. In such cases, the real-time communication tool (210), encoding tool (212) and / or reduced latency playback tool (214) is codec dependent. Here, the decisions made by these tools may depend on the bitstream syntax of a particular standard or format.

  In other cases, the use of syntax elements to indicate latency (eg, frame reordering latency) constraints are outside the scope of a particular standard or format. For example, syntax elements indicating latency constraints may be conveyed as part of the media transmission stream syntax, media storage file, or more generally the media system multiplexing protocol or transfer protocol. Alternatively, a syntax element indicating latency can be negotiated between the real-time communication tool (210), the encoding tool (212) and / or the playback tool (214) according to a media property negotiation protocol. In such cases, the real-time communication tool (210), the encoding tool (212) and / or the reduced latency playback tool (214) are codec independent. Here, these tools may work with any available video encoder and decoder, subject to the level of control over interframe dependencies set during encoding.

<III. Exemplary Encoder System>
FIG. 3 is a diagram of a related exemplary encoder system (300) in which some described embodiments may be implemented. The encoder system (300) is an arbitrary one of a plurality of encoding modes, such as a short latency encoding mode for real-time communication, a transcoding mode, and a normal encoding mode for media playback from a file or stream. It can be a general-purpose encoding tool that can operate in the following modes. Alternatively, the encoder system (300) may be a special purpose encoding tool adapted to one of the aforementioned encoding modes. The encoder system (300) can be implemented as an operating system module, as part of an application library, or as a stand-alone application. In general, the encoder system (300) receives a source video frame sequence (311) from a video source (310) and generates encoded data as output to a channel (390). The encoded data output to the channel may include one or more syntax elements that indicate constraints on latency (eg, frame reordering latency) to achieve decoding with reduced latency.

  The video source (310) can be a camera, tuner card, storage medium, or other digital video source. The video source (310) generates a video frame sequence at a frame rate of 30 frames per second, for example. As used herein, the term “frame” generally refers to source, encoded or reconstructed image data. In progressive video, the frame is a progressive video frame. In the exemplary embodiment, for interlaced video, interlaced video frames are deinterlaced before encoding. Alternatively, the two complementary interlaced video fields are encoded as interlaced video frames or separate fields. Apart from referring to progressive video frames, the term “frame” refers to a single unpaired video field, a complementary pair of video fields, a video object plane representing a video object at a given time, or a large image. May indicate a region of interest. A video object plane or region may be part of a larger image that includes multiple objects or regions of the scene.

  The incoming source frame (311) is stored in a source frame temporary storage area (320) including a plurality of frame buffer storage areas (321, 322,..., 32n). The frame buffer (321, 322, etc.) holds one source frame in the source frame storage area (320). After one or more source frames (311) are stored in the frame buffer (321, 322, etc.), the frame selector (330) periodically selects individual source frames from the source frame storage area (320). . The order in which frames are selected by the frame selector (330) for input to the encoder (340) may be different from the order in which frames are generated by the video source (310). For example, the frames may be earlier in order to achieve temporal backward prediction. Before the encoder (340), the encoder system (300) may have a preprocessor (not shown) that performs pre-processing (eg, filtering) before encoding the selected frame (331).

  The encoder (340) encodes the selected frame (331), generates an encoded frame (341), and also generates a memory management control signal (342). When the current frame is not the first frame encoded, when performing the encoding process, the encoder (340) encodes one or more previous codes stored in the decoded frame temporary storage area (360). An encoded / decoded frame (369) may be used. The stored decoded frame (369) is used as a reference frame for interframe prediction of the content of the current source frame (331). In general, the encoder (340) has a plurality of encoding modules that perform encoding tasks such as motion estimation and compensation, frequency conversion, quantization and entropy encoding. The exact operation performed by the encoder (340) may vary depending on the compression format. The format of the output encoded data includes Windows (registered trademark) Media Video format, VC-1 format, MPEG-x format (for example, MPEG-1, MPEG-2, or MPEG-4), H.264, and so forth. It can be a 26x format (eg, H.261, H.262, H.263, H.264), HEVC format, or other formats.

  The encoded frame (341) and the memory management control signal (342) are processed by the decoding processing emulator (350). The decoding processing emulator (350) performs a decoding task to reconstruct the reference frame used by the encoder (340) in some of the decoder functions, eg, motion estimation and compensation. The decoding emulator (350) uses the memory management control signal (342) to use a given encoded frame (341) as a reference frame in interframe prediction of subsequent frames to be encoded. Determine whether it needs to be reconfigured and stored. If the control signal (342) indicates that the encoded frame (341) needs to be stored, the decoding processing emulator (350) receives the encoded frame 8341) and generates a corresponding decoded frame (351). The decoding process performed by the decoder is modeled. In this case, when the encoder (340) is using the decoded frame (369) stored in the decoded frame storage area (360), the decoding processing emulator (350) is also stored as part of the decoding processing. The decoded frame (369) from the region (360) is used.

  The decoded frame temporary storage area (360) has a plurality of frame buffer storage areas (361, 362, ..., 36n). Memory management for the decoding processing emulator (350) to identify frame buffers (361, 362,..., 36n) having frames that are no longer needed by the encoder (340) for use as reference frames The control signal (342) is used to manage the contents of the storage area (360). After modeling the decoding process, the decoding process emulator (350) stores the new decoded frame (351) in the frame buffer (361, 362, etc.) identified by this method.

  The encoded frame (341) and the memory management control signal (342) are also buffered in the temporary encoded data area (370). The encoded data collected in the encoded data area (370) may have one or more syntax elements indicating latency constraints as part of the syntax of the base encoded video bitstream. Alternatively, the encoded data collected in the encoded data area (370) may be part of media metadata associated with the encoded video data (eg, one or more supplemental enhancement information (SEI)). Message or video usability information (VUI) message (as one or more parameters), may have a syntax element indicating a latency constraint.

  Data (371) collected from the temporary encoded data area (370) is processed by the channel encoder (380). A channel encoder (380) may packetize the collected data for transmission as a media stream. In this case, the channel encoder (380) can add a syntax element indicating a latency constraint as part of the syntax of the media transmission stream. Alternatively, the channel encoder (380) may organize the collected data for storage as a file. In this case, the channel encoder (380) can add a syntax element indicating a latency constraint as part of the syntax of the media storage file. Or, more generally, the channel encoder (380) may implement one or more media system multiplexing protocols or transfer protocols. In this case, the channel encoder (380) can add a syntax element indicating a latency constraint as part of the protocol syntax. A channel encoder (380) provides an output to a channel (390) that represents another channel for storage, communication connection or output.

<IV. Exemplary Decoder System>
FIG. 4 is a diagram of a related exemplary decoder system (400) in which some described embodiments may be implemented. The decoder system (400) operates in any of a plurality of decoding modes, such as a short latency decoding mode for real-time communication and a normal decoding mode for media playback from a file or stream. It can be a general purpose decryption tool possible. Alternatively, the decoder system (400) may be a special purpose decoding tool adapted to one of the aforementioned decoding modes. The decoder system (400) can be implemented as an operating system module, as part of an application library, or as a stand-alone application. In general, the decoder system (400) receives encoded data from a channel (410) and generates a reconstructed frame as output to an output destination (490). The encoded data may include one or more syntax elements that indicate constraints on latency (eg, frame reordering latency) to achieve decoding with reduced latency.

  The decoder system (400) has as input a channel (410) representing another channel for storage, communication connection or encoded data. The channel (410) generates encoded data that is channel-encoded. The channel decoder (420) can process the encoded data. For example, the channel decoder (420) may depacketize the collected data for transmission as a media stream. In this case, the channel decoder (420) can parse a syntax element indicating a latency constraint as part of the syntax of the media transmission stream. Alternatively, the channel decoder (420) separates the encoded video data collected for storage as a file. In this case, the channel decoder (420) can parse a syntax element indicating a waiting time constraint as part of the syntax of the media storage file. Or, more generally, the channel decoder (420) may implement one or more media system demultiplexing or transfer protocols. In this case, the channel decoder (420) can analyze a syntax element indicating a latency constraint as part of the protocol syntax.

  The encoded data (421) output from the channel decoder (420) is stored in the temporary encoded data area (430) until a sufficient amount of data is received. The encoded data (421) includes an encoded frame (431) and a memory management control signal (432). The encoded data (421) in the encoded data area (430) may have one or more syntax elements indicating latency constraints as part of the syntax of the base encoded video bitstream. Alternatively, the encoded data (421) in the encoded data area (430) may be part of the media metadata associated with the encoded video data (eg, one or more in one or more SEI messages or VUI messages). May have a syntax element indicating a latency constraint. In general, the encoded data area (430) temporarily stores the encoded data (421) until the encoded data (421) is used by the decoder (450). At this point, the encoded data of the encoded frame (431) and the memory management control signal (432) is sent from the encoded data area (430) to the decoder (450). As decoding continues, new encoded data is added to the encoded data area (430) and the old encoded data remaining in the encoded data area (430) is sent to the decoder (450).

  The decoder (450) periodically decodes the encoded frame (431) to generate a corresponding decoded frame (451). Where appropriate, when performing the decoding process, the decoder (450) may use one or more previously decoded frames (469) as reference frames for inter-frame prediction. The decoder (450) reads such a previously decoded frame (469) from the decoded frame temporary storage area (460). In general, the decoder (450) has a plurality of decoding modules that perform decoding tasks such as entropy decoding, inverse quantization, inverse frequency transform, and motion compensation. The exact operation performed by the decoder (450) may vary depending on the compression format.

  The decoded frame temporary storage area (460) has a plurality of frame buffer storage areas (461, 462, ..., 46n). The decoded frame storage area (460) is an example of a decoded picture buffer. The decoder (450) uses the memory management control signal (432) to identify a frame buffer (461, 462, etc.) that can store the decoded frame (451). The decoder (450) stores the decoded frame (451) in the frame buffer.

  The output sequencer (480) uses the memory management control signal (432) to identify when the next frame to be generated in output order is available in the decoded frame storage area (460). To reduce the latency of the encoding-decoding system, the output sequencer (480) uses syntax elements that indicate latency constraints to quickly process the identification of frames to be generated in output order. . When the next frame (481) to be generated in output order is available in the decoded frame storage area (460), it is read by the output sequencer (480) and output to (490) (eg, To the display). In general, the order in which frames are output from the decoded frame storage area (460) by the output sequencer (480) may be different from the order in which frames are decoded by the decoder (450).

<V. Syntax element for realizing encoding and decoding with reduced waiting time>
In most video codec systems, the encoding order (also referred to as decoding order or bitstream order) is the order in which video frames appear in the decoded data in the bitstream and are therefore processed during decoding. is there. The encoding order may be different from the order in which frames are captured by the camera prior to encoding, and the order in which decoded frames are displayed, stored or otherwise output after decoding (output order or display order) And may be different. Although the rearrangement of frames with respect to the output order is advantageous (mainly in terms of compression performance), it increases the latency between the end and end of the encoding and decoding processes.

  The techniques and tools described herein reduce latency due to video frame reordering and also provide latency information by the decoder system by providing information about the reordering latency constraint to the decoder system. Reduction is also realized. Such a reduction in waiting time is useful for many purposes. For example, latency reduction can be used to reduce the time lag that occurs in two-way video communication using a video conferencing system, allowing faster conversation flow and communication bidirectionality between remote participants. And become natural.

A. Approach for output timing and output order According to the H.264 standard, a decoder can use two approaches to determine when a decoded frame is ready for output. The decoder can use timing information in the form of a decoding timestamp and an output timestamp (eg, when conveyed in a picture timing SEI message). Alternatively, the decoder can use buffering performance limits conveyed in various syntax elements to determine when the decoded frame is ready for output.

  Timing information can be associated with each decoded frame. The decoder can use the timing information to determine when the decoded frame can be output. In practice, however, such timing information may not be available to the decoder. Furthermore, even when timing information is available, some decoders do not actually use this information (because the decoder is designed to operate regardless of whether timing information is available).

  Buffering performance limits include syntax element max_dec_frame_buffering, syntax element num_reorder_frames, relative order information (referred to as “picture order count”) and other memory management control information conveyed in the bitstream. H., et al. It is shown with several syntax elements according to the H.264 standard (and the draft version of the HEVC standard). The syntax element max_dec_frame_buffering (or a derived variable specified as MaxDpbFrames) specifies the required size of the decoded picture buffer (DPB) in the unit of the frame buffer. Thus, the syntax element max_dec_frame_buffering represents the top level memory capacity used for the encoded video sequence and causes the decoder to output pictures in the correct order. The syntax element num_reorder_frames (or max_num_reorder_frames) is a frame (or complementary field pair or non-pair field) that precedes any frame (or complementary field pair or non-pair field) in encoding order and follows in output order. ) Is the maximum number. In other words, num_reorder_frames specifies a constraint on the memory capacity necessary for picture rearrangement. The syntax element max_num_ref_frames specifies the maximum number of short-term and long-term reference frames (or complementary reference field pairs, or non-paired reference fields) that can be used by the decoding process for inter prediction of any picture in the sequence. . The syntax element max_num_ref_frames also determines the size of the sliding window for marking the decoded reference picture. Like num_reorder_frames, max_num_ref_frames specifies constraints on the required memory capacity.

  The decoder uses max_dec_frame_buffering (or MaxDpbFrames) and num_reorder_frames syntax elements to determine when the buffering performance limit has been exceeded. This occurs, for example, when a new decoded frame needs to be stored in the DPB, but there is no remaining space available in the DPB. In this situation, the decoder uses the picture order information to identify which of the decoded pictures is the earliest in output order. Next, the earliest picture in the output order is output. Such a process is often referred to as “bumping” because a new picture that needs to be stored arrives and the picture is “pushed out” of the DPB.

  For the decoder, the information indicated by the max_dec_frame_buffering (or MaxDpbFrames) and num_reorder_frames syntax elements is sufficient to determine the memory capacity required by the decoder. However, the use of such information can introduce undesirable latency when used to control a “bumping” process to output a picture. H. As defined in the H.264 standard, the max_dec_frame_buffering and num_reorder_frames syntax elements do not set a limit on the amount of reordering that can be applied to any particular picture, and therefore do not set a limit on end-to-end latency. Regardless of the value of these syntax elements, a particular picture can be kept in the DPB for any long time before output. This long time corresponds to the substantial latency added by the pre-buffering of the source picture by the encoder.

B. Syntax Element Showing Frame Reordering Latency Constraints The techniques and tools described herein reduce latency in video communication systems. Encoding tools, real-time communication tools, or other tools set limits on the reordering range that can be applied to any frame in the encoded video sequence. For example, this limit is expressed as the number of frames that can precede any given frame in the encoded video sequence in the output order and frames that can follow in the encoding order. This limit limits the reordering latency allowed for any particular frame in the sequence. In other words, this limit limits (in terms of frames) the time range of reordering between encoding order and output order that can be applied to any particular frame. Sorting order restrictions help to reduce end-to-end delay. Such limit setting may also be useful in a real-time system negotiation protocol or application specification for usage scenarios where latency reduction is important.

  One or more syntax elements indicate constraints on frame reordering latency. Communicating constraints on frame reordering latency enables system level negotiation in two-way real-time communication or other usage scenarios. This directly represents constraints on frame reordering latency and provides a way to characterize media stream or session characteristics.

  The video decoder can use the indicated constraints on frame reordering latency to reduce the latency of output of the decoded video frame. In particular, communicating constraints on frame reordering latency compared to frame “bumping” processing allows the decoder to more easily and quickly identify frames in the DPB that are ready for output. To do. For example, the decoder can determine the latency state of a frame in the DPB by calculating the difference between the frame encoding order and the output order. By comparing the frame latency state to the constraints on frame reordering latency, the decoder can determine when the constraints on frame reordering latency are reached. The decoder can immediately output a frame that has reached this limit. This helps the decoder more quickly identify frames that are ready for output, as compared to a “bumping” process that uses various syntax elements and tracking structures. In this way, the decoder can quickly (and quickly) determine when a decoded frame can be output. The more quickly (and faster) the decoder can identify when a frame can be output, the faster the decoder can output (and earlier) video to the display or subsequent processing steps.

  Thus, with constraints on frame reordering latency, the decoder can begin outputting frames from the decoded frame storage area before the decoded frame storage area is full, but is still compatible decoding. (I.e., all frames are decoded so that the frames exactly match the frames decoded using another conventional scheme at the bit level). This significantly reduces the delay when the (frame) delay indicated by the latency syntax element is much smaller than the size (in frames) of the decoded frame storage area.

5A-5E show a series of frames (501-505) with different inter-frame dependencies. The series is (1) a constraint on the memory capacity required for picture reordering (ie, the number of frame buffers used to store reference frames for reordering purposes, eg as indicated by the syntax element num_reorder_frames) And (2) characterized by different values of the constraints on the frame reordering latency, for example as specified by the variable MaxLatencyFrames. In FIGS. 5A-5E, for a given frame F _j ^k , the subscript j indicates the position of the frame in the output order, and the superscript ^k indicates the position of the frame in the encoding order. The frames are shown in output order, where the subscript value increases from left to right. Arrows indicate interframe dependence for motion compensation. Thereby, the preceding frame in the encoding order is used for prediction of the subsequent frame in the encoding order. For simplicity, FIGS. 5A-5E show inter-frame dependencies at the frame level (rather than at the level of macroblocks, blocks, etc. where the reference frame may change). 5A-5E show up to two frames as reference frames for a given frame. In practice, in some implementations, different macroblocks, blocks, etc. within a given frame can use different reference frames, and more than one reference frame can be used for a given frame. .

In FIG. 5A, the series (501) has nine frames. The last frame F ₈ ¹ in the output order uses the first frame F ₀ ⁰ as a reference frame. The other frames in the series (501) use both the last frame F ₈ ¹ and the first frame F ₀ ⁰ as reference frames. This is because frame F ₀ ⁰ is decoded first, followed by frame F ₈ ¹ , followed by frame F ₁ ² , and so on. In the series (501) shown in FIG. 5A, the value of num_reorder_frames is 1. At any point in the processing of the decoder system, only one frame (F ₈ ¹ ) is stored in the decoded frame storage area for rearrangement among the frames shown in FIG. 5A. (The first frame is also used and stored as reference frame F ₀ ⁰ but is not stored for reordering purposes. Since the output order of the first frame F ₀ ⁰ is less than the output order of the intermediate frames, the first frame F ₀ ⁰ is not counted for the purpose of num_reorder_frames.) Despite the low value of num_reorder_frames, series (501) has a relatively long latency. The value of MaxLatencyFrames is 7. After encoding the first frame F ₀ ⁰ , the encoder encodes the next frame F ₁ ² in output order because the next frame F ₁ ² depends on the last frame F ₈ ¹ of the series (501). Wait until 8 or more source frames are buffered before doing so. The MaxLatencyFrames value is effectively the maximum allowable difference between the subscript value and the superscript value for any particular encoding frame.

  In FIG. 5B, like the series (501) in FIG. 5A, the series (502) has nine frames, but the inter-frame dependencies are different. The temporal reordering of frames occurs over a short range. As a result, the series (502) has a very low latency and the value of MaxLatencyFrames is 1. The value of num_reorder_frames is still 1.

  In FIG. 5C, the series (503) has 10 frames. The longest interframe dependency is shorter (in terms of time range) than the longest interframe dependency of FIG. 5, but longer than the longest interframe dependency of FIG. 5B. Series (503) has a similarly low value 1 for num_reorder_frames, but a relatively low value 2 for MaxLatencyFrames. Thus, the series (503) allows a lower end-to-end latency than the series (501) of FIG. 5A, but not as small as the allowable latency of the series (502) of FIG. 5B.

In FIG. 5D, the series (504) has frames organized in a temporal hierarchy with three temporal layers subject to interframe dependencies. The lowest temporal resolution layer has a first frame F ₀ ⁰ and a last frame F ₈ ¹ . The next temporal resolution layer adds the first frame F ₄ ² and depends on the first frame F ₀ ⁰ and the last frame F ₈ ¹ . The top temporal resolution layer adds the remaining frames. The series (504) shown in FIG. 5D has a relatively low value of 2 for num_reorder_frames, due to the difference between the encoding order and output order of the last frame F ₈ ¹ at least in the highest temporal resolution layer, It has a relatively high value of 7 for MaxLatencyFrames. If only the intermediate time resolution layer or the lowest time resolution layer is decoded, the constraints on frame reordering delay can be reduced to 1 (in the intermediate layer) or 0 (in the lowest layer). In order to achieve low latency decoding with various temporal resolutions, syntax elements can indicate frame reordering latency constraints for different layers in the temporal hierarchy.

In FIG. 5E, the series (505) has frames organized into a temporal hierarchy with three temporal layers that follow different inter-frame dependencies. The lowest temporal resolution layer has a first frame F ₀ ⁰ , an intermediate frame F ₄ ¹ and a last frame F ₈ ⁵ . The next temporal resolution layer consists of frame F ₂ ² (which depends on the first frame F ₀ ⁰ and intermediate frame F ₄ ¹ ) and frame F ₆ ⁶ (which depends on the intermediate frame F ₄ ¹ and last frame F ₈ ⁵ ). Add The top temporal resolution layer adds the remaining frames. Compared to the series (504) in FIG. 5D, the series (505) in FIG. 5E is at least the highest temporal resolution layer, and the encoding order and output order of the intermediate frame F ₄ ¹ and the last frame F ₈ ⁵ Due to the difference between them, it has a relatively low value of 2 for num_reorder_frames but a relatively low value of 3 for MaxLatencyFrames. If only the intermediate time resolution layer or the lowest time resolution layer is decoded, the constraints on frame reordering delay can be reduced to 1 (in the intermediate layer) or 0 (in the lowest layer).

  In the example shown in FIGS. 5A-5E, if the value of MaxLatencyFrames is known, the decoder can identify a particular frame that is ready for intermediate output when it receives a preceding frame in output order. For a given frame, the frame output order value minus the frame encoding order value may be equal to the value of MaxLatencyFrames. In this case, a given frame is prepared for output as soon as the preceding frame in the output order is received. (In contrast, such frames could not be identified as ready for output using only num_reorder_frames until additional frames were received or the end of the sequence was reached.) In particular, the decoder The value of MaxLatencyFrames can be used to enable early output of the following frames:

-Frame F ₈ ¹ in the series (501) of FIG. 5A
In the series (502) of FIG. 5B, frames F ₂ ¹ , F ₄ ³ , F ₆ ⁵ , F ₈ ⁷
-In the series (503) of FIG. 5C, frames F ₃ ¹ , F ₆ ⁴ , F ₉ ⁷
-Frame F ₈ ¹ in the series (504) of FIG. 5D
-Frames F ₄ ¹ and F ₈ ^{5 in} the series (505) of FIG. 5E
In addition, the declaration or negotiation of MaxLatencyFrames values at the system level provides a summary representation of the latency characteristics of a bitstream or session in a way that cannot be used by measuring the reordered storage capacity using num_reorder_frames and indicating that capacity. it can.

C. Exemplary Implementation A syntax element indicating a frame reordering latency constraint can be communicated in various ways depending on the implementation. A syntax element is conveyed as part of a sequence parameter set (SPS), a picture parameter set (PPS), or other elements of the bitstream, and is a SEI message, VUI message or other metadata. May be communicated as part of, or in some other way. In any implementation, the syntax element indicating the constraint value may be encoded and then communicated using unsigned exponential Golomb encoding, certain other types of entropy encoding, or fixed length encoding. After receiving the syntax element, the decoder performs the corresponding decoding.

  In the first implementation, the flag max_latency_limitation_flag is transmitted. If this flag has a first binary value (e.g. 0), no restrictions are imposed on the frame reordering latency. In this case, the value of the max_latency_frames syntax element is not transmitted or ignored. In other cases (when the flag has a second binary value such as 1), the value of the max_latency_frames syntax element is communicated to indicate a frame reordering latency constraint. For example, in this case, the transmitted value of the max_latency_frames syntax element may be any non-negative integer value.

In the second implementation, the syntax element max_latency_frames_plus1 is communicated to indicate a frame reordering latency constraint. If max_latency_frames_plus1 has a first binary value (eg 0), no restrictions are imposed on the frame reordering latency. For other values (for example, non-zero values), the value of the constraint on the frame rearrangement waiting time is set to max_latency_frames_plus1-1. For example, the value of max_latency_frames_plus1 is in the range of 0 to 2 ³² -2 comprehensively.

  Similarly, in the third implementation, the syntax element max_latency_frames is communicated to indicate frame reordering latency constraints. When max_latency_frames has a first value (eg, a maximum value), no restrictions are imposed on the frame rearrangement waiting time. For other values (for example, a value smaller than the maximum value), the constraint value for the frame rearrangement waiting time is set to max_latency_frames.

In the fourth implementation, the constraint on the frame reordering latency is shown in relation to the maximum size of the frame memory. For example, latency constraints are communicated as an increase over the num_reorder_frames syntax element. Usually, the constraint on frame reordering latency (in terms of frames) is greater than or equal to num_reorder_frames. To save bits in communicating latency constraints, the difference between latency constraints and num_reorder_frames (eg, using unsigned exponential Golomb coding, certain other forms of entropy coding) Are encoded and transmitted. The syntax element max_latency_increase_plus1 is transmitted to indicate a frame rearrangement waiting time constraint. When max_latency_increase_plus1 has a first value (eg, 0), no restriction is imposed on the frame rearrangement waiting time. For other values (for example, non-zero values), the value of the constraint on the frame rearrangement waiting time is set to num_reorder_frames + max_latency_increase_plus1-1. For example, the value of max_latency_increase_plus1 is in the range of 0 to 2 ³² -2 comprehensively.

  Alternatively, one or more syntax elements indicating constraints on frame reordering latency are communicated in other specific ways.

D. Other ways to indicate latency constraints In many of the above examples, latency constraints are constraints on frame reordering latency expressed in terms of the number of frames. More generally, latency constraints are constraints on delay expressed in terms of the number of frames, or in terms of seconds, milliseconds, or another time measure. For example, the latency constraint can be expressed as an absolute time index, such as 1 second or 0.5 seconds. The encoder converts such a time index into a number of frames (considering the video frame rate) and then encodes the video so that the interframe dependency between multiple frames of the video sequence matches the number of frames. Turn into. Or, regardless of frame reordering and inter-frame dependency, the encoder can use time indices to smooth out short-term variations such as the bit rate of encoded video, encoding complexity, network bandwidth, etc. The range over which the delay is used can be limited. The decoder can use this time index to determine when a frame can be output from the decoded picture buffer.

  Latency constraints are negotiated between the sender and receiver, the ability to smooth short-term fluctuations in the bit rate of the encoded video, the ability to smooth short-term fluctuations in the coding complexity, the network Trade off response (no delay) with the ability to smooth short-term bandwidth fluctuations and / or another factor that benefits from increasing delay. In such negotiations, it can be useful to define and characterize latency constraints in a manner that is independent of the frame rate. The constraints can then be applied during encoding and decoding, taking into account the video frame rate. Alternatively, the constraints can be applied during encoding and decoding regardless of the video frame rate.

E. General Techniques for Setting and Outputting Syntax Elements FIG. 6 shows an exemplary technique (600) for setting and outputting syntax elements that achieve decoding with reduced latency. For example, the real-time communication tool or encoding tool described with reference to FIGS. 2A and 2B performs the technique (600). Alternatively, another tool performs the technique (600).

  First, the tool shows one or more constraints that indicate constraints on latency (eg, frame reorder latency, latency in terms of time index) that matches the interframe dependency between multiple frames of the video sequence. A syntax element is set (610). When the tool has a video encoder, the same tool receives the frame, encodes the frame (using interframe dependencies consistent with frame reordering latency constraints), and generates and stores encoded data Or, encoded data may be output for transmission.

  Typically, the frame reordering latency constraint is the reordering latency allowed for any frame in the video sequence. Constraints can be expressed in various ways, but have various other meanings. For example, constraints can be expressed in terms of the maximum number of frames that can precede a given frame in output order but can follow a given frame in encoding order. Alternatively, the constraint can be expressed as the maximum difference between the encoding order and the output order for any frame in the video sequence. Alternatively, focusing on individual frames, the constraint can be expressed as a reordering latency associated with a given particular frame in the video sequence. Alternatively, focusing on a group of frames, the constraint can be expressed as a reordering latency associated with the group of frames in the video sequence. Alternatively, the constraints can be expressed in certain other ways.

  Next, the tool outputs a syntax element (620). This realizes the determination when the reconstructed frame is ready for output in terms of the output order of the plurality of frames. The syntax element is part of a sequence parameter set or picture parameter set in the base encoded video stream, part of the syntax of the media storage file or media transmission stream that also contains the encoded data of the frame (e.g., system As part of a media property negotiation protocol (during the exchange of stream or session parameter values during level negotiation), as part of media system information multiplexed with the encoded data of the frame, or (for example, SEI messages or As part of the media metadata associated with the encoded data of the frame (in the VUI message). Different syntax elements can be output to indicate memory capacity requirements. For example, a buffer size syntax element (such as max_dec_frame_buffering) can indicate the maximum size of the DPB, and a frame memory syntax element (such as num_reorder_frames) can indicate the maximum size of the frame memory for reordering. it can.

  The value of the waiting time constraint is V. It can be represented in various ways as described in section C. For example, the tool outputs a flag indicating the presence or absence of syntax elements. If the flag indicates that the syntax element is absent, the latency constraint is indefinite or has a default value. In other cases, the syntax element indicates a latency constraint according to the latency constraint. Alternatively, one value of the syntax element indicates that the latency constraint is indefinite or default, and another possible value of the syntax element indicates an integer that deserves the latency constraint. Alternatively, if the latency constraint is a frame reordering latency constraint, the given value of the syntax element deserves a frame reordering latency constraint on the maximum size of the frame memory for reordering. Indicates an integer. This is indicated by different syntax elements such as num_reorder_frames. Alternatively, latency constraints are expressed in certain other ways.

  In some implementations, the frames of the video sequence are organized according to a temporal hierarchical structure. In this case, different syntax elements may indicate different constraints on frame reordering latency for different temporal layers of the temporal hierarchy.

F. General Techniques for Receiving and Using Syntax Elements FIG. 7 shows an exemplary technique (700) for receiving and using syntax elements that provides decoding with reduced latency. For example, the real-time communication tool or playback tool described with reference to FIGS. 2A and 2B implements technique (700). Alternatively, another tool performs the technique (700).

  Initially, the tool receives and analyzes (710) one or more syntax elements that indicate constraints on latency (eg, frame reordering latency, latency from a time index perspective). For example, the analysis includes reading one or more syntax elements from the bitstream that indicate latency constraints. The tool also receives (720) encoded data for multiple frames of the video sequence. The tool can parse the syntax element and determine latency constraints based on the syntax element. Typically, the frame reordering latency constraint is the reordering latency allowed for any frame in the video sequence. As stated in the previous chapter, constraints can be expressed in various ways, but have various other meanings. The syntax element is encoded as part of the sequence parameter set or picture parameter set in the base encoded video stream, as part of the syntax of the media storage file or media transmission stream, as part of the media property negotiation protocol. It can be communicated as part of the media system information multiplexed with the data or as part of the media metadata associated with the encoded data. The tool can receive and analyze different syntax elements indicating memory capacity requirements, eg, buffer size syntax elements such as max_dec_frame_buffering and frame memory syntax elements such as num_reorder_frames.

  The value of the waiting time constraint is V. It can be represented in various ways as described in section C. For example, the tool receives a flag indicating the presence or absence of syntax elements. If the flag indicates that the syntax element is absent, the latency constraint is indefinite or has a default value. In other cases, the syntax element indicates a latency constraint according to the latency constraint. Alternatively, one value of the syntax element indicates that the latency constraint is indefinite or default, and another possible value of the syntax element indicates an integer that deserves the latency constraint. Alternatively, if the latency constraint is a frame reordering latency constraint, the given value of the syntax element deserves a frame reordering latency constraint on the maximum size of the frame memory for reordering. Indicates an integer. This is indicated by different syntax elements such as num_reorder_frames. Alternatively, latency constraints are communicated in certain other ways.

  Returning to FIG. 7, the tool decodes (730) at least a portion of the encoded data to reconstruct one of the frames. The tool outputs a reconstruction frame (740). In doing this, the tool can use latency constraints, for example, to determine when a reconstructed frame is ready for output in terms of the output order of the frames of the video sequence.

  In some implementations, the frames of the video sequence are organized according to a temporal hierarchical structure. In this case, different syntax elements may indicate different constraints on frame reordering latency for different temporal layers of the temporal hierarchy. The tool can select one of the different constraints of frame reordering latency depending on the temporal resolution of the output.

  In light of the many possible embodiments to which the disclosed inventive principles can be applied, it is understood that the illustrated embodiments are merely preferred examples of the invention and are not to be construed as limiting the invention. Rather, the scope of the present invention is defined by the appended claims. Accordingly, Applicants claim as Applicant's invention all that is encompassed within the scope and spirit of the appended claims.

Claims (14)

In a computing system comprising a video decoder,
Receiving and analyzing one or more syntax elements indicative of a frame reordering latency constraint, wherein the frame reordering latency constraint can precede any frame in the video sequence in output order; A step represented by the maximum number of frames that can follow the frame in a grouping order;
Receiving encoded data of a plurality of frames of the video sequence;
Decoding at least a portion of the encoded data to reconstruct one of the plurality of frames at the video decoder;
Outputting the reconstructed frame;
Having a method. Determining the frame reordering latency constraint based on the one or more syntax elements;
Determining when the reconstructed frame is ready to be output in terms of the output order of the plurality of frames of the video sequence using the frame rearrangement latency constraint;
The method of claim 1 further comprising:   The plurality of frames of the video sequence are organized according to a temporal hierarchy, wherein different syntax elements indicate different constraints of frame reordering latency for different temporal layers of the temporal hierarchy. The method of claim 2, further comprising selecting one of the different constraints of the frame reordering latency depending on the temporal resolution of the output.   The method of claim 1, wherein the frame reordering latency constraint is an acceptable reordering latency for any frame in the video sequence. The one or more syntax elements and the encoded data are transmitted as part of a syntax of an encoded video bitstream, and the method includes:
Receiving and analyzing a buffer size syntax element indicating a maximum size of a decoded picture buffer and a frame memory syntax element indicating a maximum size of a frame memory for rearrangement, the buffer size syntax element and the buffer size syntax element; A frame memory syntax element is different from the one or more syntax elements indicating the frame reordering latency constraint,
The method of claim 1 further comprising:   The one or more syntax elements include a sequence parameter set, a picture parameter set, a syntax of a media storage file that also includes the encoded data, a syntax of a media transmission stream that also includes the encoded data, a media characteristic negotiation protocol, The method of claim 1, wherein the method is conveyed as part of media system information multiplexed with encoded data or media metadata associated with the encoded data. One possible values of the one or more syntax elements, limitation of the frame reordering latency indicates as having a certain or predetermined value is undefined, an other possible of the one or more syntax elements The value indicates an integer that deserves the frame reordering latency constraint, or the value of the one or more syntax elements is the frame reordering latency constraint on the maximum size of the frame memory for reordering. The method of claim 1, wherein the method indicates an integer that deserves and the maximum size of the frame memory for the reordering is indicated by different syntax elements. In computing systems,
Setting one or more syntax elements indicating a frame reordering latency constraint that matches interframe dependencies between a plurality of frames of a video sequence, wherein the frame reordering latency constraint is output wherein the coding sequence can precede any frame of a video sequence represented by the maximum number of frames that can be followed by the frame in order, the steps,
Outputting the one or more syntax elements, thereby realizing a determination when the reconstructed frame is ready for output in terms of the output order of the plurality of frames; and
Having a method. The computing system comprises a video encoder, and the method comprises:
Receiving a plurality of frames of a video sequence;
A step of encoding the plurality of frames to generate encoded data in the video encoder, the encoding step including the inter-frame dependency that matches the frame rearrangement waiting time constraint; Using steps,
Outputting the encoded data for storage or transmission;
9. The method of claim 8, further comprising: A computing system comprising a processor, a memory, and a storage device, the computing system comprising:
Receiving and analyzing one or more syntax elements indicative of a frame reordering latency constraint;
Determining the frame reordering latency constraint based on the one or more syntax elements, wherein the frame reordering latency constraint can precede a given frame in output order and in encoding order. Represented in terms of the maximum number of frames that can follow the given frame;
Receiving encoded data of a plurality of frames of a video sequence;
Decoding at least a portion of the encoded data to reconstruct one of the plurality of frames at a video decoder;
Outputting the reconstructed frame in order to determine when the reconstructed frame is ready for output in terms of the output order of the plurality of frames of the video sequence. Including a step of using a reordering latency constraint;
A computing system adapted to perform a method comprising:   Receiving and analyzing one or more syntax elements indicating a maximum size of a frame memory for reordering, wherein the maximum size of the frame memory for reordering is determined in the encoding order of the video sequence; Represented in terms of the maximum number of frames that can precede any frame and can follow it in output order;
  The method of claim 1 further comprising:   The value of the one or more syntax elements indicates an integer worth the constraint of the frame rearrangement waiting time with respect to the maximum size of the frame memory for rearrangement, and the maximum size of the frame memory for rearrangement is: Indicated by different syntax elements,
  The frame rearrangement waiting time constraint is determined by adding an integer worth the frame rearrangement waiting time constraint and subtracting 1 to the maximum value worth the maximum size of the frame memory for the rearrangement. The method according to claim 1.   Setting one or more syntax elements indicating a maximum size of a frame memory for rearrangement, wherein the maximum size of the frame memory for rearrangement is an arbitrary frame of the video sequence in encoding order; Represented in terms of the maximum number of frames that can precede and follow the frame in output order; and
  Outputting the one or more syntax elements indicating a maximum size of a frame memory for the rearrangement;
  9. The method of claim 8, further comprising:   The value of the one or more syntax elements indicates an integer worth the constraint of the frame rearrangement waiting time with respect to the maximum size of the frame memory for rearrangement, and the maximum size of the frame memory for rearrangement is: Indicated by different syntax elements,
  The frame rearrangement waiting time constraint is determined by adding an integer worth the frame rearrangement waiting time constraint and subtracting 1 to the maximum value worth the maximum size of the frame memory for the rearrangement. The method according to claim 8.

Images