WO2017219353A1 - Procédés et systèmes de réalisation de commande de vitesse sur la base d'une dynamique de scène et d'une dynamique de canal - Google Patents

Procédés et systèmes de réalisation de commande de vitesse sur la base d'une dynamique de scène et d'une dynamique de canal Download PDF

Info

Publication number
WO2017219353A1
WO2017219353A1 PCT/CN2016/087055 CN2016087055W WO2017219353A1 WO 2017219353 A1 WO2017219353 A1 WO 2017219353A1 CN 2016087055 W CN2016087055 W CN 2016087055W WO 2017219353 A1 WO2017219353 A1 WO 2017219353A1
Authority
WO
WIPO (PCT)
Prior art keywords
rate
quality
video
frame rate
resolution
Prior art date
Application number
PCT/CN2016/087055
Other languages
English (en)
Inventor
Yin Huang
Ruowei WANG
Xiaoyi Zhu
Liang Zhang
Original Assignee
Qualcomm Incorporated
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qualcomm Incorporated filed Critical Qualcomm Incorporated
Priority to PCT/CN2016/087055 priority Critical patent/WO2017219353A1/fr
Publication of WO2017219353A1 publication Critical patent/WO2017219353A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234327—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by decomposing into layers, e.g. base layer and one or more enhancement layers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/115—Selection of the code volume for a coding unit prior to coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation
    • H04N19/126—Details of normalisation or weighting functions, e.g. normalisation matrices or variable uniform quantisers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H04N19/149—Data rate or code amount at the encoder output by estimating the code amount by means of a model, e.g. mathematical model or statistical model
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/164—Feedback from the receiver or from the transmission channel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/587—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Definitions

  • the present disclosure generally relates to rate control for video content, and more specifically to techniques and systems for performing rate control based on scene dynamics and channel dynamics.
  • a remote-controlled device can contain one or more cameras that capture video as the device is moved throughout an environment.
  • Digital video data includes large amounts of data to meet the demands of consumers and video providers.
  • consumers of video data desire video of the utmost quality, with high fidelity, resolutions, frame rates, and the like.
  • the large amount of video data that is required to meet these demands places a burden on communication networks and devices that process and store the video data.
  • Video coding is performed according to one or more video coding standards.
  • video coding standards include MPEG-2, MPEG-4, ITU-T H. 263, ITU-T H. 264/MPEG-4, Part 10, Advanced Video Coding (AVC) , the High Efficiency Video Coding (HEVC) /H. 265 standard, and extensions of such standards.
  • Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, or the like) that take advantage of redundancy present in video images or sequences.
  • prediction methods e.g., inter-prediction, intra-prediction, or the like
  • An important goal of video coding techniques is to compress video data into a form that uses a lower bit rate, while avoiding or minimizing degradations to video quality. With ever-evolving video services becoming available, encoding techniques with better coding efficiency are needed.
  • Rate control techniques can also be used to achieve a target encoding rate. Rate control becomes complicated when remote-controlled devices capture video of dynamically changing scenes and transmit the video data over variable rate channels.
  • rate control can be performed for a remote-controlled device.
  • the techniques and systems can provide an adaptive first person view (FPV) that allows a ground controller to see what the remote-controlled device sees in real-time, and to maneuver the device based on what is seen from the FPV.
  • FPV adaptive first person view
  • the rate control techniques and systems reduce latency between when video is captured by the remote-controlled device and when the video is displayed at a receiving device (e.g., a controller device) .
  • a receiving device e.g., a controller device
  • Such latency is referred to herein as glass-to-glass latency. Reducing the glass-to-glass latency allows the ground controller to effectively operate the remote-controlled device in real-time.
  • Scene dynamics occurs as the remote-controlled device moves through an environment and captures video of numerous scenes with varying complexities.
  • the device may capture video of blue sky in a far field during a first time period or instance, in which case the encoding bitrate is relatively low.
  • the device may capture landmarks in a near field (e.g., while exploring a region of interest (ROI) ) , in which case the encoding bitrate is relatively high.
  • ROI region of interest
  • the varying scene complexities can be taken into account and used to perform rate adaptation, making the rate control techniques robust to scene changes.
  • Channel dynamics occurs, for example, due to the variable range at which remote-controlled devices can travel, with varying channel characteristics occurring as the device travels through an environment (e.g., rate versus range) .
  • the dynamically changing channel characteristics can be taken into account when performing rate estimation.
  • the rate estimation and rate adaptation can be used to perform the rate control.
  • the rate control techniques and systems allow transmitter-only adaptive-FPV over a variable rate link (e.g., a WiFi link) .
  • a variable rate link e.g., a WiFi link
  • low glass-to-glass latency can be achieved with zero receiver buffer and no receiver feedback (e.g., for channel characteristics) .
  • a method of performing rate adaptation based on scene dynamics includes receiving, by a rate adaptation engine of a device, quality feedback from an encoder of the device.
  • the quality feedback includes one or more quality parameters indicating one or more output coding bitrates for one or more coded pictures.
  • the method further includes selecting an updated resolution and frame rate using the one or more quality parameters.
  • the method further includes sending the updated resolution and frame rate to the encoder.
  • an apparatus in another example, includes a memory configured to store video data and a processor.
  • the processor is configured to and can receive quality feedback from the encoder.
  • the quality feedback includes one or more quality parameters indicating one or more output coding bitrates for one or more coded pictures of the encoded video data.
  • the processor is further configured to and can select an updated resolution and frame rate using the one or more quality parameters.
  • the processor is further configured to and can send the updated resolution and frame rate to the encoder.
  • a computer readable medium having stored thereon instructions that when executed by a processor perform a method that includes: receiving quality feedback from an encoder of the device, wherein the quality feedback includes one or more quality parameters indicating one or more output coding bitrates for one or more coded pictures; selecting an updated resolution and frame rate using the one or more quality parameters; and sending the updated resolution and frame rate to the encoder.
  • an apparatus in another example, includes means for receiving quality feedback from an encoder of the device, wherein the quality feedback includes one or more quality parameters indicating one or more output coding bitrates for one or more coded pictures.
  • the apparatus further comprises means for selecting an updated resolution and frame rate using the one or more quality parameters.
  • the apparatus further comprises means for sending the updated resolution and frame rate to the encoder.
  • the method, apparatuses, and computer readable medium described above may further include receiving a corresponding quality parameter for at least a portion of each of the one or more coded pictures.
  • the one or more quality parameters include one or more quantization parameters.
  • selecting the updated resolution and frame rate using the one or more quality parameters includes: comparing the one or more quality parameters to a low quality parameter threshold and a high quality parameter threshold; and selecting the updated resolution and frame rate based on the comparison.
  • the method, apparatuses, and computer readable medium described above may further include comparing a quality parameter to a low quality parameter threshold, and selecting a lower resolution and frame rate when the quality parameter is lower than the low quality threshold.
  • the method, apparatuses, and computer readable medium described above may further include comparing an average of a plurality of the one or more quality parameters to a high quality parameter threshold, and selecting a higher resolution and frame rate when the average is higher than the high quality parameter threshold.
  • a high quality threshold of a lower resolution-frame rate pair is higher than a low quality threshold of a higher resolution-frame rate pair adjacent to the lower resolution-frame rate pair.
  • an estimated target rate and the updated resolution and frame rate are used by the encoder to update a target video coding bitrate.
  • FIG. 1 is a block diagram illustrating an example of an encoding device and a decoding device, in accordance with some embodiments.
  • FIG. 2 is a block diagram illustrating an example of a remote-controlled device and a receiver-controller device, in accordance with some embodiments.
  • FIG. 3 is a block diagram illustrating an example of rate control components of the remote-controlled device, in accordance with some embodiments.
  • FIG. 4 is a diagram illustrating an example of a rate adaptation table, in accordance with some embodiments.
  • FIG. 5 is a diagram illustrating an example of encoding rate variation for constant and variable bit rate systems, in accordance with some embodiments.
  • FIG. 6 is a flowchart illustrating an embodiment of a process of performing rate adaptation, in accordance with some embodiments.
  • FIG. 7 is a flowchart illustrating an embodiment of a process of performing rate control, in accordance with some embodiments.
  • FIG. 8 is a block diagram illustrating an example video encoding device, in accordance with some embodiments.
  • FIG. 9 is a block diagram illustrating an example video decoding device, in accordance with some embodiments.
  • circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail.
  • well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.
  • individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged.
  • a process is terminated when its operations are completed, but could have additional steps not included in a figure.
  • a process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.
  • computer-readable medium includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction (s) and/or data.
  • a computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and/or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD) , flash memory, memory or memory devices.
  • a computer-readable medium may have stored thereon code and/or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements.
  • a code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents.
  • Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.
  • embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof.
  • the program code or code segments to perform the necessary tasks may be stored in a computer-readable or machine-readable medium.
  • a processor may perform the necessary tasks.
  • rate control can be performed for a remote-controlled device that communicates using a variable rate communications link, such as WiFi or other suitable communications link.
  • the systems and methods consider scene dynamics by monitoring the varying complexity of a scene captured by the remote-controlled device, and perform rate adaptation based on the observed scene complexity.
  • rate estimation is performed based on channel dynamics as the remote-controlled device moves throughout the environment. Rate control is performed using the rate adaptation and rate estimation.
  • the systems and methods provide an adaptive first person view (FPV) that reduces glass-to-glass latency between when video of the scene is captured by the remote-controlled device and when the video is displayed at a receiving device (referred to herein as a receiver-controller device) .
  • the adaptive-FPV provided by the system allows a ground controller to remotely observe the scene in real-time as the scene is captured by the remote-controlled device, enabling the ground controller to effectively maneuver the device throughout the environment. Further details regarding the rate control systems and methods are discussed below with respect to FIG. 2-FIG. 6.
  • the video of the various scenes in an environment captured by a remote-controlled device can contain large amounts of data.
  • digital video data can contain vast amounts of data that puts a strain on available transmission and storage capacities.
  • Video coding can be performed to reduce storage and transmission requirements necessary to handle the large amounts of data.
  • Various video coding techniques may be used to compress video data into a form that uses a lower bit rate while maintaining high video quality.
  • FIG. 1 is a block diagram illustrating an example of a system 100 including an encoding device 104 and a decoding device 112.
  • the encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device.
  • the source device and/or the receiving device may include an electronic device, such as a mobile or stationary telephone handset (e.g., smartphone, cellular telephone, or the like) , a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, or any other suitable electronic device.
  • the source device and the receiving device may include one or more wireless transceivers for wireless communications.
  • the coding techniques described herein are applicable to video coding in various multimedia applications, including streaming video transmissions (e.g., over the Internet) , television broadcasts or transmissions, encoding of digital video for storage on a data storage medium, decoding of digital video stored on a data storage medium, or other applications.
  • system 100 can support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and/or video telephony.
  • the encoding device 104 can be used to encode video data using a video coding standard or protocol to generate an encoded video bitstream.
  • Video coding standards include ITU-T H. 261, ISO/IEC MPEG-1 Visual, ITU-T H. 262 or ISO/IEC MPEG-2 Visual, ITU-T H. 263, ISO/IEC MPEG-4 Visual and ITU-T H. 264 (also known as ISO/IEC MPEG-4 AVC) , including its Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions.
  • SVC Scalable Video Coding
  • MVC Multiview Video Coding
  • a more recent video coding standard High-Efficiency Video Coding (HEVC) /H.
  • JCT-VC Joint Collaboration Team on Video Coding
  • VCEG ITU-T Video Coding Experts Group
  • MPEG ISO/IEC Moving Picture Experts Group
  • Various extensions to HEVC deal with multi-layer video coding and are also being developed by the JCT-VC, including the multiview extension to HEVC, called MV-HEVC, and the scalable extension to HEVC, called SHVC, or any other suitable coding protocol.
  • a video source 102 may provide the video data to the encoding device 104.
  • the video source 102 may be part of the source device, or may be part of a device other than the source device.
  • the video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, or the like) , a video archive containing stored video, a video server or content provider providing video data, a video feed interface receiving video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.
  • the video data from the video source 102 may include one or more input pictures or frames.
  • a picture or frame is a still image that is part of a video.
  • the encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to generate an encoded video bitstream.
  • an encoded video bitstream (or “video bitstream” or “bitstream” ) is a series of one or more coded video sequences.
  • a coded video sequence includes a series of access units (AUs) starting with an AU that has a random access point picture in the base layer and with certain properties up to and not including a next AU that has a random access point picture in the base layer and with certain properties.
  • the certain properties of a random access point picture that starts a CVS may include a RASL flag (e.g., NoRaslOutputFlag) equal to 1. Otherwise, a random access point picture (with RASL flag equal to 0) does not start a CVS.
  • An access unit (AU) includes one or more coded pictures and control information corresponding to the coded pictures that share the same output time. Coded slices of pictures are encapsulated in the bitstream level into data units called network abstraction layer (NAL) units.
  • NAL network abstraction layer
  • an HEVC video bitstream may include one or more CVSs including NAL units. Each of the NAL units has a NAL unit header. In one example, the header is one-byte for H.
  • NAL unit header take the designated bits and therefore are visible to all kinds of systems and transport layers, such as Transport Stream, Real-time Transport (RTP) Protocol, File Format, among others.
  • RTP Real-time Transport
  • VCL NAL units Two classes of NAL units exist in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units.
  • a VCL NAL unit includes one slice or slice segment (described below) of coded picture data
  • a non-VCL NAL unit includes control information that relates to one or more coded pictures.
  • An HEVC AU includes VCL NAL units containing coded picture data and non-VCL NAL units (if any) corresponding to the coded picture data.
  • Non-VCL NAL units may contain parameter sets with high-level information relating to the encoded video bitstream, in addition to other information.
  • a parameter set may include a video parameter set (VPS) , a sequence parameter set (SPS) , and a picture parameter set (PPS) .
  • VPS video parameter set
  • SPS sequence parameter set
  • PPS picture parameter set
  • Examples of goals of the parameter sets include bit rate efficiency, error resiliency, and providing systems layer interfaces.
  • Each slice references a single active PPS, SPS, and VPS to access information that the decoding device 112 may use for decoding the slice.
  • An identifier (ID) may be coded for each parameter set, including a VPS ID, an SPS ID, and a PPS ID.
  • An SPS includes an SPS ID and a VPS ID.
  • a PPS includes a PPS ID and an SPS ID.
  • Each slice header includes a PPS ID. Using the IDs, active parameter sets can be identified for a given slice.
  • NAL units may contain a sequence of bits forming a coded representation of the video data (e.g., an encoded video bitstream, a CVS of a bitstream, or the like) , such as coded representations of pictures in a video.
  • the encoder engine 106 generates coded representations of pictures by partitioning each picture into multiple slices.
  • a slice is independent of other slices so that information in the slice is coded without dependency on data from other slices within the same picture.
  • a slice includes one or more slice segments including an independent slice segment and, if present, one or more dependent slice segments that depend on previous slice segments.
  • the slices are then partitioned into coding tree blocks (CTBs) of luma samples and chroma samples.
  • CTBs coding tree blocks
  • a CTB of luma samples and one or more CTBs of chroma samples, along with syntax for the samples, are referred to as a coding tree unit (CTU) .
  • a CTU is the basic processing unit for HEVC encoding.
  • a CTU can be split into multiple coding units (CUs) of varying sizes.
  • a CU contains luma and chroma sample arrays that are referred to as coding blocks (CBs) .
  • the luma and chroma CBs can be further split into prediction blocks (PBs) .
  • a PB is a block of samples of the luma or a chroma component that uses the same motion parameters for inter-prediction.
  • a set of motion parameters is signaled in the bitstream for each PU and is used for inter-prediction of the luma PB and the one or more chroma PBs.
  • a CB can also be partitioned into one or more transform blocks (TBs) .
  • a TB represents a square block of samples of a color component on which the same two-dimensional transform is applied for coding a prediction residual signal.
  • a transform unit (TU) represents the TBs of luma and chroma samples, and corresponding syntax elements.
  • a size of a CU corresponds to a size of the coding node and may be square in shape.
  • a size of a CU may be 8 x 8 samples, 16 x 16 samples, 32 x 32 samples, 64 x 64 samples, or any other appropriate size up to the size of the corresponding CTU.
  • the phrase "N x N" is used herein to refer to pixel dimensions of a video block in terms of vertical and horizontal dimensions (e.g., 8 pixels x 8 pixels) .
  • the pixels in a block may be arranged in rows and columns. In some embodiments, blocks may not have the same number of pixels in a horizontal direction as in a vertical direction.
  • Syntax data associated with a CU may describe, for example, partitioning of the CU into one or more PUs. Partitioning modes may differ between whether the CU is intra-prediction mode encoded or inter-prediction mode encoded. PUs may be partitioned to be non-square in shape. Syntax data associated with a CU may also describe, for example, partitioning of the CU into one or more TUs according to a CTU. A TU can be square or non-square in shape.
  • transformations may be performed using transform units (TUs) .
  • TUs may vary for different CUs.
  • the TUs may be sized based on the size of PUs within a given CU.
  • the TUs may be the same size or smaller than the PUs.
  • residual samples corresponding to a CU may be subdivided into smaller units using a quadtree structure known as residual quad tree (RQT) .
  • Leaf nodes of the RQT may correspond to TUs.
  • Pixel difference values associated with the TUs may be transformed to produce transform coefficients.
  • the transform coefficients may then be quantized by the encoder engine 106.
  • the encoder engine 106 predicts each PU using a prediction mode. The prediction is then subtracted from the original video data to get residuals (described below) .
  • a prediction mode may be signaled inside the bitstream using syntax data.
  • a prediction mode may include intra-prediction (or intra-picture prediction) or inter-prediction (or inter-picture prediction) .
  • intra-prediction or intra-picture prediction
  • inter-prediction or inter-picture prediction
  • each PU is predicted using motion compensation prediction from image data in one or more reference pictures (before or after the current picture in output order) .
  • the decision whether to code a picture area using inter-picture or intra-picture prediction may be made, for example, at the CU level.
  • the one or more slices of a picture are assigned a slice type.
  • Slice types include an I slice, a P slice, and a B slice.
  • An I slice is a slice of a picture that is only coded by Intra prediction, and therefore is independently decodable since the I slice requires only the data within the frame to predict any block of the slice.
  • a P slice (uni-directional predicted frames) is a slice of a picture that may be coded with intra-prediction and uni-directional inter-prediction. Each block within a P slice is either coded with Intra-prediction or inter-prediction. When the inter-prediction applies, the block is only predicted by one reference picture, and therefore reference samples are only from one reference region of one frame.
  • a B slice (bi-directional predictive frames) is a slice of a picture that may be coded with intra-prediction and inter-prediction. A block of a B slice may be bi-directional predicted from two reference pictures, where each picture contributes one reference region and sample sets of the two reference regions are weighted (e.g., with equal weights) to produce the prediction signal of the bi-directional predicted block. As explained above, slices of one picture are independently coded. In some cases, a picture can be coded as just one slice.
  • a PU may include data related to the prediction process. For example, when the PU is encoded using intra-prediction, the PU may include data describing an intra-prediction mode for the PU. As another example, when the PU is encoded using inter-prediction, the PU may include data defining a motion vector for the PU.
  • the data defining the motion vector for a PU may describe, for example, a horizontal component of the motion vector, a vertical component of the motion vector, a resolution for the motion vector (e.g., one-quarter pixel precision or one-eighth pixel precision) , a reference picture to which the motion vector points, and/or a reference picture list (e.g., List 0, List 1, or List C) for the motion vector.
  • the encoding device 104 may then perform transformation and quantization. For example, following prediction, the encoder engine 106 may calculate residual values corresponding to the PU. Residual values may comprise pixel difference values. Any residual data that may be remaining after prediction is performed is transformed using a block transform, which may be based on discrete cosine transform, discrete sine transform, an integer transform, a wavelet transform, or other suitable transform function. In some cases, one or more block transforms (e.g., sizes 32 x 32, 16 x 16, 8 x 8, 4 x 4, or the like) may be applied to residual data in each CU. In some embodiments, a TU may be used for the transform and quantization processes implemented by the encoder engine 106.
  • a TU may be used for the transform and quantization processes implemented by the encoder engine 106.
  • a given CU having one or more PUs may also include one or more TUs.
  • the residual values may be transformed into transform coefficients using the block transforms, and then may be quantized and scanned using TUs to produce serialized transform coefficients for entropy coding.
  • the encoder engine 106 may calculate residual data for the TUs of the CU.
  • the PUs may comprise pixel data in the spatial domain (or pixel domain) .
  • the TUs may comprise coefficients in the transform domain following application of a block transform.
  • the residual data may correspond to pixel difference values between pixels of the unencoded picture and prediction values corresponding to the PUs.
  • Encoder engine 106 may form the TUs including the residual data for the CU, and may then transform the TUs to produce transform coefficients for the CU.
  • the encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value may be rounded down to an m-bit value during quantization, with n being greater than m.
  • the coded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction modes, motion vectors, or the like) , partitioning information, and any other suitable data, such as other syntax data.
  • the different elements of the coded video bitstream may then be entropy encoded by the encoder engine 106.
  • the encoder engine 106 may utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector that can be entropy encoded.
  • encoder engine 106 may perform an adaptive scan. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector) , the encoder engine 106 may entropy encode the vector.
  • the encoder engine 106 may use context adaptive variable length coding, context adaptive binary arithmetic coding, syntax-based context-adaptive binary arithmetic coding, probability interval partitioning entropy coding, or another suitable entropy encoding technique.
  • the output 110 of the encoding device 104 may send or transmit the NAL units making up the encoded video bitstream data over the communications link 120 to the decoding device 112 of the receiving device.
  • the input 114 of the decoding device 112 may receive the NAL units.
  • the communications link 120 may include a channel provided by a wireless network, a wired network, or a combination of a wired and wireless network.
  • a wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFi TM , radio frequency (RF) , UWB, WiFi-Direct, cellular, Long-Term Evolution (LTE) , WiMax TM , or the like) .
  • a wired network may include any wired interface (e.g., fiber, ethernet, powerline ethernet, ethernet over coaxial cable, digital signal line (DSL) , or the like) .
  • the wired and/or wireless networks may be implemented using various equipment, such as base stations, routers, access points, bridges, gateways, switches, or the like.
  • the encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the receiving device.
  • the encoding device 104 may store encoded video bitstream data in storage 108.
  • the output 110 may retrieve the encoded video bitstream data from the encoder engine 106 or from the storage 108.
  • Storage 108 may include any of a variety of distributed or locally accessed data storage media.
  • the storage 108 may include a hard drive, a storage disc, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.
  • the input 114 of the decoding device 112 receives the encoded video bitstream data and may provide the video bitstream data to the decoder engine 116, or to storage 118 for later use by the decoder engine 116.
  • the decoder engine 116 may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting the elements of one or more coded video sequences making up the encoded video data.
  • the decoder engine 116 may then rescale and perform an inverse transform on the encoded video bitstream data. Residual data is then passed to a prediction stage of the decoder engine 116.
  • the decoder engine 116 then predicts a block of pixels (e.g., a PU) . In some examples, the prediction is added to the output of the inverse transform (the residual data) .
  • the decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content.
  • the video destination device 122 may be part of the receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device other than the receiving device.
  • SEI messages can be included in video bitstreams.
  • SEI messages may be used to carry information (e.g., metadata) that is not essential in order to decode the bitstream by the decoding device 112. This information is useful in improving the display or processing of the decoded output (e.g. such information could be used by decoder-side entities to improve the viewability of the content) .
  • the video encoding device 104 and/or the video decoding device 112 may be integrated with an audio encoding device and audio decoding device, respectively.
  • the video encoding device 104 and/or the video decoding device 112 may also include other hardware or software that is necessary to implement the coding techniques described above, such as one or more microprocessors, digital signal processors (DSPs) , application specific integrated circuits (ASICs) , field programmable gate arrays (FPGAs) , discrete logic, software, hardware, firmware or any combinations thereof.
  • DSPs digital signal processors
  • ASICs application specific integrated circuits
  • FPGAs field programmable gate arrays
  • the video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder/decoder (codec) in a respective device.
  • codec combined encoder/decoder
  • Extensions to the HEVC standard include the Multiview Video Coding extension, referred to as MV-HEVC, and the Scalable Video Coding extension, referred to as SHVC.
  • MV-HEVC Multiview Video Coding extension
  • SHVC Scalable Video Coding extension
  • the MV-HEVC and SHVC extensions share the concept of layered coding, with different layers being included in the encoded video bitstream.
  • Each layer in a coded video sequence is addressed by a unique layer identifier (ID) .
  • ID may be present in a header of a NAL unit to identify a layer with which the NAL unit is associated.
  • MV-HEVC different layers can represent different views of the same scene in the video bitstream.
  • SHVC different scalable layers are provided that represent the video bitstream in different spatial resolutions (or picture resolution) or in different reconstruction fidelities.
  • the base layer may conform to a profile of the first version of HEVC, and represents the lowest available layer in a bitstream.
  • the enhancement layers have increased spatial resolution, temporal resolution or frame rate, and/or reconstruction fidelity (or quality) as compared to the base layer.
  • the enhancement layers are hierarchically organized and may (or may not) depend on lower layers.
  • the different layers may be coded using a single standard codec (e. g., all layers are encoded using HEVC, SHVC, or other coding standard) .
  • different layers may be coded using a multi-standard codec.
  • a base layer may be coded using AVC, while one or more enhancement layers may be coded using SHVC and/or MV-HEVC extensions to the HEVC standard.
  • a layer includes a set of VCL NAL units and a corresponding set of non-VCL NAL units.
  • the NAL units are assigned a particular layer ID value.
  • Layers can be hierarchical in the sense that a layer may depend on a lower layer.
  • a layer set refers to a set of layers represented within a bitstream that are self-contained, meaning that the layers within a layer set can depend on other layers in the layer set in the decoding process, but do not depend on any other layers for decoding. Accordingly, the layers in a layer set can form an independent bitstream that can represent video content.
  • the set of layers in a layer set may be obtained from another bitstream by operation of a sub-bitstream extraction process.
  • a layer set may correspond to the set of layers that is to be decoded when a decoder wants to operate according to certain parameters.
  • rate control based on scene dynamics and channel dynamics.
  • Various embodiments include performing rate control for a remote-controlled device that communicates using a variable rate communications link.
  • rate control systems and techniques can be used for other devices, such as automated devices that are self-controlled and maneuvered without user input.
  • FIG. 2 is a block diagram illustrating an example of a remote-controlled device 202 and a receiver-controller device 204.
  • the remote-controlled device 202 can include any device that can be controlled from a remote location using the receiver-controller device 204.
  • the remote-controlled device 202 can include an unmanned aerial vehicle (UAV) or drone and the controller-receiver device 204 can include a ground control station.
  • UAV unmanned aerial vehicle
  • the controller-receiver device 204 can include a ground control station.
  • variable rate link 220 includes a variable rate WiFi link.
  • variable rate link 220 may include one or more channels provided by a wireless network, a wired network, or a combination of a wired and wireless network.
  • a wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., a WiFi network, the Internet or other wide area network, a packet-based network, a radio frequency (RF) network, a UWB network, a WiFi-Direct network, a cellular network, a Long-Term Evolution (LTE) network, a WiMax network, or the like) .
  • a wired network may include any wired interface (e.g., fiber, ethernet, powerline ethernet, ethernet over coaxial cable, digital signal line (DSL) , or the like) .
  • the wired and/or wireless networks may be implemented using various equipment, such as base stations, routers, access points, bridges, gateways, switches, or the like.
  • the remote-controlled device 202 includes an image capture device 208 that can capture video images of scenes within an environment.
  • the image capture device 208 can include any suitable device that can capture video images.
  • the image capture device 208 can include any suitable video camera.
  • the images can be captured as video frames.
  • the captured video frames can be provided to an encoder engine 206.
  • the encoder engine 206 is similar to and can perform the same operations as the encoder engine 106 described with respect to FIG. 1.
  • the encoder engine 206 can include any suitable encoder (or codec) , such as an H.264/AVC encoder (or codec) , H. 265/HEVC encoder (or codec) , an MPEG encoder (or codec, including a H. 261, H. 262, or H.
  • the encoder engine 206 can be referred to as an H. 26x encoder (or codec) , indicating that any standard-based encoder can be used.
  • the remote-controlled device 202 can also capture audio data, which can also be compressed using an audio codec (not shown) . Coded pictures are output as an encoded video bitstream from the encoder engine 206 to the real-time transport protocol (RTP) /RTP control protocol (RTCP) engine 210.
  • RTP real-time transport protocol
  • RTCP RTP control protocol
  • the RTP/RTCP engine 210 is part of the application layer of the Internet Protocol (IP) suite.
  • the RTP/RTCP engine 210 performs RTP/RTCP network protocol operations for delivering the encoded video bitstream (and in some cases, an audio bitstream) over the variable rate link 220.
  • RTP packets can carry the encoded video bitstream (e.g., audio and video)
  • the RTCP packets can be used to monitor transmission statistics and quality of service (QoS) and also to provide synchronization of multiple streams.
  • QoS quality of service
  • the RTP/RTCP packets can be output to the User Datagram Protocol/Internet Protocol (UDP/IP) engine 212.
  • the UDP/IP engine 212 is part of the transport layer of the IP suite and can allow the remote-controlled device 202 to stream the encoded video bitstream to the controller-receiver device 204.
  • an application of the remote-controlled device 202 can send datagram messages to the receiver-controller device 204 over the IP network of the variable rate link 220.
  • the messages can include, for example, the encoded video bitstream (and an audio bitstream in some cases) .
  • the remote-controlled device 202 can also perform other IP suite operations, including Transmission Control Protocol (TCP) operations.
  • TCP Transmission Control Protocol
  • the encoded video bitstream is transmitted to the receiver-controller device 204 using the driver 214.
  • the driver 214 includes a Media Access Control/Physical layer (MAC/PHY) driver.
  • the driver 214 can include a transceiver, a digital signal processor, and other circuitry that is used to transmit (and receive in some cases) the packetized digital video data making up the encoded video bitstream using a certain protocol (e.g., a WiFi protocol using MAC/PHY driver circuitry) .
  • the receiver-controller device 204 receives the encoded video bitstream over the variable rate link 220.
  • a driver 230 receives the encoded video bitstream, and is similar to the driver 214 of the remote-controlled device 202.
  • the driver 230 can include a transceiver, a digital signal processor, and other circuitry that is used to receiver (and transmit in some cases) the packetized digital video data making up the encoded video bitstream using a certain protocol (e.g., a WiFi protocol using MAC/PHY driver circuitry) .
  • the packets are processed by the UDP/IP engine 228 and the RTP/RTCP engine 226.
  • the encoded video bitstream is then provided to the decoder engine 216 for decoding the video pictures.
  • the decoder engine 216 is similar to and can perform the same operations as the decoder engine 116 described with respect to FIG. 1.
  • the remote-controlled device 202 provides a transmitter-only adaptive first person view (FPV) over the variable rate link 220 by performing rate control based on scene dynamics and channel dynamics.
  • a transmitter-only rate control system provides low glass-to-glass latency without requiring any receiver buffer or any receiver feedback (e.g., for channel characteristics) .
  • Rate control is performed using the rate estimation engine 218, the rate adaptation engine 222, and the rate control engine 224, which provide information to the encoder engine 206 to control the bitrate when encoding the video frames.
  • First person view enables a ground controller using the receiver-controller device 204 to see what the remote-controlled device 202 sees (e.g., using the image capture device 208) in real-time and to control the maneuvering of the remote-controlled device 202 based on what is seen from the FPV.
  • FPV becomes even more crucial when the remote-controlled device 202 moves out of the line-of-sight of the ground controller.
  • the required glass-to-glass latency e.g., the camera lens glass to the display glass of the receiver-controller device 204) for providing real-time control is less than a latency limit.
  • a latency limit for providing real-time control of a remote-controlled device can be low enough to provide limited delay (e.g., 100 ms, 150ms, 200ms, or other suitable latency limit) , and any latency over the latency limit can prevent a controller from operating the remote-controlled device in real-time.
  • limited delay e.g. 100 ms, 150ms, 200ms, or other suitable latency limit
  • any latency over the latency limit can prevent a controller from operating the remote-controlled device in real-time.
  • a controller cannot tolerate any loss of the key observations from FPV.
  • a lower resolution can be tolerated.
  • Scene dynamics occurs as the remote-controlled device 202 moves through an environment and captures video of numerous fields of view with varying complexities. For example, for a given resolution, frame rate (e.g., frames per second (fps) ) , and quality, what the remote-controlled device 202 sees through the image capture device 208 is varied as it is maneuvered through the environment.
  • the device 202 may capture video of blue sky in a far field during a first time period or instance, in which case the encoding bitrate is relatively low.
  • the device 202 may capture landmarks in a near field that have much complex detail (e.g., while exploring a region of interest (ROI) ) , in which case the encoding bitrate is relatively high. However, if the device 202 limits the bitrate to a fixed target rate, resolution, and fps, the quality of the video data cannot be guaranteed, resulting in large encoding rate dynamics.
  • ROI region of interest
  • Channel dynamics is, at least in part, due to the variable range at which remote-controlled devices can travel, with varying channel characteristics occurring as a device travels through an environment (e.g., rate versus range) .
  • rate versus range A comparison of rate versus range is provided in Table 1 below, which provides the nominal rate for transmission power of 20dBm with 802.11n WiFi:
  • the transmitter buffer will build up with buffered video data (due to not enough video data being transmitted across the channel) , resulting in larger delay, packet loss (e.g., causing decoding error, jerky motion, freezing of the displayed video, and other effects) , late arrival of video frames being discarded by the player, among others.
  • packet loss e.g., causing decoding error, jerky motion, freezing of the displayed video, and other effects
  • late arrival of video frames being discarded by the player, among others.
  • channel capacity is greater than the video bitrate, the channel is not being sufficiently used and bandwidth is wasted.
  • WiFi and other transmission protocols are designed for indoor low mobility application, and are not optimized for mobile outdoor applications.
  • a remote-controlled device faces multiple outdoor mobile channel loss, including Doppler shift/spread, large delay spread, and other penetrations, which cannot be completely solved by WiFi’s end to end PHY design. Accordingly, the available bandwidth for transporting FPV data varies in a large range, from several 10 Mbps to several hundred kbps, resulting in large channel rate dynamics.
  • rate control focus on how to achieve a target bitrate in a short convergence time for a given resolution and frame rate (fps) .
  • rate control techniques discussed herein adjust encoding parameters (including bitrate, resolution, and frame rate) based on scene complexity and channel conditions.
  • Other solutions may address variable rate video encoding over variable rate channels, but target different applications and have different optimizations.
  • some solutions exist for video telephony over 3G/4G networks.
  • such solutions focus on channel dynamics as opposed to channel and scene dynamics. For instance, most of a captured scene in video telephony is a user’s face, which results in little change in the complexity of the scene (very little scene dynamics) .
  • a resolution, fps of 720p@30fps can guarantee good quality for a simple scene captured by a remote-controlled device.
  • 720p@30fps at the target rate of 2 Mbps produces very low quality, due to a more aggressive compression being required to meet the target rate.
  • scene dynamics and channel dynamics together provide information that can be used to perform quality rate control.
  • the remote-controlled device 202 considers scene dynamics by monitoring the varying complexity of scenes within an environment captured by the image capture device 208.
  • the rate adaptation engine 222 can perform rate adaptation based on the monitored scene complexity. Channel dynamics is also be taken into account by the rate estimation engine 218, which monitors channel conditions as the remote-controlled device 202 moves throughout the environment. The channel conditions are monitored locally at the remote-controlled device 202 without requiring any receiver feedback or any receiver buffering.
  • the rate control engine 224 can perform rate control using information from the rate adaptation engine 222 and the rate estimation engine 218. For example, the rate control engine 224 can determine encoding parameters to use for encoding the captured video frames based on the information from the rate adaptation engine 222 and the rate estimation engine 218.
  • the encoding parameters can include a target encoding rate, a resolution, a frame rate, and any other suitable encoding parameters. Because the adaptive-FPV techniques provided herein include a transmitter-only adaptation, a standard controller-receiver (e.g., a WiFi receiver) and standard video decoder can be used by a ground controller, allowing any device that supports standard FPV to work with the adaptive-FPV systems described herein.
  • a standard controller-receiver e.g., a WiFi receiver
  • standard video decoder can be used by a ground controller, allowing any device that supports standard FPV to work with the adaptive-FPV systems described herein.
  • FIG. 3 illustrates an embodiment of the remote-controlled device 202 that provides transmitter-only adaptive-FPV by performing bitrate control based on scene dynamics and channel dynamics.
  • the adaptive-FPV provides consistent glass-to-glass latency and adaptive video quality even in the presence of the dynamic channel conditions and scene changes.
  • the rate control includes a cross layer optimization that is robustly adaptive to both channel and scene dynamics.
  • the remote-controlled device 202 performs dual-loop rate control with a slower outer loop 334 (e.g., at 200ms) for channel rate estimation and rate adaptation, and a faster inner loop 332 (e.g., at the frame rate (fps) , such as every 33ms, 66ms, 122ms, or the like) for video rate control.
  • the remote-controlled device 202 performs a video quality driven video coding format selection in order to control the bitrate.
  • the adaptive-FPV extends the operation range of remote-controlled devices with robust low FPV latency and little to no loss of frames.
  • the outer loop 334 includes the driver 214, the rate estimation engine 218, and the rate adaptation engine 222.
  • the rate estimation engine 218 estimates the channel dynamics by estimating a target rate r t based on transmission statistics 336 received from the driver 214.
  • the target rate r t can also be referred to as a target encoding bitrate.
  • the target encoding bitrate r t is determined with respect to the channel bitrate so that an encoded video bitstream transmitted to the receiver-controller device 204 can be streamed without delay.
  • the transmission statistics 336 can include MAC/PHY statistics received from a MAC/PHY driver.
  • the transmission statistics 336 received from the driver 214 can include a nominal transmit rate R (e.g., a WiFi PHY nominal transmit rate) derived from a currently used modulation and coding scheme (MCS) .
  • the transmission statistics 336 can also include an acknowledged traffic throughput rate r (the achieved throughput rate) and a buffer size B (e.g., a MAC buffer size) of a transmitter buffer (not shown) at the remote-controlled device 202.
  • the driver 214 can report the acknowledged transmitted bits over time to the rate estimation engine 218, and the rate estimation engine 218 can derive the acknowledged traffic throughput rate r based on the acknowledged transmitted bits.
  • the nominal rate R is an upper limit of the actual acknowledged rate r.
  • the transmission statistics 336 can include all of the R, r, and B statistics. In other cases, the transmission statistics 336 can include one or more of the R, r, and B statistics, in any combination suitable for the particular application.
  • a median filter (not shown) can be applied (e.g., by the rate estimation engine 218 or the driver 214) to provide better transmission statistics 336 (e.g., nominal rate R, throughput rate r, buffer size B, or other statistics) for use by the rate estimation engine 218.
  • transmission statistics 336 e.g., nominal rate R, throughput rate r, buffer size B, or other statistics
  • multiple instances of MCS parameters can be requested.
  • a median of the multiple MCS parameters can then be used as the transmission statistics (R, r, B) by the rate estimation engine 218.
  • the median filter can be beneficial due to inaccurate MCS reporting that can occur from the driver 214.
  • an example of a PHY frame from the driver 214 can include:
  • the rate estimation engine 218 needs the traffic MCS, but may receive from the driver 214 a hybrid traffic MCS or management MCS (e.g., at random occasions) .
  • a hybrid traffic MCS or management MCS e.g., at random occasions.
  • a wrongly reported MCS from a management frame when the traffic MCS is high can negatively affect rate estimation and result in immediate A-FPV performance loss.
  • the median filter can be used to alleviate such a problem.
  • the one outlier of 1 Mbps can be from a management frame. In some examples, a larger M can be used to provide more robustness.
  • constraints can be placed on M.
  • a constraint can include that t cannot be arbitrarily small, and may depend on rate control being at a certain level (e.g., WiFi rate control) .
  • WiFi rate control is approximately 10ms
  • t can be set to 10ms.
  • Another constraint can include that T cannot be arbitrarily large, as a larger T may not converge to channel variation.
  • T can be equal to 200ms or other suitable value, resulting in M being equal to 20.
  • Such a T value is robust against a randomly received MCS (e.g., in a management frame, such as equal to 1Mbps) .
  • the rate estimation engine 218 can estimate the target rate r t using the transmission statistics 336.
  • the rate estimation engine 218 can first estimate, at each outer loop period T ol , the actual link rate of the channel by calculating:
  • r (n) is the current acknowledged throughput rate (at a current time)
  • R (n) is the nominal rate
  • 1/ ⁇ is the loop time constant to control the convergence speed (e.g., a value between 0 and 1, or other suitable value)
  • the time constant ⁇ can be set to any suitable value (e.g., 0.3, 0.4, 0.5., 0.6, or any other suitable value) .
  • the actual link rate is determined because the nominal rate R typically cannot be achieved. In one illustrative example, with a nominal rate R (n) of 30 Mbps and an acknowledged rate r (n) of 22 Mbps, the actual link rate can be estimated as 26 Mbps.
  • the target rate estimation can be modified to take into account transmitter buffer size B by calculating:
  • ⁇ controls the speed to transmit delayed buffer.
  • a larger ⁇ allows the system to clear out the transmitter buffer more quickly, whereas a smaller ⁇ clears the buffer out at a slower rate.
  • the ⁇ parameter can be set to any suitable value (e.g., 2, 3, 4, 5, 6, or any other suitable value) .
  • a moving average filter can be applied to compensate for fluctuations by calculating:
  • ⁇ is the moving average filter coefficient.
  • the ⁇ coefficient can be set to any suitable value (e.g., any value between 0 and 1, such as 0.8, 0.9, or other suitable value) .
  • the rate estimation engine 218 can determine the current video coding target bitrate r t (n) by calculating:
  • the average rate represents the current estimated rate, which can be compared to a previously estimated target rate r t (n-1) to determine the current target rate r t (n) .
  • the current target rate r t (n) is set based on whether the estimated rate is within a certain interval of the previously estimated target rate r t (n-1) .
  • An interval threshold ⁇ is used to control the interval.
  • the interval threshold ⁇ can be set to any suitable value, such as 5%, 10%, 15%, 20%, or other value based on the particular application.
  • the interval is set to 1 Mbps to 1.1 Mbps.
  • the current estimated rate is 1.05 Mbps
  • the current estimated rate falls within the interval of 1 Mbps to 1.1 Mbps (meeting the second condition in Equation 4) .
  • the current target rate r t (n) is set to be the same as the previously estimated target rate r t (n-1) , which is 1 Mbps in this example.
  • the current target rate r t (n) is set to a percentage of the previously estimated target rate r t (n-1) , as determined by the interval threshold ⁇ .
  • the current target rate r t (n) can be set to the high value of the interval. In one illustrative example, with ⁇ set to 10%and a previously estimated target rate r t (n-1) of 1 Mbps, the interval is set to 1 Mbps to 1.1 Mbps.
  • the current estimated rate is 1.2 Mbps
  • the current estimated rate is greater than the interval of 1 Mbps to 1.1 Mbps (meeting the first condition in Equation 4) .
  • the current target rate r t (n) is set to be the high end of the interval, which is 1.1 Mbps in this example.
  • the current estimated rate may be less than the interval (meeting the third condition in Equation 4) , the current target rate r t (n) is set to the current estimated rate
  • the interval is set to 1 Mbps to 1.1 Mbps.
  • the current estimated rate is 0.9 Mbps, the current estimated rate is less than the interval of 1 Mbps to 1.1 Mbps (meeting the third condition in Equation 4) .
  • the current target rate r t (n) is set to the current estimated rate
  • the rate estimation engine 218 provides the estimated target rate r t (denoted as r t (n) in the example equations above) to the rate control engine 224, as shown by communication 338 in FIG. 3.
  • the rate control engine 224 can adjust the output encoding bitrate used by the encoder engine 206 based on the estimated target rate r t .
  • the rate control engine 224 can also determine a quality parameter (e.g., a quantization parameter) for the encoder engine 206 to use for encoding a picture or frame.
  • the encoding parameters are not determined based only on the estimated target rate r t .
  • Video frame resolution includes the number of pixels displayed in each frame or picture, and is defined as the width of the frame (in pixels) versus the height of the frame (in pixels) , denoted as width x height.
  • the frame rate indicates how many unique consecutive images are displayed per second in a video sequence, and is measured in frames per second (fps) .
  • the rate adaptation engine 222 can monitor and adjust to scene dynamics by implementing a (resolution, fps) selection algorithm based on feedback 344 from the video encoder engine 206.
  • the encoder rate control engine 224 of the inner-loop 332 targets the estimated target rate r t received from the rate estimation engine 218, and the quality assessment engine 232 continuously assesses the quality of the encoded video bitstream 342 that is output from the encoder engine 206.
  • the encoder engine 206 attempts to achieve the estimated target rate r t , but may not be able to guarantee the quality.
  • the feedback 344 from the video encoder engine 206 can be used to determine the quality of the encoded video bitstream 342.
  • the quality assessment engine 232 can perform a frame-by-frame quality assessment, determining the quality of each frame or picture output from the encoder engine 206.
  • the feedback 344 can include a picture header (or frame header) sent to the quality assessment engine 232 from the encoder engine 206.
  • the quality assessment engine 232 can monitor the frames or pictures output by the encoder engine 206, and can extract the picture (or frame) header.
  • the quality assessment engine 232 can determine the quality of the picture or frame using information in the header.
  • the header includes a quality parameter indicating the quality of a picture or frame.
  • a quality parameter can be provided for each block (e.g., a macroblock, a coding tree unit (CTU) , a coding tree block (CTB) , or other portion of a picture or frame) of a picture or frame.
  • quality parameters e.g., quantization parameters or other suitable quality parameter
  • delta quality parameters e.g., delta quantization parameters or other suitable quality parameter
  • a lower level e.g., macroblock, CTU, CTB, or the like
  • a coarse granularity e.g., an average quality parameter
  • the frame level such as an average over several microblocks per frame. While the examples and embodiments described below are described using a quality parameter with respect to a frame or picture, one of ordinary skill in the art will appreciate that the discussion also applies to quality parameters for blocks of a picture or frame. Furthermore, the terms picture and frame can be used interchangeably.
  • a quality parameter can include a quantization parameter (QP) .
  • QP quantization parameter
  • the encoder engine 206 can adjust its encoding rate using a QP. For example, as described in detail with respect to FIG. 8 and FIG. 9, residuals are transformed into a spatial frequency domain by an integer transform that approximates a Discrete Cosine Transform (DCT) or a conceptually similar transform.
  • DCT Discrete Cosine Transform
  • the QP determines the step size for associating the transformed coefficients with a finite set of steps. Large values of QP represent big steps that approximate the spatial transform, so that most of the signal can be captured by fewer coefficients. Small values of QP more accurately approximate the spatial frequency spectrum of a picture, which requires more bits.
  • the QP thus indicates a quantization step size for a picture during quantization, and controls how much spatial detail is retained from the captured image after quantization. As the QP value becomes smaller, more detail is retained, leading to better video quality and a higher picture size. As QP increases, more of the detail is aggregated so that the bit rate drops, leading to loss of quality and a smaller picture size.
  • QP values can range between 0 and 51. However, one of ordinary skill in the art will appreciate that other suitable QP ranges can be used.
  • an approximate 4x4 DCT can include:
  • matrix Y includes the transform coefficients resulting from the DCT
  • A is a 4x4 DCT source value matrix
  • a T is the transpose of A
  • C T is the transpose of C
  • S is the scaling part of A that will be further quantized by performing:
  • Q step is the quantization step size with index QP with range of [0, 51] , which can be calculated as:
  • a quality parameter (e.g., a QP or other suitable quality parameter) can reflect both scene complexity and compression loss. For instance, for a given compression loss, a lower QP indicates a more complex scene.
  • a quality parameter for a picture (e.g., a QP or other suitable quality parameter) is indicative of the complexity of the scene in the picture because, for the same output bitrate and format parameters (resolution, frame rate) , the quality of the picture can change based on the complexity of the scene. For example, more information needs to be encoded by the encoder engine 206 (with a given bitrate, resolution, and frame rate) when the scene is more complex, leading to more compression and a lower quality when compared to a scene that is less complex.
  • the QP can provide information that indicates the complexity of the scene.
  • a larger quantization step results in a lower output bit rate and smaller quantization step results in a larger output rate.
  • a larger QP reflects larger quantization step, and smaller QP reflects a smaller quantization step.
  • the adaptation engine 222 can use the QP of a given picture to determine the aggressiveness of the compression performed by the encoder engine 206 for the picture, and thus the complexity of the scene.
  • the rate adaptation engine 222 can determine that compression is more aggressive (more compression in the picture because of the larger quantization step indicated by the higher QP) .
  • the rate adaptation engine 222 can determine that compression is less aggressive (less compression in the picture because of the smaller quantization step indicated by the lower QP) .
  • the aggressiveness of the compression performed by the compression engine 206 is indicative of the complexity of the scene.
  • the encoder engine 206 can operate with a constant output rate (in CBR mode) and with a variable input rate (based on the complexity of the scene) .
  • compression is more aggressive when a scene is more complex, because to fulfill the constant output rate, a more complex scene with a higher input rate means that the encoder engine 206 must perform more aggressive compression (so a higher QP) to meet the target encoding bitrate (or the output bitrate) .
  • a less complex scene and thus a lower input rate means that the encoder engine 206 can perform a less aggressive compression (so a lower QP) to meet the output bitrate.
  • the rate adaptation engine 222 can determine if the encoding format parameters (e.g., the resolution and frame rate) are suitable or not. For example, when a less complex scene is detected in a picture, as indicated by a lower quality parameter for the picture, the rate adaptation engine 222 can afford to go down in resolution and frame rate (due to a less complex scene requiring less resolution and/or frame rate) . As another example, when a quality parameter indicates a scene in a picture is more complex, the rate adaptation engine 222 may go up in resolution in frame rate to provide a better quality picture for display at the receiver-controller device 204. Channel conditions can also affect the quality of a picture.
  • the encoding format parameters e.g., the resolution and frame rate
  • the remote-controlled device 202 may want to lower the encoding format (resolution and frame rate) .
  • the output quality will be higher, and a higher coding format can be used.
  • the rate adaptation engine 222 can compare the quality parameter for a picture to one or more quality parameter thresholds to determine whether the quality parameter is too low or too high, and can go down to a lower resolution, frame rate pair or go up to a higher resolution, frame rate pair based on the comparison. For example, if a quality parameter for a picture is below a quality parameter threshold, the rate adaptation engine 222 can go down in resolution and frame rate. In another example, if a quality parameter for a picture is above a quality parameter threshold, the rate adaptation engine 222 can go up in resolution and frame rate.
  • FIG. 4 illustrates an example of a rate adaptation table 400 showing parameters used by the rate adaptation engine 222 to determine when to go up or down in resolution and frame rate.
  • Various resolution and frame rate pairs are provided, including 720P@30fps, 720P@15fps, VGA@30fps, VGA@15fps, QVGA@30fps, QVGA@15fps, and QVGA@7fps.
  • any other suitable resolution, frame rate combination can be used, such as 1080P@30fps, 1080P@15fps, 4K@30fps, 4K@15fps, or any other suitable resolution, frame rate pair.
  • Table 2 below includes quality parameter thresholds and glass-to-glass (G2G) latencies for various resolution, frame rate pairs. While the quality parameters shown in FIG. 4 and Table 2 include Quantization Parameters (QPs) , one of ordinary skill in the art will appreciate that other suitable quality parameters can be used.
  • QPs Quantization Parameters
  • the rate adaptation engine 222 can perform a picture quality-driven rate adaptation using double thresholding, including a low threshold QP l and a high threshold QP h .
  • Each resolution, frame rate pair has its own corresponding low threshold QP l and high threshold QP h , as shown in Table 4.
  • the rate adaptation engine 222 can immediately go down to a lower resolution, frame rate pair (e.g., a next lower pair) when a received QP is below the lower threshold QP l (and in some cases equal to the lower threshold QP l ) .
  • This example is illustrated by equation QP ⁇ QP l.
  • the rate adaptation engine 222 can immediately go up to a higher resolution, frame rate pair (e.g., a next highest pair) when a received QP is above the higher threshold QP l (and in some cases equal to the high threshold QP h ) .
  • a higher resolution, frame rate pair e.g., a next highest pair
  • QP>QP h Such examples are illustrated by equation QP>QP h .
  • the rate adaptation engine 222 can increase to a higher resolution, frame rate pair using a more conservative approach. For example, the rate adaptation engine 222 can go up to higher resolution, frame rate pair when an average quality is over (and in some cases equal to) the high threshold QP h , as illustrated by equation QP ⁇ QP h .
  • the average quality can be determined over a previous number of pictures, such as for all QPs received since the beginning of a current outer loop period T ol (e.g., 15 QPs for an outer loop period of 1s at a frame rate of 15 fps) , a pre-determined number of QPs (e.g., 2 QPs, 3 QPs, 4 QPs, 5 QPs, or any other suitable number) , or the like.
  • T ol e.g., 15 QPs for an outer loop period of 1s at a frame rate of 15 fps
  • a pre-determined number of QPs e.g., 2 QPs, 3 QPs, 4 QPs, 5 QPs, or any other suitable number
  • the format parameters used to encode a previous picture can be 720P@15fps.
  • the rate adaptation engine 222 can receive a quantization parameter (QP) 346 from the quality assessment engine 232 for the previous picture.
  • QP quantization parameter
  • the QP 346 can be determined from a header of the picture provided to the quality assessment engine 232 from the encoder engine 206.
  • the QP can be 32, indicating low quality and a more complex scene.
  • the rate adaptation engine 222 can compare the received QP 346 to the low threshold QP l and the high threshold QP h for the 720P@15fps pair. As shown in Table 4, the low threshold QP l is 18 and the high threshold QP h is 30.
  • the received QP of 32 is above the high threshold QP h . If an average over a certain number of QPs, including the received QP for the previous frame, is above (and in some cases equal) to the high threshold QP h (indicating a consistency of high complexity) , the rate adaptation engine 222 can go up to the higher resolution, frame rate pair of 720P@30fps.
  • the low and high thresholds have overlapping intervals to avoid a frequent up/down ping pong effect of going back and forth between the resolution, frame rate pairs.
  • a high quality threshold of a lower resolution, frame rate pair is higher than a low quality threshold of a higher resolution, frame rate pair adjacent to the lower resolution-frame rate pair.
  • the low threshold QP l of the resolution, frame rate pair VGA@30fps is lower than the high threshold QP h of the next lowest pair VGA@15fps. Accordingly, there is an overlap between rate ranges of lower pairs and higher pairs, with a rate associated with high threshold of a lower pair being higher than the rate associated with the low threshold of an adjacent higher pair.
  • the rate adaptation engine 222 determines the video encoding format parameters (e.g., format parameters 340, including a resolution, frame rate pair) , they are passed to rate control engine 224 for instructing the encoder engine 206 which format parameters to use.
  • the encoding format parameters can be passed to the inner loop 332 at every outer loop period T ol .
  • the estimated target rate r t is also passed to the rate control engine 224 (e.g., every outer loop period T ol ) .
  • the encoder engine 206 can adapt to the target rate r t based on a rate control algorithm of the rate control engine 224.
  • a rate control algorithm can include a closed loop that starts from a default quality parameter (e.g., a QP or other suitable quality parameter) with respect to given coding parameters (e.g., resolution, fps, aspect ratio, or other coding parameters) .
  • the rate control algorithm can measure the output rate (e.g., per frame or per group of pictures (GOP) ) .
  • the rate control algorithm can determine to increase or decrease its QP or other quality parameter. At convergence, the output rate is adjusted to the specified target rate.
  • the encoder engine 206 can achieve the target bitrate, but the resulting encoding quality of the picture may be degraded in order to meet the target bitrate.
  • the rate adaptation engine 222 can again lower the resolution and frame rate based on the rate adaption table shown in FIG. 4. The scene and channel dynamics can be taken into account as the remote-controlled device 202 travels through an environment, so that a consistent and quality bitstream is provided to the receiver-controller device 204 with minimal glass-to-glass latency.
  • the video coding parameters including the target rate r t and the format parameters 340 (including resolution, fps) , are updated and sent to the inner loop 332 every outer loop period T ol .
  • the outer loop period T ol can include any suitable time period that is slower than the inner loop period T il .
  • the outer loop period T ol can include every 1s, 200ms, 500ms, or other suitable period.
  • the frame level video coding quality parameter is reported to the rate adaptation engine 222 every inner loop period T il .
  • the inner loop period T il can include the frame rate (every video picture) , or other suitable period (e.g., every 2 pictures, every 3 pictures, or other number of pictures) .
  • the coding format parameters are changed at the encoder engine 206 at only each outer loop period T ol .
  • the coding format parameters can be changed more frequently, but less frequently than the inner loop period T il .
  • the dual loop system thus provides a slower outer loop 334 to determine video encoding parameters (including estimating the target rate and determining the format parameters) , and a faster inner loop 332 to adjust the output rate to meet the target rate.
  • the encoder engine 206 can operate in a constant bit rate (CBR) mode in some embodiments.
  • CBR constant bit rate
  • the CBR mode can be used to provide a consistent quality of streaming video to the receiver-controller device 204.
  • the encoder engine can also operate in any of variable bit rate (VBR) , constant frame rate (CFR) , or variable frame rate (VFR) modes.
  • VBR variable bit rate
  • CFR constant frame rate
  • VFR variable frame rate
  • the encoded pictures are output as an encoded video bitstream and sent to the receiver-controller device 204 over link 220.
  • the rate control performed by the remote-controlled device 202 allows the device 202 to provide an adaptive first person view (FPV) and thus to transmit an encoded video bitstream to the receiver-controller device 204 over the variable rate link 220 with minimal or negligible glass-to-glass latency.
  • the adaptive-FPV can allow a ground controller to remotely observe, in real-time, the scenes in an environment captured by the remote-controlled device 202. Such real-time adaptive-FPV enables the ground controller to effectively maneuver the device throughout the environment even when the remote-controlled device 202 is out of view of the ground controller.
  • FIG. 5 illustrates encoding rate variation for constant and variable bit rate systems.
  • the image 506 shows a more complex scene when compared to a more simple scene shown in the image 508.
  • the image 508 has more sky area, which is easier for compression (e.g., using DCT or other compression) as compared to the more complex scene shown in image 506.
  • the chart 502 shows the frame size versus frame index for a constant bit rate (CBR) mode.
  • the chart 504 shows the frame size versus frame index for a variable bit rate (VBR) mode.
  • Each line plotted in the charts 502 and 504 shows a unique rate@fps pair. The lines with larger frame sizes higher on the chart have higher rates, and the lower lines with smaller frame sizes have lower rates.
  • the arrow 510 from image 506 indicates a portion on the plotted line 518 when the rate goes down, and the arrow 512 from image 506 indicates a portion on the plotted line 520 when the rate goes down.
  • the arrow 514 from image 508 indicates a portion on the plotted line 518 when the rate goes up, and the arrow 516 from image 508 indicates a portion on the plotted line 520 when the rate goes up.
  • FIG. 6 illustrates an embodiment of a process 600 of performing rate adaptation based on scene dynamics.
  • the process 600 may be performed by a computing device or an apparatus, such as the rate adaptation engine 222 of the remote-controlled device 202 shown in FIG. 2 and FIG. 3.
  • the computing device or apparatus may include the rate adaptation engine 222 of the remote-controlled device 202 (e.g., an unmanned aerial vehicle, a remote-controlled car, or other such device) , or a processor, microprocessor, microcomputer, or other component of the rate adaptation engine 222 that is configured to carry out the steps of process 600.
  • the computing device or apparatus may include or communicate with a camera configured to capture the video data.
  • the computing device may include a camera device.
  • the camera may be separate from the computing device, such as image capture device 208 being separate from the rate adaptation engine 222.
  • the computing device may further include or communicate with a network interface configured to communicate the video data.
  • the network interface may be configured to communicate Internet Protocol (IP) based data over a network, such as a WiFi network or other suitable network.
  • IP Internet Protocol
  • the process 600 includes receiving, by a rate adaptation engine of a device, quality feedback from an encoder of the device.
  • the quality feedback includes one or more quality parameters indicating one or more output coding bitrates for one or more coded pictures.
  • the encoder can include the encoder engine 206, and the rate adaptation engine can include the rate adaptation engine 222 shown in FIG. 2 and FIG. 3.
  • the process 600 includes receiving a corresponding quality parameter for at least a portion of each of the one or more coded pictures.
  • a quality parameter can be received for each picture, or for each block or other component of each picture (e.g., a macroblock, a coding tree unit (CTU) , a coding tree block (CTB) , or other portion of a picture or frame) .
  • the one or more quality parameters include one or more quantization parameters.
  • a QP for a picture (or each block of a picture) can be derived from a header of the picture.
  • the process 600 includes selecting an updated resolution and frame rate using the one or more quality parameters.
  • selecting the updated resolution and frame rate using the one or more quality parameters includes comparing the one or more quality parameters to a low quality parameter threshold and a high quality parameter threshold. Examples of low quality parameter thresholds and high quality parameter thresholds are shown in FIG. 4 and Table 2, described above.
  • the process 600 can further include selecting the updated resolution and frame rate based on the comparison. For example, the process 600 can include comparing a quality parameter to a low quality parameter threshold, and selecting a lower resolution and frame rate when the quality parameter is lower than the low quality threshold.
  • the process 600 can include comparing an average of a plurality of the one or more quality parameters to a high quality parameter threshold, and selecting a higher resolution and frame rate when the average is higher than the high quality parameter threshold.
  • the average can include an average of quality parameters for a previous number of pictures output from the encoder.
  • a high quality threshold of a lower resolution-frame rate pair is higher than a low quality threshold of a higher resolution-frame rate pair adjacent to the lower resolution-frame rate pair. For example, as shown in FIG. 4, the QP h for the VGA@15fps format parameter pair is higher than the QP l for the VGA@30fps format parameter pair.
  • the process 600 includes sending the updated resolution and frame rate to the encoder.
  • the rate adaptation engine 222 can send a selected resolution and frame rate pair to the rate control engine 224 or directly to the encoder engine 206.
  • An estimated target rate e.g., from rate estimation engine 218, and the updated resolution and frame rate from the rate adaptation engine (e.g., rate adaptation engine 222) are used by the encoder to update a target video coding bitrate.
  • the process 600 of performing scene dynamics can be part of the rate control techniques described herein.
  • a remote-controlled device e.g., device 202
  • FIG. 7 illustrates an embodiment of a process 700 of performing rate control based on scene dynamics and channel dynamics.
  • the process 700 may be performed by a computing device or an apparatus, such as the remote-controlled device 202 shown in FIG. 2 and FIG. 3.
  • the computing device or apparatus may include the remote-controlled device 202 (e.g., an unmanned aerial vehicle, a remote-controlled car, or other such device) , or a processor, microprocessor, microcomputer, or other component of the remote-controlled device 202 that is configured to carry out the steps of process 700.
  • the computing device or apparatus may include or communicate with a camera configured to capture the video data.
  • the computing device may include a camera device, such as the image capture device 208.
  • the camera may be separate from the computing device.
  • the computing device may further include or communicate with a network interface configured to communicate the video data.
  • the network interface may be configured to communicate Internet Protocol (IP) based data over a network, such as a WiFi network or other suitable network.
  • IP Internet Protocol
  • the process 700 includes receiving, by an encoder of a device, an estimated target bitrate from a rate estimation engine of the device.
  • the encoder can include the encoder engine 206 and the device can include the remote-controlled device 202.
  • the estimated target bitrate is determined based on channel transmission statistics, as described above with respect to FIG. 2 and 3.
  • the driver 214 can provide the channel transmission statistics 336 to the rate estimation engine 218, which can estimate the target rate r t and provide the target rate r t to the rate control engine 222 or directly to the encoder engine 206.
  • the process 700 includes sending, by the encoder, quality feedback to a rate adaptation engine of the device.
  • the quality feedback includes one or more quality parameters indicating one or more output coding bitrates for one or more coded pictures, as described above with respect to FIG. 2 and 3.
  • the one or more quality parameters can include one or more quantization parameters (QPs) .
  • the process 700 includes receiving, by the encoder, an updated resolution and frame rate from the rate adaptation engine.
  • the updated resolution and frame rate are determined (by the rate adaptation engine) based on the quality feedback from the encoder, as described above with respect to FIG. 2 and 3.
  • the process 700 includes encoding one or more pictures using the estimated target bitrate and the updated resolution and frame rate.
  • the encoder can output a bitstream including the coded pictures to a receiver-controller device (e.g., receiver-controller device 204) .
  • the encoder can continue to provide quality parameters to the rate adaptation engine as each picture is output.
  • Process 600 and process 700 are illustrated as a logical flow diagram, the operation of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof.
  • the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations.
  • computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types.
  • the order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the processes.
  • process 600 and the process 700 may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executing collectively on one or more processors, by hardware, or combinations thereof.
  • code e.g., executable instructions, one or more computer programs, or one or more applications
  • the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors.
  • the computer-readable or machine-readable storage medium may be non-transitory.
  • a system includes a source device that provides encoded video data to be decoded at a later time by a destination device.
  • the source device provides the video data to destination device via a computer-readable medium.
  • the source device and the destination device may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called “smart” phones, so-called “smart” pads, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming device, or the like.
  • the source device and the destination device may be equipped for wireless communication.
  • the destination device may receive the encoded video data to be decoded via the computer-readable medium.
  • the computer-readable medium may comprise any type of medium or device capable of moving the encoded video data from source device to destination device.
  • computer-readable medium may comprise a communication medium to enable source device to transmit encoded video data directly to destination device in real-time.
  • the encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device.
  • the communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines.
  • the communication medium may form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet.
  • the communication medium may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from source device to destination device.
  • encoded data may be output from output interface to a storage device.
  • encoded data may be accessed from the storage device by input interface.
  • the storage device may include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.
  • the storage device may correspond to a file server or another intermediate storage device that may store the encoded video generated by source device. Destination device may access stored video data from the storage device via streaming or download.
  • the file server may be any type of server capable of storing encoded video data and transmitting that encoded video data to the destination device.
  • Example file servers include a web server (e.g., for a website) , an FTP server, network attached storage (NAS) devices, or a local disk drive.
  • Destination device may access the encoded video data through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection) , a wired connection (e.g., DSL, cable modem, etc. ) , or a combination of both that is suitable for accessing encoded video data stored on a file server.
  • the transmission of encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.
  • the techniques of this disclosure are not necessarily limited to wireless applications or settings.
  • the techniques may be applied to video coding in support of any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions, such as dynamic adaptive streaming over HTTP (DASH) , digital video that is encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.
  • system may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and/or video telephony.
  • the source device includes a video source, a video encoder, and a output interface.
  • the destination device may include an input interface, a video decoder, and a display device.
  • the video encoder of source device may be configured to apply the techniques disclosed herein.
  • a source device and a destination device may include other components or arrangements.
  • the source device may receive video data from an external video source, such as an external camera.
  • the destination device may interface with an external display device, rather than including an integrated display device.
  • the example system above merely one example.
  • Techniques for processing video data in parallel may be performed by any digital video encoding and/or decoding device.
  • the techniques of this disclosure are performed by a video encoding device, the techniques may also be performed by a video encoder/decoder, typically referred to as a “CODEC. ”
  • the techniques of this disclosure may also be performed by a video preprocessor.
  • Source device and destination device are merely examples of such coding devices in which source device generates coded video data for transmission to destination device.
  • the source and destination devices may operate in a substantially symmetrical manner such that each of the devices include video encoding and decoding components.
  • example systems may support one-way or two-way video transmission between video devices, e.g., for video streaming, video playback, video broadcasting, or video telephony.
  • the video source may include a video capture device, such as a video camera, a video archive containing previously captured video, and/or a video feed interface to receive video from a video content provider.
  • the video source may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video.
  • source device and destination device may form so-called camera phones or video phones.
  • the techniques described in this disclosure may be applicable to video coding in general, and may be applied to wireless and/or wired applications.
  • the captured, pre-captured, or computer-generated video may be encoded by the video encoder.
  • the encoded video information may then be output by output interface onto the computer-readable medium.
  • the computer-readable medium may include transient media, such as a wireless broadcast or wired network transmission, or storage media (that is, non-transitory storage media) , such as a hard disk, flash drive, compact disc, digital video disc, Blu-ray disc, or other computer-readable media.
  • a network server (not shown) may receive encoded video data from the source device and provide the encoded video data to the destination device, e.g., via network transmission.
  • a computing device of a medium production facility such as a disc stamping facility, may receive encoded video data from the source device and produce a disc containing the encoded video data. Therefore, the computer-readable medium may be understood to include one or more computer-readable media of various forms, in various examples.
  • the input interface of the destination device receives information from the computer-readable medium.
  • the information of the computer-readable medium may include syntax information defined by the video encoder, which is also used by the video decoder, that includes syntax elements that describe characteristics and/or processing of blocks and other coded units, e.g., group of pictures (GOP) .
  • a display device displays the decoded video data to a user, and may comprise any of a variety of display devices such as a cathode ray tube (CRT) , a liquid crystal display (LCD) , a plasma display, an organic light emitting diode (OLED) display, or another type of display device.
  • CTR cathode ray tube
  • LCD liquid crystal display
  • OLED organic light emitting diode
  • FIG. 8 is a block diagram illustrating an example encoding device 104 that may implement one or more of the techniques described in this disclosure.
  • Encoding device 104 may, for example, generate the syntax structures described herein (e.g., the syntax structures of a VPS, SPS, PPS, or other syntax elements) .
  • Encoding device 104 may perform intra-prediction and inter-prediction coding of video blocks within video slices. As previously described, intra-coding relies, at least in part, on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture.
  • Inter-coding relies, at least in part, on temporal prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a video sequence.
  • Intra-mode may refer to any of several spatial based compression modes.
  • Inter-modes such as uni-directional prediction (P mode) or bi-prediction (B mode) , may refer to any of several temporal-based compression modes.
  • the encoding device 104 includes a partitioning unit 35, prediction processing unit 41, filter unit 63, picture memory 64, summer 50, transform processing unit 52, quantization unit 54, and entropy encoding unit 56.
  • Prediction processing unit 41 includes motion estimation unit 42, motion compensation unit 44, and intra-prediction processing unit 46.
  • encoding device 104 also includes inverse quantization unit 58, inverse transform processing unit 60, and summer 62.
  • Filter unit 63 is intended to represent one or more loop filters such as a deblocking filter, an adaptive loop filter (ALF) , and a sample adaptive offset (SAO) filter. Although filter unit 63 is shown in FIG.
  • filter unit 63 may be implemented as a post loop filter.
  • a post processing device 57 may perform additional processing on encoded video data generated by encoding device 104.
  • the techniques of this disclosure may in some instances be implemented by encoding device 104. In other instances, however, one or more of the techniques of this disclosure may be implemented by post processing device 57.
  • encoding device 104 receives video data, and partitioning unit 35 partitions the data into video blocks.
  • the partitioning may also include partitioning into slices, slice segments, tiles, or other larger units, as wells as video block partitioning, e.g., according to a quadtree structure of LCUs and CUs.
  • Encoding device 104 generally illustrates the components that encode video blocks within a video slice to be encoded.
  • the slice may be divided into multiple video blocks (and possibly into sets of video blocks referred to as tiles) .
  • Prediction processing unit 41 may select one of a plurality of possible coding modes, such as one of a plurality of intra-prediction coding modes or one of a plurality of inter-prediction coding modes, for the current video block based on error results (e.g., coding rate and the level of distortion, or the like) . Prediction processing unit 41 may provide the resulting intra-or inter-coded block to summer 50 to generate residual block data and to summer 62 to reconstruct the encoded block for use as a reference picture.
  • error results e.g., coding rate and the level of distortion, or the like
  • Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be coded to provide spatial compression.
  • Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-predictive coding of the current video block relative to one or more predictive blocks in one or more reference pictures to provide temporal compression.
  • Motion estimation unit 42 may be configured to determine the inter-prediction mode for a video slice according to a predetermined pattern for a video sequence.
  • the predetermined pattern may designate video slices in the sequence as P slices, B slices, or GPB slices.
  • Motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes.
  • Motion estimation, performed by motion estimation unit 42 is the process of generating motion vectors, which estimate motion for video blocks.
  • a motion vector for example, may indicate the displacement of a prediction unit (PU) of a video block within a current video frame or picture relative to a predictive block within a reference picture.
  • PU prediction unit
  • a predictive block is a block that is found to closely match the PU of the video block to be coded in terms of pixel difference, which may be determined by sum of absolute difference (SAD) , sum of square difference (SSD) , or other difference metrics.
  • encoding device 104 may calculate values for sub-integer pixel positions of reference pictures stored in picture memory 64. For example, encoding device 104 may interpolate values of one-quarter pixel positions, one-eighth pixel positions, or other fractional pixel positions of the reference picture. Therefore, motion estimation unit 42 may perform a motion search relative to the full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.
  • Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU to the position of a predictive block of a reference picture.
  • the reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1) , each of which identify one or more reference pictures stored in picture memory 64.
  • Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.
  • Motion compensation performed by motion compensation unit 44, may involve fetching or generating the predictive block based on the motion vector determined by motion estimation, possibly performing interpolations to sub-pixel precision.
  • motion compensation unit 44 may locate the predictive block to which the motion vector points in a reference picture list.
  • Encoding device 104 forms a residual video block by subtracting pixel values of the predictive block from the pixel values of the current video block being coded, forming pixel difference values.
  • the pixel difference values form residual data for the block, and may include both luma and chroma difference components.
  • Summer 50 represents the component or components that perform this subtraction operation.
  • Motion compensation unit 44 may also generate syntax elements associated with the video blocks and the video slice for use by decoding device 112 (FIG. 9) in decoding the video blocks of the video slice.
  • Intra-prediction processing unit 46 may intra-predict a current block, as an alternative to the inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. In particular, intra-prediction processing unit 46 may determine an intra-prediction mode to use to encode a current block. In some examples, intra-prediction processing unit 46 may encode a current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction unit processing 46 (or mode select unit 40, in some examples) may select an appropriate intra-prediction mode to use from the tested modes.
  • intra-prediction processing unit 46 may calculate rate-distortion values using a rate-distortion analysis for the various tested intra-prediction modes, and may select the intra-prediction mode having the best rate-distortion characteristics among the tested modes.
  • Rate-distortion analysis generally determines an amount of distortion (or error) between an encoded block and an original, unencoded block that was encoded to produce the encoded block, as well as a bit rate (that is, a number of bits) used to produce the encoded block.
  • Intra-prediction processing unit 46 may calculate ratios from the distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.
  • intra-prediction processing unit 46 may provide information indicative of the selected intra-prediction mode for the block to entropy encoding unit 56.
  • Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode.
  • Encoding device 104 may include in the transmitted bitstream configuration data definitions of encoding contexts for various blocks as well as indications of a most probable intra-prediction mode, an intra-prediction mode index table, and a modified intra-prediction mode index table to use for each of the contexts.
  • the bitstream configuration data may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables) .
  • encoding device 104 forms a residual video block by subtracting the predictive block from the current video block.
  • the residual video data in the residual block may be included in one or more TUs and applied to transform processing unit 52.
  • Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform.
  • Transform processing unit 52 may convert the residual video data from a pixel domain to a transform domain, such as a frequency domain.
  • Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54.
  • Quantization unit 54 quantizes the transform coefficients to further reduce bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter.
  • quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.
  • entropy encoding unit 56 entropy encodes the quantized transform coefficients.
  • entropy encoding unit 56 may perform context adaptive variable length coding (CAVLC) , context adaptive binary arithmetic coding (CABAC) , syntax-based context-adaptive binary arithmetic coding (SBAC) , probability interval partitioning entropy (PIPE) coding or another entropy encoding technique.
  • CAVLC context adaptive variable length coding
  • CABAC context adaptive binary arithmetic coding
  • SBAC syntax-based context-adaptive binary arithmetic coding
  • PIPE probability interval partitioning entropy
  • the encoded bitstream may be transmitted to decoding device 112, or archived for later transmission or retrieval by decoding device 112.
  • Entropy encoding unit 56 may also entropy encode the motion vectors and the other syntax elements for the current video slice being coded.
  • Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transformation, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture.
  • Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the reference pictures within a reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation.
  • Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in picture memory 64.
  • the reference block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block to inter-predict a block in a subsequent video frame or picture.
  • encoding device 104 of FIG. 8 represents an example of a video encoder configured to generate syntax for a encoded video bitstream.
  • Encoding device 104 may, for example, generate VPS, SPS, and PPS parameter sets as described above.
  • the encoding device 104 may perform any of the techniques described herein, including the processes described above with respect to FIG. 6 and FIG. 7.
  • the techniques of this disclosure have generally been described with respect to encoding device 104, but as mentioned above, some of the techniques of this disclosure may also be implemented by post processing device 57.
  • FIG. 9 is a block diagram illustrating an example decoding device 112.
  • the decoding device 112 includes an entropy decoding unit 80, prediction processing unit 81, inverse quantization unit 86, inverse transform processing unit 88, summer 90, filter unit 91, and picture memory 92.
  • Prediction processing unit 81 includes motion compensation unit 82 and intra prediction processing unit 84.
  • Decoding device 112 may, in some examples, perform a decoding pass generally reciprocal to the encoding pass described with respect to encoding device 104 from FIG. 8.
  • decoding device 112 receives an encoded video bitstream that represents video blocks of an encoded video slice and associated syntax elements sent by encoding device 104 (FIG. 8) .
  • the decoding device 112 may receive the encoded video bitstream from the encoding device 104.
  • the decoding device 112 may receive the encoded video bitstream from a network entity 79, such as a server, a media-aware network element (MANE) , a video editor/splicer, or other such device configured to implement one or more of the techniques described above.
  • Network entity 79 may or may not include encoding device 104.
  • network entity 79 may be implemented by network entity 79 prior to network entity 79 transmitting the encoded video bitstream to decoding device 112.
  • network entity 79 and decoding device 112 may be parts of separate devices, while in other instances, the functionality described with respect to network entity 79 may be performed by the same device that comprises decoding device 112.
  • the entropy decoding unit 80 of decoding device 112 entropy decodes the bitstream to generate quantized coefficients, motion vectors, and other syntax elements. Entropy decoding unit 80 forwards the motion vectors and other syntax elements to prediction processing unit 81. Decoding device 112 may receive the syntax elements at the video slice level and/or the video block level. Entropy decoding unit 80 may process and parse both fixed-length syntax elements and variable-length syntax elements in or more parameter sets, such as a VPS, SPS, and PPS.
  • intra prediction processing unit 84 of prediction processing unit 81 may generate prediction data for a video block of the current video slice based on a signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture.
  • motion compensation unit 82 of prediction processing unit 81 produces predictive blocks for a video block of the current video slice based on the motion vectors and other syntax elements received from entropy decoding unit 80.
  • the predictive blocks may be produced from one of the reference pictures within a reference picture list.
  • Decoding device 112 may construct the reference frame lists, List 0 and List 1, using default construction techniques based on reference pictures stored in picture memory 92.
  • Motion compensation unit 82 determines prediction information for a video block of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to produce the predictive blocks for the current video block being decoded. For example, motion compensation unit 82 may use one or more syntax elements in a parameter set to determine a prediction mode (e.g., intra-or inter-prediction) used to code the video blocks of the video slice, an inter-prediction slice type (e.g., B slice, P slice, or GPB slice) , construction information for one or more reference picture lists for the slice, motion vectors for each inter-encoded video block of the slice, inter-prediction status for each inter-coded video block of the slice, and other information to decode the video blocks in the current video slice.
  • a prediction mode e.g., intra-or inter-prediction
  • an inter-prediction slice type e.g., B slice, P slice, or GPB slice
  • Motion compensation unit 82 may also perform interpolation based on interpolation filters. Motion compensation unit 82 may use interpolation filters as used by encoding device 104 during encoding of the video blocks to calculate interpolated values for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by encoding device 104 from the received syntax elements, and may use the interpolation filters to produce predictive blocks.
  • Inverse quantization unit 86 inverse quantizes, or de-quantizes, the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 80.
  • the inverse quantization process may include use of a quantization parameter calculated by encoding device 104 for each video block in the video slice to determine a degree of quantization and, likewise, a degree of inverse quantization that should be applied.
  • Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform) , an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to produce residual blocks in the pixel domain.
  • decoding device 112 After motion compensation unit 82 generates the predictive block for the current video block based on the motion vectors and other syntax elements, decoding device 112 forms a decoded video block by summing the residual blocks from inverse transform processing unit 88 with the corresponding predictive blocks generated by motion compensation unit 82.
  • Summer 90 represents the component or components that perform this summation operation.
  • loop filters (either in the coding loop or after the coding loop) may also be used to smooth pixel transitions, or to otherwise improve the video quality.
  • Filter unit 91 is intended to represent one or more loop filters such as a deblocking filter, an adaptive loop filter (ALF) , and a sample adaptive offset (SAO) filter. Although filter unit 91 is shown in FIG.
  • filter unit 91 may be implemented as a post loop filter.
  • the decoded video blocks in a given frame or picture are then stored in picture memory 92, which stores reference pictures used for subsequent motion compensation.
  • Picture memory 92 also stores decoded video for later presentation on a display device, such as video destination device 122 shown in FIG. 1.
  • Such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.
  • programmable electronic circuits e.g., microprocessors, or other suitable electronic circuits
  • the techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods described above.
  • the computer-readable data storage medium may form part of a computer program product, which may include packaging materials.
  • the computer-readable medium may comprise memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM) , read-only memory (ROM) , non-volatile random access memory (NVRAM) , electrically erasable programmable read-only memory (EEPROM) , FLASH memory, magnetic or optical data storage media, and the like.
  • RAM random access memory
  • SDRAM synchronous dynamic random access memory
  • ROM read-only memory
  • NVRAM non-volatile random access memory
  • EEPROM electrically erasable programmable read-only memory
  • FLASH memory magnetic or optical data storage media, and the like.
  • the techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer, such as propagated signals or waves.
  • the program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs) , general purpose microprocessors, an application specific integrated circuits (ASICs) , field programmable logic arrays (FPGAs) , or other equivalent integrated or discrete logic circuitry.
  • DSPs digital signal processors
  • ASICs application specific integrated circuits
  • FPGAs field programmable logic arrays
  • a general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
  • a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
  • processor e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
  • the term “processor, ” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.
  • the functionality described herein may be provided within dedicated software modules or hardware modules configured for encoding and decoding, or incorporated in a combined video encoder-decoder (CODEC) .
  • CDEC combined video encoder-decoder

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Algebra (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

La présente invention concerne des techniques et des systèmes de réalisation d'une commande de vitesse sur la base d'une dynamique de scène et d'une dynamique de canal. Par exemple, un procédé de réalisation d'adaptation de vitesse peut être effectué par un moteur d'adaptation de vitesse d'un dispositif. Le procédé peut consister à recevoir une rétroaction de qualité provenant d'un codeur du dispositif. La rétroaction de qualité peut comprendre un ou plusieurs paramètres de qualité qui indiquent un ou plusieurs débits binaires de codage de sortie destinés à une ou plusieurs images codées. Un paramètre de qualité correspondant peut être reçu pour chaque image ou partie de l'image (par exemple, un bloc). Le procédé peut en outre consister à sélectionner une résolution mise à jour et une vitesse de trame à l'aide desdits paramètres de qualité, et à envoyer la résolution mise à jour et la vitesse de trame au codeur.
PCT/CN2016/087055 2016-06-24 2016-06-24 Procédés et systèmes de réalisation de commande de vitesse sur la base d'une dynamique de scène et d'une dynamique de canal WO2017219353A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/087055 WO2017219353A1 (fr) 2016-06-24 2016-06-24 Procédés et systèmes de réalisation de commande de vitesse sur la base d'une dynamique de scène et d'une dynamique de canal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/087055 WO2017219353A1 (fr) 2016-06-24 2016-06-24 Procédés et systèmes de réalisation de commande de vitesse sur la base d'une dynamique de scène et d'une dynamique de canal

Publications (1)

Publication Number Publication Date
WO2017219353A1 true WO2017219353A1 (fr) 2017-12-28

Family

ID=60783791

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/087055 WO2017219353A1 (fr) 2016-06-24 2016-06-24 Procédés et systèmes de réalisation de commande de vitesse sur la base d'une dynamique de scène et d'une dynamique de canal

Country Status (1)

Country Link
WO (1) WO2017219353A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110418177A (zh) * 2019-04-19 2019-11-05 腾讯科技（深圳）有限公司 视频编码方法、装置、设备和存储介质
WO2020036502A1 (fr) * 2018-08-14 2020-02-20 Huawei Technologies Co., Ltd Adaptation basée sur un apprentissage machine de paramètres de codage pour un codage vidéo à l'aide d'une détection de mouvement et d'objet
EP3685574A4 (fr) * 2018-09-30 2020-07-29 SZ DJI Technology Co., Ltd. Appareil et procédé pour une transmission sans fil hiérarchique de vidéos et de graphiques basée sur un prétraitement vidéo
US11363322B2 (en) * 2016-12-01 2022-06-14 Brightcove, Inc. Optimization of encoding profiles for media streaming

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080063054A1 (en) * 2004-01-29 2008-03-13 International Business Machines Corporation System and method for the dynamic resolution change for video encoding
CN102547225A (zh) * 2010-12-29 2012-07-04 中国移动通信集团公司 视频监控场景判别方法及其监控图像编码方法、及装置
US20140044197A1 (en) * 2012-08-10 2014-02-13 Yiting Liao Method and system for content-aware multimedia streaming
CN103974060A (zh) * 2013-01-31 2014-08-06 华为技术有限公司 视频质量调整方法和装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080063054A1 (en) * 2004-01-29 2008-03-13 International Business Machines Corporation System and method for the dynamic resolution change for video encoding
CN102547225A (zh) * 2010-12-29 2012-07-04 中国移动通信集团公司 视频监控场景判别方法及其监控图像编码方法、及装置
US20140044197A1 (en) * 2012-08-10 2014-02-13 Yiting Liao Method and system for content-aware multimedia streaming
CN103974060A (zh) * 2013-01-31 2014-08-06 华为技术有限公司 视频质量调整方法和装置

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11363322B2 (en) * 2016-12-01 2022-06-14 Brightcove, Inc. Optimization of encoding profiles for media streaming
US20230308704A1 (en) * 2016-12-01 2023-09-28 Brightcove, Inc. Optimization of encoding profiles for media streaming
US11979627B2 (en) * 2016-12-01 2024-05-07 Brightcove Inc. Optimization of encoding profiles for media streaming
WO2020036502A1 (fr) * 2018-08-14 2020-02-20 Huawei Technologies Co., Ltd Adaptation basée sur un apprentissage machine de paramètres de codage pour un codage vidéo à l'aide d'une détection de mouvement et d'objet
US11671632B2 (en) 2018-08-14 2023-06-06 Huawei Technologies Co., Ltd. Machine-learning-based adaptation of coding parameters for video encoding using motion and object detection
EP3685574A4 (fr) * 2018-09-30 2020-07-29 SZ DJI Technology Co., Ltd. Appareil et procédé pour une transmission sans fil hiérarchique de vidéos et de graphiques basée sur un prétraitement vidéo
CN110418177A (zh) * 2019-04-19 2019-11-05 腾讯科技（深圳）有限公司 视频编码方法、装置、设备和存储介质
CN110418177B (zh) * 2019-04-19 2021-06-11 腾讯科技（深圳）有限公司 视频编码方法、装置、设备和存储介质

Similar Documents

Publication Publication Date Title
US11159805B2 (en) Methods and systems for generating regional nesting messages for video pictures
US11323712B2 (en) Chroma quantization parameter (QP) offset
CA2952460C (fr) Systemes et procedes pour signaler de facon selective differents nombres de structures de syntaxe d'informations de signal video dans un ensemble de parametres
CN107852498B (zh) 一种编码视频数据的方法、设备及计算机可读媒体
US10148983B2 (en) Methods and systems of improved video stream switching and random access
US10638133B2 (en) Delta quantization parameter (QP) coding options for video
JP2019517205A (ja) ビデオ用のコンテンツカラーボリュームメッセージを生成および処理するための方法およびシステム
TW202121899A (zh) 針對局部雙樹的調色板預測器更新
US20220109865A1 (en) Systems and methods for signaling picture buffer information for intra random access point picture sub-bitstreams in video coding
WO2017219353A1 (fr) Procédés et systèmes de réalisation de commande de vitesse sur la base d'une dynamique de scène et d'une dynamique de canal
TW202203650A (zh) 解碼圖片緩衝器（ｄｐｂ）操作和存取單元定界符（ａｕｄ）
WO2023056159A1 (fr) Décalage de bord pour filtre à décalage adaptatif d'échantillon à composante transversale (ccsao)
JP2023552980A (ja) 高ビット深度ビデオコーディングのライスパラメータ導出のための低複雑度の履歴使用

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16905894

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16905894

Country of ref document: EP

Kind code of ref document: A1