Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/103—Selection of coding mode or of prediction mode
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/124—Quantisation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/136—Incoming video signal characteristics or properties
  • H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
  • H04N19/192—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding the adaptation method, adaptation tool or adaptation type being iterative or recursive
  • H04N19/194—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding the adaptation method, adaptation tool or adaptation type being iterative or recursive involving only two passes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
  • H04N19/436—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation using parallelised computational arrangements
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream

Definitions

• the present invention generally relates to video encoding and more specifically to systems and methods for efficiently encoding multiple streams of video content for adaptive bitrate streaming from a source video stream.
• streaming media describes the playback of media on a playback device, where the media is stored on a server and continuously sent to the playback device over a network during playback.
• the playback device stores a sufficient quantity of media in a buffer at any given time during playback to prevent disruption of playback due to the playback device completing playback of all the buffered media prior to receipt of the next portion of media.
• Adaptive bitrate streaming, or adaptive streaming involves detecting the present streaming conditions (e.g. the user's network bandwidth and CPU capacity) in real time and adjusting the quality of the streamed media accordingly.
• the source media is encoded at multiple bit rates and the playback device or client switches between streaming the different encodings depending on available resources.
• the playback device When a playback device commences adaptive bitrate streaming, the playback device typically starts by requesting portions of media from the lowest bitrate streams (where alternative streams are available). As the playback device downloads the requested media, the playback device can measure the available bandwidth. In the event that there is additional bandwidth available, the playback device can switch to higher bitrate streams.
• the source media is typically stored on a media server as a top level index file pointing to a number of alternate streams that contain the actual video and audio data. Each stream is typically stored in one or more container files.
• Different adaptive streaming solutions typically utilize different index and media containers.
• the Matroska container is a media container developed as an open standard project by the Matroska non-profit organization of Aussonne, France.
• the Matroska container is based upon Extensible Binary Meta Language (EBML), which is a binary derivative of the Extensible Markup Language (XML). Decoding of the Matroska container is supported by many consumer electronics (CE) devices.
• EBML Extensible Binary Meta Language
• XML Extensible Markup Language
• CE consumer electronics
• the DivX Plus file format developed by DivX, LLC of San Diego, California utilizes an extension of the Matroska container format, including elements that are not specified within the Matroska format.
• Other commonly used media container formats are the MP4 container format specified in MPEG-4 Part 14 (i.e. ISO/IEC 14496-14) and the MPEG transport stream (TS) container specified in MPEG-2 Part 1 (i.e. ISO/IEC Standard 13818-1 ).
• the MP4 container format is utilized in IIS Smooth Streaming and Flash Dynamic Streaming.
• the TS container is used in HTTP Adaptive Bitrate Streaming.
• the video in the alternate streams can be encoded pursuant to a variety of block-oriented video compression standards (or codecs) such as High Efficiency Video Coding (HEVC/H.265) specified jointly by the ISO/IEC Moving Picture Experts Group (MPEG) and the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) of Geneva, Switzerland and H.264/MPEG-4 AVC (Advanced Video Coding) standard specified by the ITU-T.
• HEVC/H.265 High Efficiency Video Coding
• MPEG Moving Picture Experts Group
• ITU-T International Telecommunication Union Telecommunication Standardization Sector
• H.264/MPEG-4 AVC Advanced Video Coding
• a source encoder configured to encode source video as a number of alternative video streams includes memory containing a source encoder application, a shared memory, and a parallel processing system configured by the source encoding application to receive multimedia content, where the multimedia content includes source video data having a primary resolution, collect statistics on source video data and write the statistics to shared memory in a first pass through the received multimedia content, where the statistics include complexity measures of blocks of pixels, determine initial encoding information for source video data and write the initial encoding information to shared memory during the first pass through the received multimedia content, encode the source video data in parallel using collected statistics and initial encoding information to produce a plurality of alternative video streams during a second pass through the received multimedia content with a plurality of parallel encoding processes, where the encoding of the source video utilizes additional encoding information, the parallel encoding processes are configured to reuse additional encoding information that has already been determined for
• statistics on source video data includes statistics selected from the group of average quantization parameter, size of header bits, size of texture bits, number of intra blocks, number of inter blocks, and number of skip blocks.
• the parallel processing system being configured to determine initial encoding information for source video data also includes the parallel processing system configured to calculate a frame complexity measure.
• the parallel encoding processes being configured to generate additional encoding information that has not already been determined for a portion of video by another parallel encoding process also includes determining a Coding Tree Unit (CTU) size for encoding a portion of a frame of video in the source video data.
• CTU Coding Tree Unit
• determining a CTU size for encoding a portion of a frame of video in the source video data includes selecting a portion of a frame of video to encode as at least one output CTU in a first output stream, checking if a size has been determined for a similar CTU, selecting a CTU size if a size has not been determined for the similar CTU, selecting a previously determined CTU size that was determined for a second output stream and comparing the resolution of the first output stream to the resolution of the second output stream if a size has been determined for a similar CTU, scaling the CTU size if the resolution of the first output stream is not the same resolution as the second output stream, determining if the selected CTU size is acceptable for the output CTU, selecting a smaller CTU size when the selected CTU size is not acceptable, applying the selected CTU size to the portion of the frame of video if the selected CTU size is acceptable for the output CTU.
• the parallel processing system being configured to determine initial encoding information for source video data also includes the parallel processing system configured to determine a mode distribution for at least one frame of video in at least one of the plurality of alternative video streams.
• the parallel processing system being configured to encode the source video data in parallel using collected statistics and initial encoding information to produce a plurality of alternative video streams also includes the parallel processing system being configured to maintain a count of blocks processed in a frame of video in an alternative video stream, determine a threshold number of blocks based on the mode distribution, and adjust criteria for block type decisions if the count of blocks meets the threshold number of blocks.
• the parallel encoding processes being configured to reuse additional encoding information that has already been determined for a portion of video by another parallel encoding process and stored in the shared memory also includes the parallel encoding processes being configured to determine whether a motion vector exists for a second corresponding block in a second alternative stream when encoding a first block in a video frame in a first alternative stream, determine if the first alternative stream and the second alternative stream are the same resolution, scale the motion vector if the first alternative stream and the second alternative stream are not the same resolution, refine the motion vector, and apply the motion vector in encoding the first block in the video frame in the first alternative stream.
• initial encoding information also includes header size, macroblock size, and the relative proportion of header size to macroblock size.
• initial encoding information also includes hypothetical reference decoder data.
• each of the parallel encoding processes encodes at a different resolution.
• each of the parallel encoding processes encodes one or more alternative video streams and each of the alternative video streams encoded by a parallel encoding process is a different bitrate.
• each of the parallel encoding processes encodes a block from the source video data into each stream in a subset of the plurality of alternative video streams sequentially one after another.
• additional encoding information includes rate distortion information and quantization parameters.
• Still another embodiment again includes receiving multimedia content, where the multimedia content includes source video data having a primary resolution using a source encoder, collecting statistics on source video data and writing the statistics to shared memory in a first pass through the received multimedia content using a source encoder, determining initial encoding information for source video data and writing the initial encoding information to a shared memory during the first pass through the received multimedia content using a source encoder, where the statistics include complexity measures of blocks of pixels, encoding the source video data in parallel using collected statistics, initial encoding information, and additional encoding information to produce a plurality of alternative video streams during a second pass through the received multimedia content with a plurality of parallel encoding processes using a source encoder, where encoding of the source video also includes reusing additional encoding information that has already been determined for a portion of video by another parallel encoding process and stored in the shared memory using at least one of the plurality of parallel encoding processes, and generating additional encoding information that has not already been determined for a portion of video by
• statistics on source video data includes statistics selected from the group consisting of: average quantization parameter, size of header bits, size of texture bits, number of intra blocks, number of inter blocks, and number of skip blocks.
• determining initial encoding information for source video data also includes calculating a frame complexity measure.
• generating additional encoding information that has not already been determined for a portion of video by another parallel encoding process also includes determining a Coding Tree Unit (CTU) size for encoding a portion of a frame of video in the source video data.
• CTU Coding Tree Unit
• determining a CTU size for encoding a portion of a frame of video in the source video data also includes selecting a portion of a frame of video to encode as at least one output CTU in a first output stream, checking if a size has been determined for a similar CTU, selecting a CTU size if a size has not been determined for the similar CTU, selecting a previously determined CTU size that was determined for a second output stream and comparing the resolution of the first output stream to the resolution of the second output stream if a size has been determined for a similar CTU, scaling the CTU size if the resolution of the first output stream is not the same resolution as second output stream, determining if the selected CTU size is acceptable for the output CTU, selecting a smaller CTU size if the selected CTU size is not acceptable, applying the selected CTU size to the portion of the frame of video if the selected CTU size is acceptable for the output CTU.
• determining initial encoding information for source video data also includes determining a mode distribution for at least one frame of video in at least one of the plurality of alternative video streams.
• encoding the source video data in parallel using collected statistics, initial encoding information, and additional encoding information to produce a plurality of alternative video streams also includes maintaining a count of blocks processed in a frame of video in an alternative video stream, determining a threshold number of blocks based on the mode distribution, and adjusting criteria for block type decisions if the count of blocks meets the threshold number of blocks.
• reusing additional encoding information that has already been determined for a portion of video by another parallel encoding process and stored in the shared memory also includes determining whether a motion vector exists for a second corresponding block in a second alternative stream when encoding a first block in a video frame in a first alternative stream, determining if the first alternative stream and the second alternative stream are the same resolution, scaling the motion vector if the first alternative stream and the second alternative stream are not the same resolution, refining the motion vector, and applying the motion vector in encoding the first block in the video frame in the first alternative stream.
• initial encoding information also includes header size, macroblock size, and the relative proportion of header size to macroblock size.
• initial encoding information also includes hypothetical reference decoder data.
• each of the parallel encoding processes encodes at a different resolution.
• each of the parallel encoding processes encodes one or more alternative video streams and each of the alternative video streams encoded by a parallel encoding process is a different bitrate.
• each of the parallel encoding processes encodes a block from the source video data into each stream in a subset of the plurality of alternative video streams sequentially one after another.
• additional encoding information includes rate distortion information and quantization parameters.
• the blocks of pixels are coding tree units.
• the blocks of pixels are coding units.
• the blocks of pixels are transform units.
• the quantization parameters for blocks of pixels are generated using complexity measures for blocks of pixels. [0036] In yet another further embodiment, the quantization parameters for blocks of pixels are generated using previously generated quantization parameters.
• the quantization parameters for blocks of pixels are generated using distortion rate and bit rate.
• FIG. 1 is a system diagram of an adaptive streaming system in accordance with embodiments of the invention.
• FIG. 2 conceptually illustrates a media server configured to encode streams of video data for use in adaptive streaming systems in accordance with embodiments of the invention.
• FIG. 3 is a flow chart illustrating a process for reusing block size information in encoding streams of video in accordance with embodiments of the invention.
• FIG. 4 is a flow chart illustrating a process for adjusting block type decisions in encoding streams of video in accordance with embodiments of the invention.
• FIG. 5 is a flow chart illustrating a process for reusing motion vectors in encoding streams of video in accordance with embodiments of the invention.
• FIG. 6 is a flow chart illustrating a process for sharing statistics and encoding information in the encoding of alternative streams of video data in accordance with embodiments of the invention.
• FIG. 7 is a flow chart illustrating a process for generating a complexity measure for a block of pixels in accordance with embodiments of the invention.
• FIG. 8 is a flow chart illustrating a process for generating a quantization parameter for a block of pixels in accordance with embodiments of the invention.
• FIG. 9 is a flow chart illustrating a process for generating a quantization parameter for a block of pixels using a quantization parameter for a similar block of pixels in accordance with embodiments of the invention.
• FIG. 10A is a graph illustrating a plot of complexity measures of CTU blocks.
• FIG. 10B is a graph illustrating a plot of complexity measures and quantization parameters of CTU blocks.
• encoders can analyze media content for statistics about the content, determine encoding information used to encode the content, and encode the content as multiple video streams at different resolutions and bitrates.
• encoders can analyze media content for statistics about the content, determine encoding information used to encode the content, and encode the content as multiple video streams at different resolutions and bitrates.
• present invention is described below with respect to adaptive streaming systems and block-based video encoding techniques such as HEVC/H.265 and H.265 AVC, the systems and methods described are equally applicable in conventional streaming systems where different streams of video data are selected based upon a network client's connection quality and video encoding techniques that are not block-based.
• multimedia content is encoded as a set of alternative streams of video data. Because each alternative stream of video data is encoded using the same source multimedia content, similar encoding information is determined in the encoding of each alternative stream of video data. Encoding information can include, but is not limited to, frame complexity measure, selection of block sizes, block mode distribution and motion estimation results. Systems and methods in accordance with many embodiments of the invention reuse encoding information determined in the encoding of one alternative stream of video data in the encoding of at least one other alternative stream of video data. By reusing encoding information in the encoding of several alternative streams of video data, significant improvements in the encoding of the alternative streams of video data can be achieved and particularly significant time savings may be realized in accordance with embodiments of the invention.
• Adaptive streaming systems are configured to stream multimedia content encoded at different maximum bitrates and resolutions over a network, such as the Internet. Adaptive streaming systems stream the highest quality multimedia content, which can be supported based upon current streaming conditions. Multimedia content typically includes video and audio data, subtitles, and other related metadata. In order to provide the highest quality video experience independent of the network data rate, adaptive streaming systems are configured to switch between the available sources of video data throughout the delivery of the video data according to a variety of factors, including, but not limited to, the available network data rate and video decoder performance. When streaming conditions deteriorate, an adaptive streaming system typically attempts to switch to multimedia streams encoded at lower maximum bitrates.
• a source encoder can be configured to encode a plurality of alternative streams of video data from a source video contained in a piece of multimedia content.
• Systems and method for encoding a source video for use in adaptive streaming systems are disclosed in U.S. Patent Application No. 13/221 ,794, entitled “Systems and Methods for Encoding Source Media in Matroska Container Files for Adaptive Bitrate Streaming Using Hypertext Transfer Protocol" to Braness et al., filed August 30, 201 1 , the entirety of which is incorporated by reference.
• a source encoder may be implemented using a media source and/or a media server.
• alternative streams of video data based on the same source video contain similar content, therefore, statistics determined from the source content and encoding information determined for one alternative stream of video data may be used in the encoding of one or more of the other alternative streams of video data.
• a set of alternative streams of video data based on the same source video may contain video data at the same resolution but differing bitrates.
• the motion estimation results calculated for the encoding of a particular alternative stream of video data can be reused amongst other alternative streams of video data.
• a variety of statistics and encoding information determined in the encoding of alternative streams of video data may be reused amongst the alternative streams of video data.
• an adaptive streaming system includes a source encoding server that performs the encoding of multiple streams of video from source media.
• An adaptive streaming system in accordance with embodiments of the invention is illustrated in FIG. 1 .
• the illustrated adaptive streaming system 10 includes a source encoding server 12 configured to encode source media as a number of alternative streams.
• the source media may be stored on the encoding server 12 or retrieved from a media source 13.
• the source encoding server 12 generates container files containing the encoded streams, at least a plurality of which are alternative streams of encoded video.
• the encoding server makes a first pass to collect statistics on the content at each output resolution and a second pass to encode the content into multiple output streams, where the streams can have various resolutions and bitrates.
• the first pass is completed before the second pass begins.
• the second pass can commence before the first pass is completed.
• computational processes for the first and second passes can run simultaneously where frames are processed by the first pass process(es) before being processed by the second pass process(es).
• These files are uploaded to a content server 14, which can be an HTTP server.
• a variety of playback devices 18, 20, and 22 can then request portions of the encoded streams from the content server 14 via a network 16 such as the Internet.
• the adaptive bitrate streaming system includes one or more source encoders capable of encoding a source streaming of video content into alternative streams of encoded video having different resolutions and/or bitrates.
• the source encoder can be implemented using any device capable of encoding streams of multimedia, where the streams are encoded at different resolutions, sampling rates, and/or maximum bitrates.
• FIG. 2 The basic architecture of an adaptive streaming system source encoder in accordance with an embodiment of the invention is illustrated in FIG. 2.
• the source encoder 200 includes a processor 210 in communication with memory 230 and network interface 240.
• the volatile memory 230 includes a source encoding application 250.
• the processor is configured by the source encoding application 250 to encode a plurality of streams of video data from source video data 260, which is also in volatile memory.
• the source video data 260 may already be present in memory or may be received via the network interface 240.
• the source encoder includes multiple processors and the encoding process can be distributed among the multiple processors.
• the source encoding application 250 can launch multiple processes that execute on one or more processors where each process is an encoder controller 270 that encodes one or more output streams.
• each encoder controller encodes multiple output streams at the same resolution and at different bitrates.
• an encoder controller for each of three output resolutions is launched to be executed on one or more processors, where the output resolutions are 768 x 432, 1280 x 720, and 1920 x 1080.
• an encoder controller 270 encodes 768 x 432 outputs streams at two different bitrates, an encoder controller 280 at 1280 x 720 at three different bitrates, and an encoder controller 290 at 1920 x 1080 at three different bitrates.
• Encoder controllers 270, 280, and 290 typically reside in memory 230 when they are executing.
• encoder controllers 270, 280, and 290 have shared data buffers 295 in memory for data exchange of statistics, encoding information, and other information between the controllers.
• FIG. 2 Although a specific architecture for a source encoder is illustrated in FIG. 2, any of a variety of architectures including architectures where the video encoder 250 is located on disk or some other form of storage and is loaded into memory 230 at runtime can be utilized to encode multimedia content in accordance with embodiments of the invention.
• Systems and methods for the reuse of statistics and encoding information in the encoding of alternative streams of video data in accordance with embodiments of the invention are discussed further below.
• statistics and encoding information are determined for a piece of media content before encoding the content into multiple output streams.
• the statistics and encoding information can be saved and shared between encoding processes to accelerate decision making about how to encode the content.
• Statistics to collect in a first pass can include (but are not limited to): average quantization parameter, size of header bits, size of texture bits, number of intra macroblocks/CTUs, number of inter macroblocks/CTUs, number of skip macroblocks/CTUs.
• Encoding information can include (but is not limited to): frame complexity measures, coding tree unit (CTU) structure, mode distribution, and motion information. The collection and use of encoding information in the encoding of alternative streams of video in accordance with embodiments of the invention are discussed below.
• a frame complexity measure can be assigned to each frame of the content.
• the frame complexity measure represents the level of complexity of visual information in a frame and thereby an indication of the data that will result in the output content stream from encoding that frame (i.e., in bits).
• Algorithms including those known in the art can be utilized in accordance with embodiments of the invention to calculate a frame complexity measure.
• Such algorithms may consider inter (i.e., between) and intra (i.e., within) frame measurements such as deviation of values such as color and brightness from the average values across pixels in a frame or corresponding pixels in multiple frames, and/or similarities between pixels and/or blocks of pixels in frames with those in previous frames.
• algorithms may calculate measurements similar to those used in encoding content in conducting motion estimation.
• the frame complexity measure can be used in choosing the parameters with which to encode a frame from an input stream into a frame in an output stream.
• a frame complexity measure is assigned to each frame of the content and is expressed as an integer, where a greater value indicates greater complexity within the frame.
• a frame complexity measure is assigned to a frame in an input stream and the same measure is used to choose the parameters for encoding the corresponding frames in a plurality of alternative output streams.
• Bitrate control algorithms including those known in the art can be utilized in accordance with embodiments of the invention to determine how many bits to allocate to each encoded frame of video and buffer levels of the output stream and to choose the quantization levels to apply when encoding each frame of video.
• these algorithms can utilize the frame complexity measure that was computed once for an input content stream and reuse the measure to determine the quantization level for multiple output streams encoded from that input stream. Block Size Decisions
• pixels are taken as partitions or "blocks" in the encoding process.
• the block size is variable where partitions in a frame can have different sizes.
• the HEVC standard uses partitions known as Coding Tree Units (CTUs) where the block size of a CTU is typically 64 x 64, 32 x 32, 16 x 16, or 8 x 8 pixels.
• a frame of video can include a mixture of CTUs of different sizes and arrangements.
• block sizes of CTUs (or other "block" partitions of pixels as per the relevant video compression standard) in a frame are chosen based on efficiency and the resulting size in bits of an encoded frame.
• An encoder in accordance with embodiments of the invention can accelerate decisions on block sizes to use in encoding frames of content for an output stream when it has already made similar decisions for corresponding portions of frames in another alternative output stream.
• An encoder typically chooses block sizes based at least in part on the size (in bits) of the resulting encoded output frame while maintaining image quality of the frame.
• a smaller size of output frames i.e., more efficient
• the encoder can try using larger block sizes in encoding a portion of a frame and, if quality or accuracy of the resulting image does not meet a threshold, try progressively smaller block sizes for that portion of the frame. For example, a 32 x 32 pixel portion of a frame can be tested to encode as four 16 x 16 pixel CTUs. If any of the 16 x 16 pixel CTUs is not desirable, it can be further divided into four 8 x 8 pixel CTUs. In other video standards, the larger and smaller block sizes specified in a particular standard can be tested in similar fashion.
• certain data can be retained and shared between encoder controllers that are encoding alternative output streams from the same input stream.
• the shared data can include block sizes that were tested on portions of frames. For example, if an encoder controller has chosen to process a 32 x 32 pixel portion of the frame as four 16 x 16 CTUs (or any other combination of CTU sizes), when the encoder controller or another encoder controller encodes the same portion of the frame at the same resolution but a different bitrate it can skip the previously made decision(s) and use the same size CTUs and/or test other CTU sizes.
• an encoder controller encodes the same portion of a frame at a higher bitrate, it can skip the previous test/decision not to encode with 32 x 32 pixel CTUs and simply use 16 x 16 CTUs and/or test if 8 x 8 pixel CTUs would achieve the same or better quality while meeting the target bitrate.
• Past CTU size choices can also be useful in picking CTU sizes in a stream of a different resolution. For example, suppose a CTU size of 32 x 32 was tested and rejected for a portion of a frame in a stream with resolution 1920 x 1080 in favor of 16 x 16 or smaller partitions. The same portion of the frame in a stream of resolution 3840 x 2160 would cover a size of 64 x 64. The encoder can skip checking a CTU size of 64 x 64 for that portion in the higher resolution frame because the size of 32 x 32 was rejected in the lower resolution stream. The encoder can start by testing CTU sizes of 32 x 32 and smaller.
• FIG. 3 A process for reusing block size information in encoding a stream of video in accordance with embodiments of the invention is illustrated in FIG. 3.
• the process includes checking (310) if a size has been determined for a similar CTU (i.e., a corresponding CTU in a corresponding frame in another stream). If not then a CTU size is chosen (312) to be tested, typically starting from a larger size. If a size has been determined for a similar CTU, the resolution of the source stream of the similar CTU is compared (312) to the current stream. If the resolution is different, the CTU size is scaled (314) to the current stream.
• a similar CTU i.e., a corresponding CTU in a corresponding frame in another stream.
• the predetermined CTU size (if predetermined) or the chosen CTU size (if not predetermined) is tested (316) for the current CTU being encoded. If the size is not acceptable, a smaller CTU size is attempted (318). If the size is acceptable, the CTU size is applied (320) to the current CTU.
• Mode distribution refers to the overall ratio of intra, inter, and skip macro blocks (or other "block" partitions of pixels as per the relevant video compression standard) within a frame of video.
• Predicted frames such as P frames and B frames can contain intra (encoded only with information within the frame) and/or inter (encoded with reference to information in another frame) frame macroblocks.
• block pixel partitions may be referred to variously as macroblocks or Coding Tree Units (CTUs).
• Mode distribution can refer to block types whether the block itself is called a macroblock or CTU.
• Tests and/or algorithms can be utilized to analyze each macroblock in a frame to see which type of block is the most efficient at encoding that block while maintaining image quality.
• a measure of encoding efficiency can be the number of bits needed to encode a particular macroblock.
• the corresponding frame p1 in a first alternate stream at the same resolution and a different bitrate should have a similar distribution of intra, inter, and skip blocks, i.e., a target mode distribution.
• the corresponding frame p2 in a second alternate stream at a different resolution and a different bitrate ideally will also have a similar distribution of block types.
• the mode distribution can be used to accelerate decisions made by the encoder as to what block type to use when processing blocks in a frame.
• blocks can be chosen to be a certain type (e.g., intra, inter, skip) in order to bring the resulting mode distribution after encoding those blocks closer to the target mode distribution while maintaining good quality and integrity of the image.
• the decisions in choosing block types may differ between streams at different bitrates because more complexity (e.g., more intra coded blocks) can be accommodated at higher bitrates.
• FIG. 4 A process for adjusting block type decisions in accordance with embodiments of the invention is illustrated in FIG. 4.
• the process 400 can be utilized when blocks in a frame are being encoded or when the frame is being analyzed and block types are being chosen without being encoded.
• the process includes maintaining (410) a count of blocks that have been processed (e.g., analyzed or encoded). If a threshold number of blocks has been reached, or alternatively a proportion of the total blocks in the frame, the criteria for choosing block types can be adjusted (414) more in favor of choosing block types that would maintain a target mode distribution of blocks in the frame. If the end of the frame has not been reached (416), processing continues for blocks in the frame and updating (410) a count of the blocks that have been processed.
• predicted frames such as P frames and B frames draw information from a reference frame (which can be an I, P, or B frame) and contain information such as motion vectors and residual data used in motion estimation.
• a motion vector for a block of pixels e.g., a macroblock, CTU, or similar partition
• Residual data describes any differences between the block of pixels and the reference block of pixels.
• motion estimation information such as motion vectors can be reused from one output stream to another output stream at a different resolution and/or bitrate.
• motion vectors are generated for at least one output stream at a given resolution and bitrate. The motion vectors are then be applied and refined when encoding at least one other output stream at the same resolution at a different bitrate. In further embodiments of the invention, motion vectors that are generated for an output stream are applied and refined when encoding at least one other output stream at a higher resolution. The motion vectors are scaled proportionally to the difference in resolution between the first stream and second stream and then refined to take advantage of the higher precision available in the higher resolution.
• motion vectors generated for a stream of resolution 768 x 432 can be used in encoding a stream of resolution 1280 x 720 by multiplying the horizontal component of a vector by 1280/768 and the vertical component of a vector by 720/432.
• the multipliers 1280/768 and 720/432 can be considered scale factors for that change in resolution.
• an encoder may determine that the encoding cost of a motion vector and residual data for a macroblock is very low and may skip checking the encoding cost for that macroblock in another output stream. In this way, motion vectors can be reused across multiple output streams and motion estimation is accelerated by not having to generate new motion vectors.
• an encoder can check (510) if a motion vector was already created for a corresponding macroblock/CTU in a different stream of video that can be reused in encoding the current frame. If not, a new motion vector is created and stored (512). If a motion vector exists, the encoder can check (514) if the other stream of video is at the same resolution as the current stream. If it is the same resolution, the motion vector can be applied directly (516). The motion vector can also be refined with a search, such as with a small diamond search over one or two neighboring pixels or with sub-pixel refinement. Various other search patterns can alternatively be utilized to refine the motion vector. If it is a different resolution, the motion vector can be scaled (518) and refined (520) to the current resolution before being applied.
• Encoding content using a two-pass process can provide efficiencies through the sharing and reuse of information.
• an encoder receives content, analyzes the content in a first pass and encodes the content in a second pass. Statistics and other information collected in the first pass can be used to accelerate encoding in the second pass and statistics and encoding information can be shared between encoding processes in the second pass.
• the first pass is completed before the second pass begins.
• the second pass can commence before the first pass is completed.
• computational processes for the first and second passes can run simultaneously where frames are processed by the first pass process(es) before being processed by the second pass process(es).
• the encoding process 600 includes receiving media content (610).
• the content can be contained in a single input file (e.g., a multimedia file or container format file) or collection of media files.
• the content may also be an input stream of video received by the encoder.
• the encoding process is implemented on an encoding server as an encoding application and disk read/write operations are reduced because each input frame only needs to be read once.
• the encoding process also includes making a first pass to collect statistics on the media stream (612).
• the first pass is intended to make an initial analysis of the content in the source input video stream and collect statistics for each resolution to be encoded.
• the input stream is converted into a stream for each desired output resolution and statistics are collected on each stream.
• the conversion can include color conversion (e.g., RGB to YUV etc., ITU601 to 709, or format specific conversions) and resizing the dimensions/resolution of each frame.
• Encoding information such as a frame complexity measure can be assigned to each frame in a stream based on the level of complexity of the image data in the frame using techniques as discussed further above.
• a mode distribution can be determined for each frame as well. Complexity measures and mode distributions can be used as input to the encoding of the final streams.
• Encoding information can include the size of header data and the size of macroblock data to be used in encoding the streams. Given header size, the encoder can estimate how many bits are allocated to overhead data and how many bits are available to encode the content. Macroblock size can also contribute to better estimates of bit allocation when encoding the content. For effective rate control, the relative proportion of header size to macroblock size should be consistent across output streams. Header size, macroblock size, and/or the relative proportion of header size to macroblock size can be provided as encoding parameters to the encoding process. [0082] Encoding information can also include hypothetical reference decoder (HRD) data. An HRD model is specified in some video compression standards to aid in calculating frame sizes and buffer fullness. The HRD reproduces decoder behavior on an encoder to avoid buffer overflow in a playback device buffer. HRD data can be determined in a first analysis pass and saved in shared buffers or local caches for use in a second encoding pass.
• HRD hypothetical reference decoder
• Statistics to collect in a first pass can include (but are not limited to): average quantization parameter, size of header bits, size of texture bits, number of intra macroblocks/CTUs, number of inter macroblocks/CTUs, number of skip macroblocks/CTUs.
• statistics can be cached internally (e.g., output to single file) and/or stored in shared buffers in memory that are accessible to each encoder controller when encoding in the second pass.
• the content is encoded (616) into output streams.
• a separate thread or process is launched (614) by the encoding application for each output resolution where each thread or process is referred to as an encoder controller.
• each encoder controller encodes multiple output streams at the same resolution but at different bitrates.
• an encoding server has multiple processors and the encoder controllers may each run on different processors or otherwise distributed across the processors.
• Output streams may be encoded at a constant bit rate (CBR) or variable bit rate (VBR).
• the encoder controllers can run simultaneously using information and statistics stored in internal caches and/or shared buffers in memory.
• An encoder controller can check (620) the memory for whether encoding information exists for the blocks and/or frames it is encoding and utilize (622) that information in processing the current block or frame.
• different types of encoding information can be used to allow the encoder to skip decisions or can be used as a starting point to determine encoding information for processing the current block or frame. If the information does not yet exist, it can be created (624) and encoding (616) of the stream continues.
• Shared information can include motion estimation information such as motion vectors.
• Motion vectors generated for a stream may be reused in encoding another stream at the same resolution or at a different resolution.
• a motion vector reused to encode a stream at a higher resolution may be scaled to compensate for the difference in resolution and refined.
• Some statistics may be specific to the resolution of the stream being encoded.
• CTU size choices at a certain resolution and bitrate may be retained and applied to encoding the same resolution at a higher bitrate.
• an encoder controller When an encoder controller has made a decision for CTU size in a portion of a frame, it may skip that decision when encoding the same portion of the frame at a higher bitrate and reuse the CTU size or try smaller CTU sizes.
• a CTU in a given portion of a frame is encoded sequentially by the same encoder controller for a plurality of output streams at different bitrates.
• an encoder controller encodes a CTU for an output stream, if it encodes the CTU for a different output stream soon thereafter, the data required to encode that CTU is likely to still be in the processor cache or other local memory.
• an encoder controller encodes a CTU for each output stream sequentially one after another. The corresponding CTUs for each output stream may be encoded using different quantization parameters for streams at different bitrates.
• Encoding with block-oriented video compression typically involves factoring in a quantization parameter that contributes to the compression of the image information within a frame.
• the quantization parameter for a frame is typically selected based upon factors such as a frame complexity measure and mode distribution of the frame and is selected so that the encoded frame is a target size (in bits) while maintaining the best image quality possible.
• values for a frame complexity measure and mode distribution can be determined for a frame of an input stream and utilized in processing corresponding output frames in multiple output streams.
• Quantization parameter(s) from a previous frame can be used as a starting value and refined using a frame complexity measure and mode distribution for the current frame.
• quantization parameters can be increased for that frame because a less complex image can be compressed more without severe loss of fidelity.
• Quantization parameters can be decreased when encoding the same frame for a higher bitrate stream because of the higher capacity for data in the higher bitrate.
• quantization parameters can be independently determined for corresponding frames in each output stream using the same frame complexity measure and mode distribution while adjusting for the target bitrate of each output stream. Quantization parameters determined for CTUs in a frame in lower bitrate output stream can be referred to and decreased for the corresponding CTUs in a higher bitrate output stream.
• Encoders in accordance with embodiments of the invention can calculate rate distortion information when encoding output streams and reuse the rate distortion information to choose quantization parameters when encoding an output stream at a higher bitrate.
• each frame in each output stream is encoded such that its mode distribution is approximately equal to a target mode distribution determined using the source frame from the input stream.
• Other types of shared information that can be accessible and used by different encoder controllers further above.
• one or more frames of video are partitioned into blocks that encode the motion prediction residual. These blocks are referred to as Transform Units (TU) in the HEVC standard and can have sizes of 4 x 4, 8 x 8, 16 x 16, or 32 x 32.
• TU Transform Units
• Encoders in accordance with embodiments of the invention can use quantization parameters in encoding CTUs, TUs, coding units (CUs) and/or other types of encoding blocks that are defined by the standard or video codec the encoder is configured for.
• the process of choosing quantization parameters for blocks in a frame of video based on statistics of a stream and/or previously calculated quantization parameters can be referred to as adaptive quantization.
• Quantization parameters for blocks of pixels can be calculated in the first pass and/or second pass based on complexity measures for those blocks of pixels (similar to the frame complexity measures discussed further above) and/or quantization parameters already calculated for similar blocks of pixels.
• Quantization parameters and/or complexity measures that are calculated in the first pass can be stored, for example, in shared buffers that are accessible by encoder controllers and utilized in the second pass. For example, values that are calculated for the input stream at one resolution can be utilized when encoding output streams at the same resolution at different bit rates.
• a statistic that is calculated can include a complexity measure for a block of pixels in a frame within the input video stream.
• a complexity measure for a block of pixels can be stored and used in calculating a quantization parameter for encoding the block in the second pass.
• a process for generating a complexity measure in accordance with embodiments of the invention is illustrated in FIG. 7. The process includes selecting 710 a block of pixels in a frame of video. The process analyzes 712 the complexity of the block.
• a complexity measure can be calculated from, for example in the AVC standard, a histogram of luminance and/or contrast level distribution.
• FIG. 10A illustrates a plot of complexity measures of CTUs as calculated in a first pass of an input stream. As shown in the figure, complexity measures can vary between different CTUs.
• Encoding information can include a quantization parameter for a particular block of pixels (such as a CTU or TU).
• a process for generating a quantization parameter in accordance with embodiments of the invention is illustrated in FIG. 8. The process includes selecting 810 a block of pixels.
• a stored complexity measure for the block is retrieved 812.
• a quantization parameter for encoding a block (such as a CTU or TU) in the second pass may be determined 814 using statistics such as a complexity measure calculated for the block in the first pass. The calculation can also consider the deviation from the average base picture quantization parameter.
• Quantization parameters can utilize complexity measures determined in the first pass and/or quantization parameters determined for blocks in other output streams that were already encoded (for example in a lower or higher bit rate stream).
• a process for using a quantization parameter already generated for a similar block of pixels is illustrated in FIG. 9. The process includes selecting 910 a block of pixels.
• a stored quantization parameter is retrieved 912 for a similar block.
• a similar block is a block in a frame in an output stream that was generated from the same block in a frame in the input stream as the selected block.
• a similar block can be a corresponding block already encoded in an output stream of the same resolution but a different bit rate.
• a quantization parameter is calculated for the selected block using the retrieved quantization parameter.
• the quantization parameter can further be adjusted 916 for statistics calculated for the input stream and/or already encoded frames of output streams such as, but not limited to, distortion rate and bit rate.
• the quantization parameter can be calculated to achieve minimum distortion and a target bit rate.
• FIG. 10B illustrates a plot of quantization parameters that are calculated for CTUs in three different output streams (at the same resolution and different bit rates) along with the plot of complexity measures illustrated in FIG. 10A. As shown in the figure, the complexity measure values were used in determining the quantization parameter values and so the plots have a similar general shape due to their relationship.
• a set of alternative streams of video content should have intra frames (I frames) in the same position in each stream.
• the corresponding frames at the same position (i.e., time) in each of the other alternative streams should be an I frame.
• an I frame includes all the information necessary to render the frame, when a playback device needs to switch streams it can switch at an I frame so that playback is seamless without missing frames.
• I frames are encoded for frames in the same position in a plurality of alternative output streams. Where a P frame or B frame has been encoded for an output stream, an encoder can skip testing to encode the corresponding frame in the same position as an I frame when encoding another output stream and can encode the frame as a B frame or P frame.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Computing Systems (AREA)
• Theoretical Computer Science (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)

Applications Claiming Priority (3)

Publications (2)

Family

ID=51934234

Family Applications (1)

Country Status (5)

Cited By (4)

Families Citing this family (18)

Family Cites Families (16)

• 2014
  • 2014-05-23 WO PCT/US2014/039425 patent/WO2014190308A1/en active Application Filing
  • 2014-05-23 KR KR1020157036373A patent/KR101823321B1/ko active IP Right Grant
  • 2014-05-23 CN CN201480038648.6A patent/CN105359511A/zh active Pending
  • 2014-05-23 JP JP2016515129A patent/JP2016526336A/ja not_active Ceased
  • 2014-05-23 EP EP14800901.2A patent/EP3005689A4/de not_active Withdrawn
  • 2014-05-23 KR KR1020187002004A patent/KR20180010343A/ko not_active Application Discontinuation

Also Published As

Similar Documents

Legal Events