Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/124—Quantisation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/103—Selection of coding mode or of prediction mode
  • H04N19/105—Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/117—Filters, e.g. for pre-processing or post-processing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/12—Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
  • H04N19/122—Selection of transform size, e.g. 8x8 or 2x4x8 DCT; Selection of sub-band transforms of varying structure or type
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
  • H04N19/13—Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/136—Incoming video signal characteristics or properties
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
  • H04N19/146—Data rate or code amount at the encoder output
  • H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/186—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a colour or a chrominance component
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/46—Embedding additional information in the video signal during the compression process
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/513—Processing of motion vectors
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/527—Global motion vector estimation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/53—Multi-resolution motion estimation; Hierarchical motion estimation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/537—Motion estimation other than block-based
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/573—Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
  • H04N19/61—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding in combination with predictive coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
  • H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
  • H04N19/91—Entropy coding, e.g. variable length coding [VLC] or arithmetic coding

Definitions

• a video encoder compresses video information so that more information can be sent over a given bandwidth.
• the compressed signal may then be transmitted to a receiver having a decoder that decodes or decompresses the signal prior to display.
• High Efficient Video Coding is the latest video compression standard, which is being developed by the Joint Collaborative Team on Video Coding (JCT-VC) formed by ISO/IEC Moving Picture Experts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG).
• JCT-VC Joint Collaborative Team on Video Coding
• MPEG Moving Picture Experts Group
• VCEG ITU-T Video Coding Experts Group
• HEVC is being developed in response to the previous H.264/ AVC (Advanced Video Coding) standard not providing enough compression for evolving higher resolution video applications. Similar to previous video coding standards, HEVC includes basic functional modules such as intra/inter prediction, transform, quantization, in-loop filtering, and entropy coding.
• the ongoing HEVC standard may attempt to improve on limitations of the H.264/AVC standard such as limited choices for allowed prediction partitions and coding partitions, limited allowed multiple references and prediction generation, limited transform block sizes and actual transforms, limited mechanisms for reducing coding artifacts, and inefficient entropy encoding techniques.
• limitations of the H.264/AVC standard such as limited choices for allowed prediction partitions and coding partitions, limited allowed multiple references and prediction generation, limited transform block sizes and actual transforms, limited mechanisms for reducing coding artifacts, and inefficient entropy encoding techniques.
• the ongoing HEVC standard may use iterative approaches to solving such problems.
• NVM Next Generation Video
• the present description developed within the context of a Next Generation Video (NGV) codec project, addresses the general problem of designing an advanced video codec that maximizes the achievable compression efficiency while remaining sufficiently practical for implementation on devices. For instance, with ever increasing resolution of video and expectation of high video quality due to availability of good displays, the corresponding bitrate/bandwidth required using existing video coding standards such as earlier MPEG standards and even the more recent H.264/AVC standard, is relatively high. H.264/AVC was not perceived to be providing high enough compression for evolving higher resolution video applications.
• Fig. 1 is an illustrative diagram of an example next generation video encoder
• Fig. 2 is an illustrative diagram of an example next generation video decoder
• FIG. 3(a) is an illustrative diagram of an example next generation video encoder and subsystems
• FIG. 3(b) is an illustrative diagram of an example next generation video decoder and subsystems
• FIG. 4 is an illustrative diagram of modified prediction reference pictures
• FIG. 5 is a diagram of a frame sequence for explaining a method of providing super- resolution synthesized reference frames
• FIG. 6 is an illustrative diagram of an example encoder subsystem
• FIG. 7 is an illustrative diagram of an example decoder subsystem
• FIG. 8 is an illustrative diagram of an example encoder prediction subsystem
• FIG. 9 is an illustrative diagram of an example decoder prediction subsystem
• FIG. 10 is an illustrative diagram showing frames used to illustrate dominant motion compensation using motion vectors
• FIG. 11 is a flow chart of a method of performing dominant motion compensation
• FIG. 12 is a flow chart of another method of dominant motion compensation
• FIG. 13 is a flow chart of a detailed method of dominant motion compensation using motion vectors;
• FIG. 14-16 are illustrative diagrams showing frames to explain global motion compensation;
• FIG. 17 is an illustrative diagram of frames showing dominant motion compensation using motion vectors
• FIG. 18 is a diagram of a dominant motion compensation subsystem at an encoder
• FIG. 19 is a diagram of a dominant motion compensation subsystem at a decoder
• FIGS. 20-23 are diagrams of frames showing alternative dominant motion compensation techniques using motion vectors
• FIG. 24 is a diagram of a dominant motion compensation subsystem at an encoder
• FIG. 25 is a diagram of a dominant motion compensation subsystem at a decoder
• FIG. 26 is a flow chart of a detailed method of dominant motion compensation using local global motion compensation
• FIGS. 27-31 are diagrams of alternative dominant motion compensation techniques using local global motion compensation
• FIG. 32 is a diagram of a dominant motion compensation subsystem at an encoder
• FIG. 33 is a diagram of a dominant motion compensation subsystem at a decoder
• FIG. 34 is a diagram of an example video coding system and video coding process operation.
• FIG. 35 is an illustrative diagram of an example video coding system
• FIG. 36 is an illustrative diagram of an example system.
• FIG. 37 illustrates an example device.
• SoC system-on-a-chip
• implementation of the techniques and/or arrangements described herein are not restricted to particular architectures and/or computing systems and may be implemented by any architecture and/or computing system for similar purposes.
• various architectures employing, for example, multiple integrated circuit (IC) chips and/or packages, and/or various computing devices and/or consumer electronic (CE) devices such as set top boxes, smart phones, etc. may implement the techniques and/or arrangements described herein.
• IC integrated circuit
• CE consumer electronic
• claimed subject matter may be practiced without such specific details.
• some material such as, for example, control structures and full software instruction sequences, may not be shown in detail in order not to obscure the material disclosed herein.
• a machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device).
• a machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.); and others.
• references in the specification to "one implementation”, “an implementation”, “an example implementation”, etc., indicate that the implementation described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same implementation. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other implementations whether or not explicitly described herein.
• the H.264/AVC coding standard while it represents improvement over past MPEG standards, it is still very limiting in choices of prediction due to the following reasons: the choices for allowed prediction partitions are very limited, the accuracy of prediction for prediction partitions is limited, and the allowed multiple reference predictions are very limited as they are discrete based on past decoded frames rather than accumulation of resolution over many frames.
• the aforementioned limitations of the state of the art standards such as H.264 are recognized by the ongoing work in HEVC that uses an iterative approach to fixing these limitations.
• the problem of improved prediction is currently being solved in an ad hoc manner by using decoded multiple references in the past and/or future for motion compensated prediction in inter- frame coding of video. This is done with the hope that in the past or future frames, there might be some more similar areas to the area of current frame being predicted than in the past frame (for P-pictures/slices), or in the past and future frames (for B- pictures/slices).
• NVM Next Generation Video
• a new type of prediction is disclosed herein called locally adaptive dominant motion compensated prediction (or simply dominant motion compensation prediction) that provides improved prediction which in turn reduces the prediction error, thereby improving the overall video coding efficiency.
• techniques described herein may differ from standards based approaches as it naturally incorporates significant content based adaptivity in video coding process to achieve higher compression.
• standards based video coding approaches typically tend to squeeze higher gains by adaptations and fine tuning of legacy approaches.
• all standards based approaches heavily rely on adapting and further tweaking of motion compensated interframe coding as the primary means to reduce prediction differences to achieve gains.
• some video coding implementations disclosed herein in addition to exploiting interframe differences due to motion, also exploits other types of interframe differences (gain, blur, registration, dominant/global motion) that naturally exist in typical video scenes, as well as prediction benefits of frames synthesized from past decoded frames only or a combination of past and future decoded frames.
• the morphed frames used for prediction include dominant motion compensated (DMC) prediction.
• the synthesized frames used for prediction include Super Resolution (SR) frames and PI (Projected Interpolation) frames. Besides the issue of exploiting other sources of interframe differences besides motion, some video coding implementations disclosed herein differ from standards in other ways as well.
• DMC dominant motion compensation
• a block (such as a 16 x 16 block of pixels) of a current frame being analyzed is matched during motion estimation to a similar block in a previously decoded reference frame.
• the motion estimation process thus involves estimating motion of blocks to determine mv (mvx, mvy) for each block.
• the computed motion estimates are then efficiently encoded (by first differencing them with prediction, and entropy coding of the difference) and sent via bitstream to the decoder where they are decoded (by entropy decoding and adding the prediction back in) and used for motion compensation.
• motion estimation/compensation is performed with high accuracy (such as 1 ⁇ 4 pixel or 1/8 pixel accuracy rather than integer pixel accuracy) by use of a fixed or adaptive interpolation filter for generation of a prediction block.
• the block sizes themselves may be square or non-square (e.g. 16x8, 8x16) and of multiple sizes (e.g. 4x4, 8x8, 16x16, and others).
• H.264 includes several good ways of generating predictions derived from block- based motion vectors, this may result in two limitations: (1) block-based motion vectors, regardless of block sizes or references, are all modeled based on an assumption of translatory motion, which may disregard alternative types of motion between frames and resulting in large prediction error, and (2) block-based motion vectors, while they provide good compensation of local motion, the inherent substantial bit-cost associated with the block-based motion can limit the potential gain that may otherwise be possible. Improvements have involved use of variable size blocks which helps in reducing overhead, but the overhead reduction is still quite limited.
• the block-based translator motion representation and coding of motion vectors can be particularly inefficient. Since it was realized that global motion in a video can present a challenge to block-based prediction due to a large prediction resulting from a translatory motion model, and significant amount of motion vector overhead, an alternative approach was investigated that directly estimates/compensates global motion due to its potential to adapt to nontranslatory/complex motion, and a more compact representation of motion parameters as they are needed only once per picture (note that herein the term frame is used interchangeably with the term picture).
• the affine model uses six parameters and is able to address a large range of complex motions (translation, zoom, shearing, and rotation). In the typical process, the model results in a warped frame used to form the predictions by reading the blocks on the warped frame.
• the perspective model is more complex than the affine model and in addition to the motions listed for affine, this method can also handle a change in perspective. Due to the higher complexity of the perspective model, it is not discussed here in detail, but in general it is applicable in the same manner as the affine model.
• the details for global motion compensation, at least as used by the system and process herein, are discussed below.
• MPEG-4 part 2 standard represents an improvement (due to inclusion of Global Motion and other aspects) over past MPEG or ITU-T standards, it still offers only a limited improvement in motion compensated prediction due to the following reasons. Even though a combination of local and global motion compensation is allowed, the local motion compensation occurs at a very small block size (16x16 size at most). Thus, there is considerable overhead in signaling, at a 16 x 16 basis, when to use local versus global motion compensation. This overhead cuts into the possible gains due to GMC. Also, since P-pictures only use one reference frame and B-pictures only use two, GMC is limited to being applied only on immediately past decoded frames.
• global motion parameters are computed only once for the entire picture (including blocks where local motion is found to be more suitable) causing global motion parameters to be inaccurate often, especially in the case of a frame that contains a mixture of both local and global motion. Otherwise, other than using or not using global motion compensated prediction, no adjustment or correction of GMC generated prediction is possible. Lastly, the process for generating interpolation (e.g. 1 ⁇ 4 or 1/8 pel precision) is simplistic and results in blurry prediction.
• the affine model uses six parameters, and is able to address a large range of complex motions, while the perspective model is more complex and flexible, but can use up to eight parameters.
• the affine model may be sufficient for many cases and can allow global compensation for motion of types such as translation, zoom, shear, and rotation.
• affine model based global motion estimation or compensation (GME/C) was a notable improvement for scenes with global motion over use of block based translatory motion, in reality both block-based local and global motion is combined here for best coding efficiency results.
• the affine model can also be applied for motion compensation of non-overlapping tiles, or regions/objects in a scene. This results in multiple global motion parameter sets, and the process is referred to as performing dominant motion compensation (DMC).
• DMC dominant motion compensation
• coder may refer to an encoder and/or a decoder.
• coding may refer to performing video encoding via an encoder and/or performing video decoding via a decoder.
• a video encoder and video decoder may both be examples of coders capable of coding video data.
• codec may refer to any process, program or set of operations, such as, for example, any combination of software, firmware, and/or hardware that may implement an encoder and/or a decoder.
• video data may refer to any type of data associated with video coding such as, for example, video frames, image data, encoded bit stream data, or the like.
• encoder 100 may receive input video 101.
• Input video 101 may include any suitable input video for encoding such as, for example, input frames of a video sequence.
• input video 101 may be received via a content pre-analyzer module 102.
• Content pre-analyzer module 102 may be configured to perform analysis of the content of video frames of input video 101 to determine various types of parameters for improving video coding efficiency and speed performance.
• content pre-analyzer module 102 may determine horizontal and vertical gradient information (for example, Rs, Cs), variance, spatial complexity per frame, temporal complexity per frame (tpcpx), scene change detection, motion range estimation, gain detection, prediction distance estimation (pdist), number of objects estimation, region boundary detection, spatial complexity map computation, focus estimation, film grain estimation, or the like.
• the parameters generated by content pre-analyzer module 102 may be used by encoder 100 (e.g., via encode controller 103) and/or quantized and communicated to a decoder.
• video frames and/or other data may be transmitted from content pre- analyzer module 102 to adaptive picture organizer module 104 (also referred to as the hierarchical picture group structure organizer).
• the adaptive organizer module 104 determines the picture group structure and the picture types of each picture in the group as well as reorder pictures in encoding order as needed.
• the adaptive organizer module 104 outputs control signals indicating the picture group structure and picture types (the abbreviations for the output/input controls shown on system 100 are recited below).
• the NGV coding described herein uses I-pictures (intra-coding), P-pictures (formed from inter-prediction from past/previous reference frames), and F-pictures (functional as described below). In some cases, B-pictures might also be used.
• adaptive picture organizer module 104 may include a frame portion generator configured to generate frame portions.
• content pre-analyzer module 102 and adaptive picture organizer module 104 may together be considered a pre-analyzer subsystem of encoder 100.
• video frames and/or other data may be transmitted from adaptive picture organizer module 104 to prediction partitions generator module 105.
• prediction partitions generator module 105 first may divide a frame or picture into tiles or super-fragments or the like (herein the terms frame, picture, and image may be used interchangeably except as otherwise noted and except that a frame is used to generally refer to a frame that is not necessarily assigned a specific picture type (I, P, F, or B -pictures for example)).
• an additional module for example, between modules 104 and 105) may be provided for dividing a frame into tiles or super- fragments or the like.
• a frame may be divided into tiles of 32 x 32 or 64 x 64 pixels where 64 x 64 is used for all standard definition and higher resolution video for coding of all picture types (I-, P-, or F-). For low resolution sequences, 64 x 64 is still used for coding of I- and F- pictures, while 32 x 32 is used for P-pictures.
• prediction partitions generator module (which also may be referred to as Pred KdTree/BiTree Partitions Generator) 105 may then divide each tile or super- fragment into potential prediction partitionings or partitions.
• the potential prediction partitionings may be determined using a partitioning technique such as, for example, a k-d tree partitioning technique, a bi-tree partitioning technique, or the like, which may be determined based on the picture type (for example, I-, P-, or F-picture) of individual video frames, a characteristic of the frame portion being partitioned, or the like.
• a partitioning technique such as, for example, a k-d tree partitioning technique, a bi-tree partitioning technique, or the like, which may be determined based on the picture type (for example, I-, P-, or F-picture) of individual video frames, a characteristic of the frame portion being partitioned, or the like.
• a partitioning technique such as, for example, a k-d tree partitioning technique, a bi-tree partitioning technique, or the like, which may be determined based on the picture type (for example, I-, P-, or F-picture) of individual video frames, a characteristic of the frame portion
• the options for dividing the space may include no further division, division into two equal halves, division into two parts that are 1/4 and 3/4 of the space, or division into two parts that are 3/4 and 1/4 of the space. So, with I-pictures using 64 x 64 as the largest size (and allowing a minimum size of 4 x 4), a very large number of partitions of a tile can be generated if no other constraints are imposed. For example, one constraint is to set that the first pair of cuts are pre-decided for a 64 x 64 tile to halve the space in both the horizontal and vertical dimension so that four 32 x 32 sub-tiles are formed, and then sub-partitioning each 32 x 32 sub-tile by KdTree partitioning. Other restrictions are also possible to reduce the number of possible partition combinations.
• each tile partition may be used for spatial prediction (directional angular prediction or other types of prediction) and coding of prediction differences.
• P-picture tiles can also be partitioned in this manner for prediction except that for lower resolutions, P-picture partitions start with a 32 x 32 tile, and KdTree based partitions are not used, but rather a simpler Bi-Tree partitioning is used.
• Bi-Tree partitioning divides a space into two equal parts, one dimension at a time, alternating between the two dimensions.
• P-picture partitions are mainly predicted using motion (with one or more references) rather than spatial prediction, although some sub-partitions can use intra spatial prediction to deal with, for instance, uncovered background.
• P-pictures start with 64 x 64 tiles before being divided.
• F-pictures also use Bi-Tree partitioning and start with 64 x 64 tiles for generating prediction partitions that mainly use motion (with one or more partitions), although some sub-partitions can also use spatial prediction (for intra coding).
• a largest coding unit also called a coding tree unit (CTU)
• CTU coding tree unit
• CTBs coding tree blocks
• CUs coding units
• a tile as used herein covers HEVC and generally refers to a large block such as an LCU or at least a block larger than a macroblock (MB) of 16 x 16 pixels unless the context suggests otherwise.
• each sub-partition's prediction is identified by including a prediction mode.
• the prediction modes include skip, auto, intra, inter, multi, and split.
• Skip mode is used to skip prediction coding when, for example, there is no, or relatively little change, from a reference frame to a current frame being reconstructed so that the pixel data need not be encoded and merely copied from one frame to the other when decoded.
• Auto mode is used when only partial data is needed so that for example, motion vectors may not be needed but transform coefficients are still used to code the data.
• Intra mode means that the frame or partition is spatially coded.
• Inter mode means that multiple reference frames are determined for a current frame, and motion estimations are obtained by using each reference separately, and then the best result is used for the motion prediction data.
• Multi mode also uses multiple reference frames, but in this case, the motion estimation data from the multiple reference frames is combined, such as averaged, or weighted averaged, to obtain a single result to be used for the prediction.
• One of the outputs of prediction partitions generator module 105 may be hundreds of potential partitionings (and more or less depending on the limits placed on the partitioning) of a tile. These partitionings are indexed as 1...m and are provided to the encode controller 103 to select the best possible prediction partitioning for use. As mentioned, the determined potential prediction partitionings may be partitions for prediction (for example, inter- or intra- prediction) and may be described as prediction partitions or prediction blocks or the like.
• a selected prediction partitioning (for example, prediction partitions) may be determined from the potential prediction partitionings.
• the selected prediction partitioning may be based on determining, for each potential prediction partitioning, predictions using characteristics and motion based multi-reference predictions or intra- predictions, and determining prediction parameters.
• a potential prediction error may be determined by differencing original pixels with prediction pixels, and the selected prediction partitioning may be the potential prediction partitioning with the minimum prediction error.
• the selected prediction partitioning may be determined based on a rate distortion optimization including a weighted scoring based on number of bits for coding the partitioning and a prediction error associated with the prediction partitioning.
• the original pixels of the selected prediction partitioning may be differenced with predicted partitions (for example, a prediction of the prediction partition of the current frame based on a reference frame or frames and other predictive data such as inter- or intra-prediction data) at differencer 106.
• predicted partitions for example, a prediction of the prediction partition of the current frame based on a reference frame or frames and other predictive data such as inter- or intra-prediction data
• the determination of the predicted partitions will be described further below and may include a decode loop 135 as shown in FIG. 1.
• the original partitioned blocks also are differenced with the prediction blocks to determine whether or not any residual signal exists that warrants encoding. Thus, not all sub-partitions of a tile actually need to be coded (using transform coding for example) as prediction may have been sufficient for certain sub-partitions.
• any residuals or residual data (for example, partition prediction error data) from the differencing that indicate that the partition cannot be compensated by prediction alone (such as motion compensation alone) may be transmitted to coding partitions generator module (or by one example, coding bitree partitions generator) 107 to be further subpartitioned into smaller partitions for transform coding (coding partitions), and particularly for P-pictures and F-pictures by one example.
• the coding partitions may equal the size of the entire tile, or the coding partitions and prediction partitions may have the same size in these cases.
• some P- and F-picture tiles may contain no coding partitioning, one coding partitioning, or multiple coding partitionings. These coding partitions are indexed as l ...n, and are provided to encode controller 103 to select the best possible combination of prediction and coding partitioning from the given choices. Also, in some of these examples, such as for intra-prediction of prediction partitions in any picture type (I-, F- or P-pictures), or otherwise where prediction partitions are not further divided into coding partitions (where coding partitions are skipped), coding partitions generator module 107 may be bypassed via switches 107a and 107b. In such examples, only a single level of partitioning may be performed.
• partitioning where only a single level of partitioning exists, it may be described as prediction partitioning (as discussed) or coding partitioning or both.
• partitioning may be performed via prediction partitions generator module 105 (as discussed) or, as is discussed further herein, such partitioning may be performed via a k-d tree intra-prediction/coding partitioner module or a bi- tree intra-prediction/coding partitioner module implemented via coding partitions generator module 107.
• the partition prediction error data may not be significant enough to warrant encoding.
• coding partitions generator module 107 may determine coding partitions of the prediction partitions. In some examples, coding partitions generator module 107 may not be needed as the partition may be encoded without coding partitioning (e.g., as shown via the bypass path available via switches 107a and 107b). With or without coding partitioning, the partition prediction error data (which may subsequently be described as coding partitions in either event) may be transmitted to adaptive transform module 108 in the event the residuals or residual data require encoding.
• prediction partitions generator module 105 and coding partitions generator module 107 may together be considered a partitioner subsystem of encoder 100.
• coding partitions generator module 107 may operate on partition prediction error data, original pixel data, residual data, or wavelet data.
• Coding partitions generator module 107 may generate potential coding partitionings (for example, coding partitions) of, for example, partition prediction error data using bi-tree and/or k-d tree partitioning techniques or the like.
• the partitions may be transformed using adaptive or fixed transforms with various block sizes via adaptive transform module 108 (also, in one form, referred to as the Adaptive Multi-size Rect Hybrid Parametric Haar Transform (HPHT)/ Discrete Cosine Transform (DCT) unit).
• adaptive transform module 108 may perform forward HPHT or forward DCT on rectangular blocks.
• partition/block size as well as selected transforms may be determined based on a rate distortion optimization (RDO) or other basis.
• both the selected coding partitioning and/or the selected transform(s) may be determined based on a predetermined selection method based on coding partitions size or the like.
• adaptive transform module 108 may include a first portion or component for performing a parametric transform to allow locally optimal transform coding of small to medium size blocks, and a second portion or component for performing globally stable, low overhead transform coding using a fixed transform, such as DCT or a picture based transform from a variety of transforms, including parametric transforms, or any other configuration.
• HPHT may be performed for locally optimal transform coding.
• transforms may be performed on 2D blocks of rectangular sizes between about 4 x 4 pixels and 64 x 64 pixels, with actual sizes depending on a number of factors such as whether the transformed data is luma or chroma, or inter or intra, or if the determined transform used is
• HPHT transform small to medium block sizes are supported while for DCT transform a large number of block sizes are supported.
• some overhead is needed to identify the direction, either horizontal or vertical in which DCT is applied while the PHT is applied in the orthogonal direction, as well as the mode (at least for intra-coding where a mode can be based on decoded pixels or prediction difference pixels).
• the actual PHT transform basis used for transforming a particular block may be content adaptive as it depends on decoded neighboring pixels. Since both encoder and decoder require calculation of the same basis matrix, the complexity of the calculation is kept low by allowing a limited number of good transforms known (to both encoder and decoder) that one can select from.
• the resultant transform coefficients may be transmitted to adaptive quantize module 109, while a quantizer adapter control 133 at the encode controller 103 performs analysis of content to come up with locally adaptive quantization parameters that are then represented by a multi-level map that can be efficiently coded and included in the bitstream.
• the computed quantizer set (qs, and a matrix applied to a coefficient block) may be used by the adaptive quantizer module 109 to perform scaling of the resultant transform coefficients.
• any data associated with a parametric transform may be transmitted to either adaptive quantize module 109 (if quantization is desired) or adaptive entropy encoder module 110.
• the quantized coefficients may be scanned and transmitted to adaptive entropy encoder module 110.
• Adaptive entropy encoder module 110 may entropy encode the quantized coefficients and include them in output bitstream 111.
• adaptive transform module 108 and adaptive quantize module 109 may together be considered a transform encoder subsystem of encoder 100.
• encoder 100 includes the local decode loop 135 to form predicted partitions (or frames) for comparison to the prediction partitions as mentioned above. Preliminarily, depending on the RDO operation, not all of the hundreds or more tile partitions described above need to be fully coded such as when lookup of bitcounts are sufficient. Once the best partitioning of a tile is determined, however, in that case full coding may be provided.
• the local decode loop 135 may begin at adaptive inverse quantize module 112.
• Adaptive inverse quantize module 112 may be configured to perform the opposite operation(s) of adaptive quantize module 109 such that an inverse scan may be performed and quantized coefficients may be de-scaled to determine transform coefficients. Such an adaptive quantize operation may be lossy, for example.
• the transform coefficients may be transmitted to an adaptive inverse transform module 113.
• Adaptive inverse transform module 113 may perform the inverse transform as that performed by adaptive transform module 108, for example, to generate residuals or residual values or partition prediction error data (or original data or wavelet data, as discussed) associated with coding partitions.
• adaptive inverse quantize module 112 and adaptive inverse transform module 113 may together be considered a transform decoder subsystem of encoder 100.
• the partition prediction error data (or the like) for P and F-pictures may be transmitted to optional coding partitions assembler 114.
• Coding partitions assembler 114 may assemble coding partitions into decoded prediction partitions as needed (as shown, in some examples, coding partitions assembler 114 may be skipped such as for I-picture tile partitioning, and via switches 114a and 114b such that decoded prediction partitions may have been generated at adaptive inverse transform module 113) to generate prediction partitions of prediction error data or decoded residual prediction partitions or the like.
• the decoded residual prediction partitions (inter or intra) may be added to predicted partitions (for example, prediction pixel data) at adder 115 to generate reconstructed prediction partitions.
• the reconstructed prediction partitions may be transmitted to prediction partitions assembler 116.
• Prediction partitions assembler 116 may assemble the reconstructed prediction partitions to generate reconstructed tiles or super- fragments.
• coding partitions assembler module 114 and prediction partitions assembler module 116 may together be considered an un-partitioner sub-system of encoder 100.
• the next set of steps involve filtering, and intermingling of filtering and prediction generation.
• Overall four types of filtering are shown.
• the reconstructed partitions are deblocked and dithered by a blockiness analyzer & deblock filtering module (also Recon Blockiness Analyzer & DD Filt Gen) 117.
• the resulting parameters for analysis ddi are used for filtering operation and are also coded and sent to the decoder via the bitstream 111.
• the deblocked reconstructed output is then handed over to the quality analyzer & quality restoration filtering module ( or quality improvement filter also referred to as Recon Quality Analyzer & QR Filt Gen) 118, which computes QR filtering parameters and uses them for filtering.
• These parameters are also coded and sent via the bitstream 111 to the decoder.
• the QR filtered output is the final reconstructed or decoded frame that is also used as a prediction for coding future frames.
• the blockiness analyzer and deblock filtering module 117 may deblock and dither the reconstructed tiles or super- fragments (or prediction partitions of tiles or super-fragments).
• the generated deblock and dither filter parameters may be used for the current filter operation and/or coded in bitstream 111 for use by a decoder, for example.
• the output of blockiness analyzer and deblock filtering module 117 may be transmitted to the quality analyzer and quality restoration filtering module 118.
• Quality analyzer and quality restoration filtering module 118 may determine QR filtering parameters (for example, for a QR decomposition) and use the determined parameters for filtering.
• the QR filtering parameters may also be coded in bitstream 111 for use by a decoder.
• blockiness analyzer and deblock filtering module 117 and quality analyzer and quality restoration filtering module 118 may together be considered a filtering subsystem of encoder 100.
• the output of quality analyzer and quality restoration filtering module 118 may be a final reconstructed frame that may be used for prediction for coding other frames (for example, the final reconstructed frame may be a reference frame or the like).
• the output of quality analyzer and quality restoration filtering module 118 may be transmitted to a multi-reference frame storage and frame selector (or multi reference control) 119 which also may be referred to as, or may include, the decoded picture storage or buffer.
• a dependency logic module 128 may provide indices for listing the reference frames and the relationship among the frames such as frame dependencies, or more specifically partition dependencies, for proper ordering and use for the frames by the multi reference control 119 and when certain frames are to be selected for prediction of another frame. This may include providing the dependency logic for picture group structures such as multi-reference prediction, chain prediction, hierarchal structures, and/or other prediction techniques as described below.
• encoder 100 may perform inter- and/or intra-prediction operations.
• inter-prediction may be performed by one or more modules including morphing generation and local buffer module 120 (and in one example is referred to as Morph Gen & Loc Buf, or referred to herein as the in-loop morphing generation module), synthesizing generation and local buffer module 121 (and in one example is referred to as Synth Gen & Pic Buffer or referred to herein as in-loop synthesizing generation module), motion estimator 122, characteristics and motion filtering and predictor module 123 (also in some examples may be referred to as Char and Motion AP Filter Analyzer & 1/4 & 1/8 Pel Compensated Predictor), morphing analyzer and generation module (or out-of-loop morphing analyzer module) 130, and synthesizing analyzer and generation module (or out-of-loop synthesizing analyzer module) 132, where the morphing and synthesis generators 120 and 121 are
• both in-loop and out-of-loop modules may perform the same or similar tasks (forming modified frames and modification parameters for morphing and/or synthesis).
• morphing generation module 120, or morphing analyzer 130 may permit various forms of morphing of a decoded frame to then be used as a reference frame for motion prediction on other frames.
• the module 120 may analyze a current picture to determine morphing parameters for (1) changes in gain, and specifically to perform gain compensation for changes in brightness from one frame to another frame, (2) changes in dominant (or global) motion and as discussed in detail below, (3) changes in registration, and/or (4) changes in blur with respect to a reference frame or frames with which it is to be coded, and prior to motion compensated prediction.
• the out-of-loop morphing analyzer 130 and the synthesizing analyzer 132 receive picture group structure data from the adaptive picture organizer 104 and communicate with the encoder controller 103 to form the morphing and synthesis parameters ⁇ mop, syp) and modified reference frames based on the non-quantized, non-decoded, original frame data.
• the formation of the modified reference frames and modification parameters from the out-of-loop morphing and synthesis analyzers 130 and 132 may be much faster than that provided through the decoder loop 135, and this is especially advantageous for real time encoding.
• the use of the modified frames and parameters to perform compensation at another location should be performed by the in- loop morphing and synthesis generators 120 and 121 on the decoding loop side of the encoder so that the correct compensation can be repeated when reconstructing frames at the decoder.
• the resulting modification parameters from the out-of-loop analyzers 130 and 132 are used by the in-loop morphing and synthesizing generator 120 and 121 to form the modified reference frames and for motion estimation by the motion estimator 122 to compute motion vectors.
• the computed morphing and synthesis parameters ⁇ mop and syp) may be quantized/de-quantized and used (for example, by morphing generation module 120) to generate morphed reference frames that may be used by motion estimator module 122 for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• the synthesizing generation module 121 uses several types of synthesized frames including super resolution (SR) pictures, projected interpolation (PI) pictures, among others in which motion compensated prediction can result in even higher gains by determining motion vectors for efficient motion compensated prediction in these frames.
• SR super resolution
• PI projected interpolation
• Motion estimator module 122 may generate motion vector data based at least in part on morphed reference frame(s) and/or super resolution (SR) pictures and projected interpolation (PI) pictures along with the current frame.
• motion estimator module 122 may be considered an inter-prediction module.
• the motion vector data may be used for inter-prediction. If inter-prediction is applied, characteristics and motion filtering predictor module 123 may apply motion compensation as part of the local decode loop as discussed. Also, characteristics and motion filtering predictor module 123 may include adaptive precision (AP) filtering where filtering and prediction are intertwined.
• the filtering parameters (api) are coded and may be sent to the decoder via the bitstream 1 1 1.
• Intra-prediction may be performed by intra-directional prediction analyzer and prediction generation module 124.
• Intra-directional prediction analyzer and prediction generation module 124 may be configured to perform spatial directional prediction and may use decoded neighboring partitions. In some examples, both the determination of direction and generation of prediction may be performed by intra-directional prediction analyzer and prediction generation module 124. In some examples, intra-directional prediction analyzer and prediction generation module 124 may be considered an intra-prediction module.
• prediction modes and reference types analyzer module 125 may allow for selection of prediction modes as introduced above and from among, “skip”, “auto”, “inter”, “split”, “multi”, and “intra”, for each prediction partition of a tile (or super-fragment), all of which may apply to P- and F-pictures (as well as B-pictures when they are present). It should be noted that while the system considers a configuration where I, P, and F picture are available, it is possible to still provide B-pictures where no morphing or synthesis is available for the B-pictures. In addition to prediction modes, it also allows for selection of reference types that can be different depending on "inter” or "multi” mode, as well as for P- and F- pictures.
• the prediction signal at the output of prediction modes and reference types analyzer module 125 may be filtered by prediction analyzer and prediction fusion filtering module 126.
• Prediction analyzer and prediction fusion filtering module 126 may determine parameters (for example, filtering coefficients, frequency, overhead) to use for filtering and may perform the filtering.
• filtering the prediction signal may fuse different types of signals representing different modes (e.g., intra, inter, multi, split, skip, and auto).
• intra-prediction signals may be different than all other types of inter-prediction signal(s) such that proper filtering may greatly enhance coding efficiency.
• the filtering parameters may be encoded in bitstream 1 1 1 for use by a decoder.
• the filtered prediction signal may provide the second input (e.g., prediction partition(s)) to differencer 106, as discussed above, that may determine the prediction difference signal (e.g., partition prediction error) for coding discussed earlier. Further, the same filtered prediction signal may provide the second input to adder 115, also as discussed above. As discussed, output bitstream 111 may provide an efficiently encoded bitstream for use by a decoder for the presentment of video.
• encoder 100 may operate as an encoder prediction subsystem.
• an encoder prediction subsystem of encoder 100 may include multi-reference frame storage and frame selector 119, in-loop morphing analyzer and generation module 120, in-loop synthesizing analyzer and generation module 121, motion estimator module 122, and/or characteristics and motion compensated precision adaptive filtering predictor module 123 as well as out-of-loop morphing analyzer 130 and synthesizing analyzer 132.
• such an encoder prediction subsystem of encoder 100 may incorporate a number of components and the combined predictions generated by these components in an efficient video coding algorithm.
• proposed implementation of the NGV coder may include one or more of the following features: 1. Gain Compensation (e.g., explicit compensation for changes in gain/brightness in a scene); 2. Blur Compensation: e.g., explicit compensation for changes in blur/sharpness in a scene; 3. Dominant/Global Motion Compensation (e.g., explicit compensation for dominant motion in a scene); 4. Registration Compensation (e.g., explicit compensation for registration mismatches in a scene); 5. Super Resolution (e.g., explicit model for changes in resolution precision in a scene); 6. Projection (e.g., explicit model for changes in motion trajectory in a scene); the like, and/or combinations thereof.
• Gain Compensation e.g., explicit compensation for changes in gain/brightness in a scene
• Blur Compensation e.g., explicit compensation for changes in blur/sharpness in a scene
• the output of quality analyzer and quality restoration filtering may be transmitted to multi-reference frame storage and frame selector 119.
• the output of quality analyzer and quality restoration filtering may be a final reconstructed frame that may be used for prediction for coding other frames (e.g., the final reconstructed frame may be a reference frame or the like).
• prediction operations may include inter- and/or intra-prediction. As shown, inter-prediction may be performed by one or more modules including morphing generation module 120, synthesizing generation module 121, and/or characteristics and motion compensated precision adaptive filtering predictor module 123.
• morphing generation module 120 may analyze a current frame to determine parameters for changes in gain, changes in dominant motion, changes in registration, and changes in blur with respect to a reference frame or frames with which it is to be coded.
• the determined morphing parameters may be quantized/de-quantized and used (e.g., by morphing generation module 120) to generate morphed reference frames.
• Such generated morphed reference frames may be stored in a buffer and may be used by motion estimator module 122 for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• synthesizing analyzer and generation module 121 may generate super resolution (SR) pictures and projected interpolation (PI) pictures or the like for determining motion vectors for efficient motion compensated prediction in these frames.
• SR super resolution
• PI projected interpolation
• Such generated synthesized reference frames may be stored in a buffer and may be used by motion estimator module 122 for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• motion estimator module 122 may generate motion vector data based on morphed reference frame(s) and/or super resolution (SR) pictures and projected interpolation (PI) pictures along with the current frame.
• motion estimator module 122 may be considered an inter-prediction module.
• the motion vector data may be used for inter-prediction. If inter- prediction is applied, characteristics and motion filtering predictor module 123 may apply motion compensation as part of the local decode loop as discussed.
• the proposed implementation of the NGV coder may use one or more of the above components besides the usual local motion compensation with respect to decoded past and/or future, picture/slices. As such the implementation does not mandate a specific solution for instance for dominant motion compensation, or for any other characteristics compensated reference frame generation.
• FIG. 1 illustrates example control signals associated with operation of video encoder 100, where the following abbreviations may represent the associated information: scnchg Scene change information
• qs quantizer information set (includes Quantizer parameter (Qp), Quantizer
• the various signals and data items that may need to be sent to the decoder i.e., pgst, ptyp, prp, pptn, cptn, modes, reftype, ethp, xmtyp, xmdir, xmmod, idir, mv, qs, mop, syp, ddi, qri, api, fii, quant coefficients and others may then be entropy encoded by adaptive entropy encoder 110 that may include different entropy coders collectively referred to as an entropy encoder subsystem.
• the adaptive entropy encoder 110 may be used to encode various types of control data/signals, parameters, modes and ref types, motion vectors, and transform coefficients. It is based on a generic class of low complexity entropy coders called adaptive variable length coders (vie). The data to be entropy coded may be divided into several categories when convenient (seven in our case), and starting from generic vie coders, specialized coders are developed for each category. While these control signals are illustrated as being associated with specific example functional modules of encoder 100 in Fig. 1, other implementations may include a different distribution of control signals among the functional modules of encoder 300. The present disclosure is not limited in this regard and, in various examples, implementation of the control signals herein may include the undertaking of only a subset of the specific example control signals shown, additional control signals, and/or in a different arrangement than illustrated.
• FIG. 2 is an illustrative diagram of an example next generation video decoder 200, arranged in accordance with at least some implementations of the present disclosure and that utilizes the content adaptive P- and F-pictures and resulting picture groups herein.
• the general operation of this NGV decoder 200 may be similar to the local decoding loop in the NGV Encoder 100 discussed earlier with the caveat that the motion compensation decoding loop in a decoder does not require any components that require analysis to determine parameters as the parameters are actually sent via the bitstream 111 or 201 to decoder 200.
• the bitstream 201 to be decoded is input to adaptive entropy encoder (Content and Context Adaptive Entropy Decoder) 202 which decodes headers, control signals and encoded data.
• adaptive entropy encoder Content and Context Adaptive Entropy Decoder
• it decodes ptyp, pgst, prp , pptn, cptn, ethp, mop, syp, mod, reftyp, idir, qs, xmtyp, xmdir, xmmod, ddi, qri, api, fii, mv, listed above, and quantized transform coefficients that constitute the overhead, control signals and data that is distributed for use throughout the decoder.
• the quantized transform coefficients are then inverse quantized and inverse transformed by adaptive inverse quantize module 203 and adaptive inverse transform (also Adaptive Multi- size Rect HPHT/DCT) 204 to produce rectangular partitions of decoded pixel differences that are assembled as per coding partitioning used. Predictions are added to the differences resulting in generation of recon (reconstructed) coded partitions that undergo further reassembly as per motion partitioning to generate reconstructed tiles and frames that undergo deblocking and dithering in deblocking filter (Recon DD Filt Gen) 208 using decoded ddi parameters, followed by quality restoration filtering (or Recon QR Filt Gen) 209 using decoded qri parameters, a process that creates the final recon frames.
• Recon DD Filt Gen decodedddi parameters
• quality restoration filtering or Recon QR Filt Gen
• the final recon frames are saved in multi-reference frame storage and frame selector (also may be called decoded picture buffer) 210, and are used (or morphed) to create morphed pictures/local buffers (at morphed picture generator and buffer 211) depending on the applied, decoded mop parameters.
• synthesized picture and local buffers are created by applying decoded syp parameters to multi- reference frame storage and frame selector 210 (or in other words, the reconstructed frames in the storage or buffer 210).
• a dependency logic 220 may hold the index for, and perform the indexing for, the stored frames in the multi-reference frame storage 210.
• the indexing may be used for prediction techniques such as multi-reference frames, chain prediction and/or hierarchal (or pyramid) frame structures, and/or others as described below.
• the morphed local buffers and synthesized frames are used for motion compensated prediction that uses adaptive precision (AP) filtering based on api parameters, and keeps either 1/4 or 1/8 pel prediction depending on a decoded the ethp signal.
• AP adaptive precision
• a characteristics and motion compensated filtering predictor 213, depending on the mod generates "inter” multi" "skip” or “auto” partitions while an intra-directional prediction generation module 214 generates "intra” partitions, and prediction modes selector 215, based on an encoder selected option, allows partition of the correct mode to pass through.
• selective use of prediction fusion filter generation module (or Pred FI Filter Gen) 216 to filter and output the prediction is performed as needed as the second input to the adder.
• the recon frames at the output of the quality filter generation module 209 (or Recon QR Filt Gen) are reordered (as F-pictures are out of order) by adaptive picture reorganizer (or
• decoder 200 may receive an input bitstream 201.
• input bitstream 201 may be encoded via encoder 100 and/or via the encoding techniques discussed herein.
• input bitstream 201 may be received by an adaptive entropy decoder module 202.
• Adaptive entropy decoder module 202 may decode the various types of encoded data (e.g., overhead, motion vectors, transform coefficients, etc.). In some examples, adaptive entropy decoder 202 may use a variable length decoding technique. In some examples, adaptive entropy decoder 202 may perform the inverse operation(s) of adaptive entropy encoder module 110 discussed above.
• the decoded data may be transmitted to adaptive inverse quantize module 203.
• Adaptive inverse quantize module 203 may be configured to inverse scan and de-scale quantized coefficients to determine transform coefficients. Such an adaptive quantize operation may be lossy, for example.
• adaptive inverse quantize module 203 may be configured to perform the opposite operation of adaptive quantize module 109 (e.g., substantially the same operations as adaptive inverse quantize module 112).
• the transform coefficients (and, in some examples, transform data for use in a parametric transform) may be transmitted to an adaptive inverse transform module 204.
• Adaptive inverse transform module 204 may perform an inverse transform on the transform coefficients to generate residuals or residual values or partition prediction error data (or original data or wavelet data) associated with coding partitions.
• adaptive inverse transform module 204 may be configured to perform the opposite operation of adaptive transform module 108 (e.g., substantially the same operations as adaptive inverse transform module
• adaptive inverse transform module 204 may perform an inverse transform based on other previously decoded data, such as, for example, decoded neighboring partitions.
• adaptive inverse quantize module 203 and adaptive inverse transform module 204 may together be considered a transform decoder subsystem of decoder 200.
• the residuals or residual values or partition prediction error data may be transmitted to coding partitions assembler 205.
• Coding partitions assembler 205 may assemble coding partitions into decoded prediction partitions as needed (as shown, in some examples, coding partitions assembler 205 may be skipped via switches 205a and 205b such that decoded prediction partitions may have been generated at adaptive inverse transform module 204).
• the decoded prediction partitions of prediction error data (e.g., prediction partition residuals) may be added to predicted partitions (e.g., prediction pixel data) at adder 206 to generate reconstructed prediction partitions.
• the reconstructed prediction partitions may be transmitted to prediction partitions assembler 207.
• Prediction partitions assembler 207 may assemble the reconstructed prediction partitions to generate reconstructed tiles or super-fragments.
• coding partitions assembler module 205 and prediction partitions assembler module 207 may together be considered an un-partitioner subsystem of decoder 200.
• the reconstructed tiles or super-fragments may be transmitted to deblock filtering module 208.
• Deblock filtering module 208 may deblock and dither the reconstructed tiles or super-fragments (or prediction partitions of tiles or super-fragments).
• the generated deblock and dither filter parameters may be determined from input bitstream 201, for example.
• the output of deblock filtering module 208 may be transmitted to a quality restoration filtering module 209.
• Quality restoration filtering module 209 may apply quality filtering based on QR parameters, which may be determined from input bitstream 201, for example. As shown in FIG.
• the output of quality restoration filtering module 209 may be transmitted to multi- reference frame storage and frame selector (which may be referred to as a multi-reference control, and may be, or may include, a decoded picture buffer) 210.
• the output of quality restoration filtering module 209 may be a final reconstructed frame that may be used for prediction for coding other frames (e.g., the final reconstructed frame may be a reference frame or the like).
• deblock filtering module 208 and quality restoration filtering module 209 may together be considered a filtering subsystem of decoder
• inter-prediction compensation may be performed by one or more modules including morphing generation module 211, synthesizing generation module 212, and characteristics and motion compensated filtering predictor module 213.
• Morphing generation module 211 may use de-quantized morphing parameters (e.g., determined from input bitstream 201) to generate morphed reference frames.
• Synthesizing generation module 212 may generate super resolution (SR) pictures and projected interpolation (PI) pictures or the like based on parameters determined from input bitstream 201. If inter-prediction is applied, characteristics and motion compensated filtering predictor module 213 may apply motion compensation based on the received frames and motion vector data or the like in input bitstream 201.
• SR super resolution
• PI projected interpolation
• Intra-prediction compensation may be performed by intra-directional prediction generation module 214.
• Intra-directional prediction generation module 214 may be configured to perform spatial directional prediction and may use decoded neighboring partitions according to intra-prediction data in input bitstream 201.
• prediction modes selector module 215 may determine a prediction mode selection from among, “skip”, “auto”, “inter”, “multi”, and “intra”, for each prediction partition of a tile, all of which may apply to P- and F-pictures, based on mode selection data in input bitstream 201. In addition to prediction modes, it also allows for selection of reference types that can be different depending on "inter” or “multi” mode, as well as for P- and F- pictures.
• the prediction signal at the output of prediction modes selector module 215 may be filtered by prediction fusion filtering module 216. Prediction fusion filtering module 216 may perform filtering based on parameters (e.g., filtering coefficients, frequency, overhead) determined via input bitstream 201.
• filtering the prediction signal may fuse different types of signals representing different modes (e.g., intra, inter, multi, skip, and auto).
• intra-prediction signals may be different than all other types of inter- prediction signal(s) such that proper filtering may greatly enhance coding efficiency.
• the filtered prediction signal may provide the second input (e.g., prediction partition(s)) to differencer 206, as discussed above.
• the output of quality restoration filtering module 209 may be a final reconstructed frame.
• Final reconstructed frames may be transmitted to an adaptive picture re- organizer 217, which may re -order or re-organize frames as needed based on ordering parameters in input bitstream 201.
• Re-ordered frames may be transmitted to content post- restorer module 218.
• Content post-restorer module 218 may be an optional module configured to perform further improvement of perceptual quality of the decoded video. The improvement processing may be performed in response to quality improvement parameters in input bitstream 201 or it may be performed as standalone operation.
• content post- restorer module 218 may apply parameters to improve quality such as, for example, an estimation of film grain noise or residual blockiness reduction (e.g., even after the deblocking operations discussed with respect to deblock filtering module 208).
• decoder 200 may provide display video 219, which may be configured for display via a display device (not shown).
• some components of decoder 200 may operate as a decoder prediction subsystem.
• a decoder prediction subsystem of decoder 200 may include multi-reference frame storage and frame selector 210, dependency logic 220 to index the frames at the multi-reference frame storage and frame selector 210, morphing analyzer and generation module 211, synthesizing analyzer and generation module 212, and/or characteristics and motion compensated precision adaptive filtering predictor module 213.
• such a decoder prediction subsystem of decoder 200 may incorporate a number of components and the combined predictions generated by these components in an efficient video coding algorithm.
• proposed implementation of the NGV coder may include one or more of the following features: 1.
• Gain Compensation e.g., explicit compensation for changes in gain/brightness in a scene
• Blur Compensation e.g., explicit compensation for changes in blur/sharpness in a scene
• Dominant/Global Motion Compensation e.g., explicit compensation for dominant motion in a scene
• Registration Compensation e.g., explicit compensation for registration mismatches in a scene
• Super Resolution e.g., explicit model for changes in resolution precision in a scene
• Projection e.g., explicit model for changes in motion trajectory in a scene
• the output of quality restoration filtering module may be transmitted to multi-reference frame storage and frame selector 210.
• the output of quality restoration filtering module may be a final reconstructed frame that may be used for prediction for coding other frames (e.g., the final reconstructed frame may be a reference frame or the like).
• compensation due to prediction operations may include inter- and/or intra-prediction compensation.
• inter-prediction compensation may be performed by one or more modules including morphing analyzer and generation module 211, synthesizing analyzer and generation module 212, and/or characteristics and motion compensated precision adaptive filtering predictor module 213.
• morphing analyzer and generation module 211 may use de-quantized morphing parameters (e.g., determined from input bitstream) to generate morphed reference frames.
• de-quantized morphing parameters e.g., determined from input bitstream
• Such generated morphed reference frames may be stored in a buffer and may be used by characteristics and motion compensated precision adaptive filtering predictor module 213.
• synthesizing analyzer and generation module 212 may be configured to generate one or more types of synthesized prediction reference pictures such as super resolution (SR) pictures and projected interpolation (PI) pictures or the like based on parameters determined from input bitstream 201.
• synthesized prediction reference pictures such as super resolution (SR) pictures and projected interpolation (PI) pictures or the like based on parameters determined from input bitstream 201.
• Such generated synthesized reference frames may be stored in a buffer and may be used by motion compensated filtering predictor module 213.
• characteristics and motion compensated filtering predictor module 213 may apply motion compensation based on morphed reference frame(s) and/or super resolution (SR) pictures and projected interpolation (PI) pictures along with the current frame.
• SR super resolution
• PI projected interpolation
• the proposed implementation of the NGV coder may use one or more of the above components besides the usual local motion compensation with respect to decoded past and/or future, picture/slices. As such the implementation does not mandate a specific solution for instance for dominant motion compensation, or for any other characteristics compensated reference frame generation.
• FIG. 2 illustrates example control signals associated with operation of video decoder 200, where the indicated abbreviations may represent similar information as discussed with respect to FIG. 1 above. While these control signals are illustrated as being associated with specific example functional modules of decoder 200, other implementations may include a different distribution of control signals among the functional modules of encoder 100. The present disclosure is not limited in this regard and, in various examples, implementation of the control signals herein may include the undertaking of only a subset of the specific example control signals shown, additional control signals, and/or in a different arrangement than illustrated.
• FIGS. 1 and 2 illustrate particular encoding and decoding modules
• various other coding modules or components not depicted may also be utilized in accordance with the present disclosure.
• the present disclosure is not limited to the particular components illustrated in FIGS. 1 and 2 and/or to the manner in which the various components are arranged.
• Various components of the systems described herein may be implemented in software, firmware, and/or hardware and/or any combination thereof.
• various components of encoder 100 and/or decoder 200 may be provided, at least in part, by hardware of a computing System-on-a-Chip (SoC) such as may be found in a computing system such as, for example, a mobile phone.
• SoC computing System-on-a-Chip
• encoder 100 may be associated with and/or provided by a content provider system including, for example, a video content server system, and that output bitstream 111 may be transmitted or conveyed to decoders such as, for example, decoder 200 by various communications components and/or systems such as transceivers, antennae, network systems, and the like not depicted in FIGS. 1 and 2.
• a content provider system including, for example, a video content server system
• output bitstream 111 may be transmitted or conveyed to decoders such as, for example, decoder 200 by various communications components and/or systems such as transceivers, antennae, network systems, and the like not depicted in FIGS. 1 and 2.
• decoder 200 may be associated with a client system such as a computing device (e.g., a desktop computer, laptop computer, tablet computer, convertible laptop, mobile phone, or the like) that is remote to encoder 100 and that receives input bitstream 201 via various communications components and/or systems such as transceivers, antennae, network systems, and the like not depicted in FIGS. 1 and 2. Therefore, in various implementations, encoder 100 and decoder subsystem 200 may be implemented either together or independent of one another.
• a computing device e.g., a desktop computer, laptop computer, tablet computer, convertible laptop, mobile phone, or the like
• encoder 100 and decoder subsystem 200 may be implemented either together or independent of one another.
• FIG. 3 is an illustrative diagram of example subsystems associated with next generation video encoder 100, arranged in accordance with at least some implementations of the present disclosure.
• encoder 100 may include a structure subsystem 310, a partitioning subsystem 320, a prediction subsystem 330, a transform subsystem 340, a filtering subsystem 350, and/or an entropy coding subsystem 360.
• FIG. 3(a) is an illustrative diagram of an example next generation video encoder 300a, arranged in accordance with at least some implementations of the present disclosure.
• FIG. 3(a) presents a similar encoder to that shown in FIG. 1, and similar elements will not be repeated for the sake of brevity.
• encoder 300a may include pre-analyzer subsystem 310a, partitioner subsystem 320a, prediction encoding subsystem 330a, transform encoder subsystem 340a, filtering encoding subsystem 350a, entropy encoder system 360a, transform decoder subsystem 370a, and/or unpartitioner subsystem 380a.
• Pre-analyzer subsystem 310a may include content pre-analyzer module 102 and/or adaptive picture organizer module 104.
• Partitioner subsystem 320a may include prediction partitions generator module 105, and/or coding partitions generator 107.
• Prediction encoding subsystem 330a may include motion estimator module 122, characteristics and motion compensated filtering predictor module 123, and/or intra-directional prediction analyzer and prediction generation module 124.
• Transform encoder subsystem 340a may include adaptive transform module 108 and/or adaptive quantize module 109.
• Filtering encoding subsystem 350a may include blockiness analyzer and deblock filtering module 117, quality analyzer and quality restoration filtering module 118, motion estimator module 122, characteristics and motion compensated filtering predictor module 123, and/or prediction analyzer and prediction fusion filtering module 126.
• Entropy coding subsystem 360a may include adaptive entropy encoder module 110.
• Transform decoder subsystem 370a may include adaptive inverse quantize module 112 and/or adaptive inverse transform module 113.
• Unpartitioner subsystem 380a may include coding partitions assembler 114 and/or prediction partitions assembler 116.
• Partitioner subsystem 320a of encoder 300a may include two partitioning subsystems: prediction partitions generator module 105 that may perform analysis and partitioning for prediction, and coding partitions generator module 107 that may perform analysis and partitioning for coding.
• Another partitioning method may include adaptive picture organizer 104 which may segment pictures into regions or slices may also be optionally considered as being part of this partitioner.
• Prediction encoder subsystem 330a of encoder 300a may include motion estimator 122 and characteristics and motion compensated filtering predictor 123 that may perform analysis and prediction of "inter" signal, and intra-directional prediction analyzer and prediction generation module 124 that may perform analysis and prediction of "intra” signal.
• Motion estimator 122 and characteristics and motion compensated filtering predictor 123 may allow for increasing predictability by first compensating for other sources of differences (such as gain, global motion, registration), followed by actual motion compensation. They may also allow for use of data modeling to create synthesized frames (super resolution, and projection) that may allow better predictions, followed by use of actual motion compensation in such frames.
• Transform encoder subsystem 340a of encoder 300a may perform analysis to select the type and size of transform and may include two major types of components.
• the first type of component may allow for using parametric transform to allow locally optimal transform coding of small to medium size blocks; such coding however may require some overhead.
• the second type of component may allow globally stable, low overhead coding using a generic/fixed transform such as the DCT, or a picture based transform from a choice of small number of transforms including parametric transforms.
• PHT Parametric Haar Transform
• Transforms may be performed on 2D blocks of rectangular sizes between 4x4 and 64x64, with actual sizes that may depend on a number of factors such as if the transformed data is luma or chroma, inter or intra, and if the transform used is PHT or DCT.
• the resulting transform coefficients may be quantized, scanned and entropy coded.
• Entropy encoder subsystem 360a of encoder 300a may include a number of efficient but low complexity components each with the goal of efficiently coding a specific type of data (various types of overhead, motion vectors, or transform coefficients).
• Components of this subsystem may belong to a generic class of low complexity variable length coding techniques, however, for efficient coding, each component may be custom optimized for highest efficiency. For instance, a custom solution may be designed for coding of "Coded/Not Coded" data, another for "Modes and Ref Types" data, yet another for "Motion Vector” data, and yet another one for "Prediction and Coding Partitions" data.
• a custom solution may be designed for coding of "Coded/Not Coded" data, another for "Modes and Ref Types" data, yet another for "Motion Vector” data, and yet another one for "Prediction and Coding Partitions" data.
• transform coefficient multiple approaches for efficient handling of specific block sizes, as well as an algorithm that may adapt between multiple tables may
• Filtering encoder subsystem 350a of encoder 300a may perform analysis of parameters as well as multiple filtering of the reconstructed pictures based on these parameters, and may include several subsystems. For example, a first subsystem, blockiness analyzer and deblock filtering module 117 may deblock and dither to reduce or mask any potential block coding artifacts. A second example subsystem, quality analyzer and quality restoration filtering module 118, may perform general quality restoration to reduce the artifacts due to quantization operation in any video coding. A third example subsystem, which may include motion estimator 122 and characteristics and motion compensated filtering predictor module 123, may improve results from motion compensation by using a filter that adapts to the motion characteristics (motion speed/degree of blurriness) of the content.
• a first subsystem, blockiness analyzer and deblock filtering module 117 may deblock and dither to reduce or mask any potential block coding artifacts.
• a second example subsystem, quality analyzer and quality restoration filtering module 118 may perform general quality restoration to
• a fourth example subsystem, prediction fusion analyzer and filter generation module 126 may allow adaptive filtering of the prediction signal (which may reduce spurious artifacts in prediction, often from intra prediction) thereby reducing the prediction error which needs to be coded.
• Encode controller module 103 of encoder 300a may be responsible for overall video quality under the constraints of given resources and desired encoding speed. For instance, in full RDO (Rate Distortion Optimization) based coding without using any shortcuts, the encoding speed for software encoding may be simply a consequence of computing resources (speed of processor, number of processors, hyperthreading, DDR3 memory etc.) availability.
• encode controller module 103 may be input every single combination of prediction partitions and coding partitions and by actual encoding, and the bitrate may be calculated along with reconstructed error for each case and, based on lagrangian optimization equations, the best set of prediction and coding partitions may be sent for each tile of each frame being coded.
• the full RDO based mode may result in best compression efficiency and may also be the slowest encoding mode.
• content analysis parameters from content pre-analyzer module 102 and using them to make RDO simplification (not test all possible cases) or only pass a certain percentage of the blocks through full RDO, quality versus speed tradeoffs may be made allowing speedier encoding.
• VBR variable bitrate
• Encode controller module 103 may also include a rate controller that can be invoked in case of constant bitrate (CBR) controlled coding.
• pre-analyzer subsystem 310a of encoder 300a may perform analysis of content to compute various types of parameters useful for improving video coding efficiency and speed performance. For instance, it may compute horizontal and vertical gradient information (Rs, Cs), variance, spatial complexity per picture, temporal complexity per picture, scene change detection, motion range estimation, gain detection, prediction distance estimation, number of objects estimation, region boundary detection, spatial complexity map computation, focus estimation, film grain estimation etc.
• the parameters generated by preanalyzer subsystem 310a may either be consumed by the encoder or be quantized and communicated to decoder 200.
• subsystems 310a through 380a are illustrated as being associated with specific example functional modules of encoder 300a in Fig. 3(a), other implementations of encoder 300a herein may include a different distribution of the functional modules of encoder 300a among subsystems 310a through 380a.
• the present disclosure is not limited in this regard and, in various examples, implementation of the example subsystems 310a through 380a herein may include the undertaking of only a subset of the specific example functional modules of encoder 300a shown, additional functional modules, and/or in a different arrangement than illustrated.
• FIG. 3(b) is an illustrative diagram of an example next generation video decoder 300b, arranged in accordance with at least some implementations of the present disclosure.
• FIG. 3(b) presents a similar decoder to that shown in FIG. 2, and similar elements will not be repeated for the sake of brevity.
• decoder 300b may include prediction decoder subsystem 330b, filtering decoder subsystem 350b, entropy decoder subsystem 360b, transform decoder subsystem 370b, unpartitioner_2 subsystem 380b, unpartitioner l subsystem 351b, filtering decoder subsystem 350b, and/or post-restorer subsystem 390b.
• Prediction decoder subsystem 330b may include characteristics and motion compensated filtering predictor module 213 and/or intra-directional prediction generation module 214.
• Filtering decoder subsystem 350b may include deblock filtering module 208, quality restoration filtering module 209, characteristics and motion compensated filtering predictor module 213, and/or prediction fusion filtering module 216.
• Entropy decoder subsystem 360b may include adaptive entropy decoder module 202.
• Transform decoder subsystem 370b may include adaptive inverse quantize module 203 and/or adaptive inverse transform module 204.
• Unpartitioner subsystem 380b may include coding partitions assembler 205 and prediction partitions assembler 207.
• Post-restorer subsystem 390b may include content post restorer module 218 and/or adaptive picture re -organizer 217.
• Entropy decoding subsystem 360b of decoder 300b may perform the inverse operation of the entropy encoder subsystem 360a of encoder 300a, i.e., it may decode various data (types of overhead, motion vectors, transform coefficients) encoded by entropy encoder subsystem 360a using a class of techniques loosely referred to as variable length decoding.
• various types of data to be decoded may include "Coded/Not Coded" data, "Modes and Ref Types" data, "Motion Vector” data, "Prediction and Coding Partitions" data, and "Transform
• Transform decoder subsystem 370b of decoder 300b may perform inverse operation to that of transform encoder subsystem 340a of encoder 300a.
• Transform decoder subsystem 370b may include two types of components.
• the first type of example component may support use of the parametric inverse PHT transform of small to medium block sizes, while the other type of example component may support inverse DCT transform for all block sizes.
• the PHT transform used for a block may depend on analysis of decoded data of the neighboring blocks.
• Output bitstream 111 and/or input bitstream 201 may carry information about partition/block sizes for PHT transform as well as in which direction of the 2D block to be inverse transformed the PHT may be used (the other direction uses DCT).
• the partition/ block sizes information may be also retrieved from output bitstream 111 and/or input bitstream 20 land used to apply inverse DCT of appropriate size.
• Unpartitioner subsystem 380b of decoder 300b may perform inverse operation to that of partitioner subsystem 320a of encoder 300a and may include two unpartitioning subsystems, coding partitions assembler module 205 that may perform unpartitioning of coded data and prediction partitions assembler module 207 that may perform unpartitioning for prediction. Further if optional adaptive picture organizer module 104 is used at encoder 300a for region segmentation or slices, adaptive picture re-organizer module 217 may be needed at the decoder.
• Prediction decoder subsystem 330b of decoder 300b may include characteristics and motion compensated filtering predictor module 213 that may perform prediction of "inter" signal and intra-directional prediction generation module 214 that may perform prediction of "intra” signal. Characteristics and motion compensated filtering predictor module 213 may allow for increasing predictability by first compensating for other sources of differences (such as gain, dominant motion, registration) or creation of synthesized frames (super resolution, and projection), followed by actual motion compensation.
• characteristics and motion compensated filtering predictor module 213 may allow for increasing predictability by first compensating for other sources of differences (such as gain, dominant motion, registration) or creation of synthesized frames (super resolution, and projection), followed by actual motion compensation.
• Filtering decoder subsystem 350b of decoder 300b may perform multiple filtering of the reconstructed pictures based on parameters sent by encoder 300a and may include several subsystems.
• the first example subsystem, deblock filtering module 208 may deblock and dither to reduce or mask any potential block coding artifacts.
• the second example subsystem, quality restoration filtering module 209 may perform general quality restoration to reduce the artifacts due to quantization operation in any video coding.
• the third example subsystem, characteristics and motion compensated filtering predictor module 213, may improve results from motion compensation by using a filter that may adapt to the motion characteristics (motion speed/degree of blurriness) of the content.
• the fourth example subsystem, prediction fusion filtering module 216 may allow adaptive filtering of the prediction signal (which may reduce spurious artifacts in prediction, often from intra prediction) thereby reducing the prediction error which may need to be coded.
• Post-restorer subsystem 390b of decoder 300b is an optional block that may perform further improvement of perceptual quality of decoded video. This processing can be done either in response to quality improvement parameters sent by encoder 100, or it can be standalone decision made at the post-restorer subsystem 390b.
• specific parameters computed at encoder 100 that can be used to improve quality at post-restorer subsystem 390b may be estimation of film grain noise and residual blockiness at encoder 100 (even after deblocking).
• the film grain noise if parameters can be computed and sent via output bitstream 111 and/or input bitstream 201 to decoder 200, then these parameters may be used to synthesize the film grain noise.
• post-restorer subsystem 390b may decode these parameters and may use them to optionally perform additional deblocking prior to display.
• encoder 100 also may have access to scene change, spatial complexity, temporal complexity, motion range, and prediction distance information that may help in quality restoration in post-restorer subsystem 390b.
• subsystems 330b through 390b are illustrated as being associated with specific example functional modules of decoder 300b in Fig. 3(b), other implementations of decoder
• 300b herein may include a different distribution of the functional modules of decoder 300b among subsystems 330b through 390b.
• the present disclosure is not limited in this regard and, in various examples, implementation of the example subsystems 330b through 390b herein may include the undertaking of only a subset of the specific example functional modules of decoder 300b shown, additional functional modules, and/or in a different arrangement than illustrated.
• FIG. 4 is an illustrative diagram of modified prediction reference pictures 400, arranged in accordance with at least some implementations of the present disclosure.
• the output of quality analyzer and quality restoration filtering may be a final reconstructed frame that may be used for prediction for coding other frames (e.g., the final reconstructed frame may be a reference frame or the like).
• the proposed implementation of the NGV coder may implement P-picture coding using a combination of Morphed Prediction References 428 through 438 (MRO through 3) and/or Synthesized Prediction References 412 and 440 through
• NGV coding involves use of three picture types referred to as I-pictures, P-pictures, and F/B-pictures.
• the proposed implementation of the NGV coder (e.g., encoder 100 and/or decoder 200) may use one or more of four previously decoded references R0 412, Rl 414, R2 416, and R3 418.
• the proposed implementation of the NGV coder may generate modified (morphed or synthesized) references from such previously decoded references and then use motion compensated coding based at least in part on such generated modified (morphed or synthesized) references.
• the proposed implementation of the NGV coder e.g., encoder 100 and/or decoder 200
• proposed implementation of the NGV coder may include one or more of the following features: 1.
• Gain Compensation e.g., explicit compensation for changes in gain/brightness in a scene
• Blur Compensation e.g., explicit compensation for changes in blur/sharpness in a scene
• Dominant/Global Motion Compensation e.g., explicit compensation for dominant motion in a scene
• Registration Compensation e.g., explicit compensation for registration mismatches in a scene
• Super Resolution e.g., explicit model for changes in resolution precision in a scene
• Projection e -g- > explicit model for changes in motion trajectory in a scene); the like, and/or combinations thereof.
• a characteristics and motion filtering predictor module may apply motion compensation to a current picture 410 (e.g., labeled in the figure as P-pic (curr)) as part of the local decode loop.
• motion compensation may be based at least in part on future frames (not shown) and/or previous frame RO 412 (e.g., labeled in the figure as RO), previous frame Rl 414 (e.g., labeled in the figure as Rl), previous frame R2 416 (e.g., labeled in the figure as R2), and/or previous frame R3 418 (e.g., labeled in the figure as R3).
• prediction operations may include inter- and/or intra-prediction.
• Inter-prediction may be performed by one or more modules including a morphing analyzer and generation module and/or a synthesizing analyzer and generation module.
• a morphing analyzer and generation module may analyze a current picture to determine parameters for changes in blur 420 (e.g., labeled in the figure as Blur par), changes in gain 422 (e.g., labeled in the figure as Gain par and explained in detail below), changes in registration 424 (e.g., labeled in the figure as Reg par), and changes in dominant motion 426 (e.g., labeled in the figure as Dom par), or the like with respect to a reference frame or frames with which it is to be coded.
• blur 420 e.g., labeled in the figure as Blur par
• changes in gain 422 e.g., labeled in the figure as Gain par and explained in detail below
• changes in registration 424 e.g., labeled in
• the determined morphing parameters 420, 422, 424, and/or 426 may be used to generate morphed reference frames. Such generated morphed reference frames may be stored and may be used for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• determined morphing parameters 420, 422, 424, and/or 426 may be used to generate morphed reference frames, such as blur compensated morphed reference frame 428 (e.g., labeled in the figure as MR3b), gain compensated morphed reference frame 430 (e.g., labeled in the figure as MR2g), gain compensated morphed reference frame 432 (e.g., labeled in the figure as MRlg), registration compensated morphed reference frame 434 (e.g., labeled in the figure as MRlr), dominant motion compensated morphed reference frame 436 (e.g., labeled in the figure as MROd), and/or registration compensated morphed reference frame 438 (e.g., labeled in the figure as MROr), the like or combinations thereof, for example.
• blur compensated morphed reference frame 428 e.g., labeled in the figure as MR3b
• a synthesizing analyzer and generation module may generate super resolution (SR) pictures 440 (e.g., labeled in the figure as SO (which is equal to previous frame R0 412), SI, S2, S3) and projected interpolation (PI) pictures 442 (e.g., labeled in the figure as PE) or the like for determining motion vectors for efficient motion compensated prediction in these frames.
• SR super resolution
• PI projected interpolation
• Such generated synthesized reference frames may be stored and may be used for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• the determined morphing parameters 420, 422, 424, and/or 426 may be used to morph the generated synthesis reference frames super resolution (SR) pictures 440 and/or projected interpolation (PI) pictures 442.
• a synthesizing analyzer and generation module may generate morphed registration compensated super resolution (SR) pictures 444 (e.g., labeled in the figure as MR4r, MR5r, and MR6r) and/or morphed registration compensated projected interpolation (PI) pictures 446 (e.g., labeled in the figure as MR7r) or the like from the determined registration morphing parameter 424.
• SR morphed registration compensated super resolution
• PI projected interpolation
• Such generated morphed and synthesized reference frames may be stored and may be used for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• changes in a set of characteristics may be explicitly computed.
• a set of characteristics may be computed in addition to local motion.
• previous and next pictures/slices may be utilized as appropriate; however, in other cases such a set of characteristics may do a better job of prediction from previous picture/slices.
• a modified reference frame associated with the set of characteristics may be selected that yields the best estimate.
• the proposed approach that utilizes modified reference frames associated with the set of characteristics may explicitly compensate for differences in these characteristics.
• the proposed implementation may address the problem of how to improve the prediction signal, which in turn allows achieving high compression efficiency in video coding.
• inter-prediction may be performed.
• up to 4 decoded past and/or future pictures and several morphing/synthesis predictions may be used to generate a large number of reference types (e.g., reference pictures). For instance in 'inter' mode, up to nine reference types may be supported in P-pictures, and up to ten reference types may be supported for F/B-pictures.
• 'multi' mode may provide a type of inter prediction mode in which instead of 1 reference picture, two reference pictures may be used and P- and F/B-pictures respectively may allow 3, and up to 8 reference types.
• prediction may be based on a previously decoded frame generated using at least one of a morphing technique or a synthesizing technique.
• the bitstream may include a frame reference, morphing parameters, or synthesizing parameters associated with the prediction partition.
• One type of morphed prediction used by NGV coding is gain compensated prediction, and includes detecting and estimating the gain and/or offset luminance values, parameterizing them, using them for compensation of gain/offset at the encoder, transmitting them to the decoder, and using them at the decoder for gain compensation by replicating the gain compensation process at the encoder.
• the compensation of interframe changes in brightness can potentially improve compression efficiency in video coding.
• the brightness change parameters gain and offset
• gain and offset are applied both at a video encoder and a decoder so that both should be efficiently communicating with low bit-cost from encoder to decoder via the bitstream and the processing complexity for the decoder should be minimized.
• only techniques for global brightness change have been disclosed, but local compensation in brightness changes have not been successfully addressed.
• the following equation relates brightness of a pixel s t (i,j) at (i,j) location in frame 't' to brightness of a pixel at the same location (i,j) in a previous frame 't- , with 'a' and 'b' being the gain and offset factors. Motion is assumed to be small and only the brightness changes are modeled.
• 'a' and 'b' are calculated as per equation (2), they are quantized (for efficient transmission), encoded and sent to the decoder.
• decoded dequantized values of 'a', and 'b' are put back into equation (1), and using decoded values of pixels in the previous frame, a gain compensated modified version of a previous reference frame is calculated that is lower in error than the original previous frame, and is then used for generating (gain compensated) motion compensated prediction.
• the corresponding predictions from modified previous reference frames are added to generate the final decoded frame (or blocks of the frame).
• a stationary video camera imaging a scene might still result in shaky or unstable video that differs frame to frame due to environmental factors (such as wind), vibrations from nearby objects, a shaky hand, or a jittery capture process, rather than global movement of the scene or motion of large objects in the scene.
• Wiener filtering For computing registration parameters between a current frame and a previous reference frame, Wiener filtering can be employed. Let x(n) be the input signal, y(n) be the output, and h(n) represent filter coefficients.
• h is the vector of filter coefficients.
• the cross-correlation row vector (between source frame and reference frame):
• the autocorrelation matrix (based on block data):
• the Wiener Hopf equation to solve for h as then as follows.
• the Wiener Hopf equation determines optimum filter coefficients in mean square error, and the resulting filter is called the 'wiener' filter.
• a fast camera pan of a scene may, due to charge integration, result in blurry image. Further, even if a camera is still, or in motion, if a scene involves fast moving objects, for instance football players in a football game, the objects can appear blurry as the temporal resolution of the imaging is not sufficient. In both of the aforementioned cases, compensation of blur prior to or in conjunction with other forms of compensation, may improve compression efficiency of video coding.
• a Lucy-Richardson method For motion blur estimation, a Lucy-Richardson method can be used. It is an iterative algorithm for successively computing reduced blur frame (X) at iteration i, from Y the source frame, using B, the blur operator (blur frame using estimated blur vectors) and B* an adjoint operator.
• the operator B* can be roughly thought of as the same as B as B* can be replaced by B resulting in roughly the same visual quality.
• super resolution is a technique used to create a high resolution reconstruction image of a single video frame using many past frames of the video to help fill in the missing information.
• the goal of a good super resolution technique is to be able to produce a reconstructed image better than up-sampling alone when tested with known higher resolution video.
• the super resolution generation technique herein may use coded video codec data to create an in-loop super resolution frame.
• the in-loop super resolution frame is used again within the coding loop as the name implies.
• a process 500 is diagrammed where the principle of generation of SR prediction is applied to P-pictures, which is a type of synthesized prediction used by NGV coding.
• P-pictures which is a type of synthesized prediction used by NGV coding.
• both the encoder and decoder generate the synthesized frame from previously available decoded frames and data.
• a SR frame 518 double the size of frame 'n' 504 in both the horizontal and vertical dimensions is generated by blending upsampled decoded P frame 516 at 'n', and motion compensated picture 514 constructed by using a previous SR frame 508 at ' ⁇ - .
• the previous SR frame 508 is de -interleaved and combined with the motion estimation values at de-interleaved blocks 510 by using the current P-picture 504.
• the blocks 510 are used for motion compensation to form motion compensated, de-interleaved blocks 512, which are then re-interleaved onto a block to form the motion compensated picture 514.
• Multi reference prediction is also shown for the P-picture at frame n+1 by arrow D.
• a picture sequence such as frame sequence 400 may also be used to illustrate the principle of generation and use of projected interpolation frames (PI -pictures) shown as frame PE 442 on FIG. 4.
• PI -pictures projected interpolation frames
• F-pictures behave like B-pictures and can reference two anchors, one in the past, and another in the future (this is only one example case).
• a co-located interpolated frame can be generated by a specific type of interpolation referred to as projected interpolation using the future and the past reference anchor frames. Projected interpolation takes object motion into account that has non- constant (or non-linear) velocity over a sequence of frames, or relatively large motions.
• PI uses weighting factors depending on the distance from the co-located or current frame to be replaced and to each of the two reference frames being used for the interpolation.
• a best fit motion vector is determined that is proportional to these two distances, with the closer reference usually given more weight.
• a two scale factor x factor and y factor
• Further motion compensation may then be allowed to adjust small mismatches.
• a Pi-picture is generated co-located at this time using anchor or reference frames at times 'n' and 'n+2'.
• corresponding Pi-pictures can be generated using anchor frames at times 'n+2' and 'n+5'. This process may repeat for each future F-picture as a Pi-picture is synthesized to correspond in time to each F-picture.
• the corresponding synthesized PI- pictures can then be used as a third reference in the same or similar way the two reference anchors were going to be used for prediction.
• Some prediction partitions may use prediction references directly while others may use them implicitly such as to generate bi-prediction.
• synthesized Pi-pictures can be used for prediction, instead of the original F-pictures, with multi-reference prediction and with two reference anchors.
• the proposed implementation improves video compression efficiency by improving interframe prediction, which in turn reduces interframe prediction difference (error signal) that needs to be coded.
• interframe prediction difference error signal
• the proposed NCV codec may be highly adaptive to changing characteristics (such as gain, blur, dominant motion, registration, resolution precision, motion trajectory, the like, or combinations thereof, for example) of the content by employing, in addition or in the alternative to motion compensation, approaches to explicitly compensate for changes in the characteristics of the content.
• the NGV codec may address a key source of limitation of standards based codecs, thereby achieving higher compression efficiency.
• This change in interframe prediction output may be achieved due to ability of the proposed NCV codec to compensate for a wide range of reasons for changes in the video content.
• Typical video scenes vary from frame to frame due to many local and global changes (referred to herein as characteristics). Besides local motion, there are many other characteristics that are not sufficiently addressed by current solutions that may be addressed by the proposed implementation.
• the proposed implementation may explicitly compute changes in a set of characteristics (such as gain, blur, dominant motion, registration, resolution precision, motion trajectory, the like, or combinations thereof, for example) in addition to local motion, and thus may do a better job of prediction from previous picture/slices than only using local motion prediction from previous and next pictures/slices. Further, since there can be error in any estimation procedure, from multiple past or multiple past and future pictures/slices the NGV coder may choose the frame that yields the best by explicitly compensating for differences in various characteristics.
• characteristics such as gain, blur, dominant motion, registration, resolution precision, motion trajectory, the like, or combinations thereof, for example
• the proposed implementation of the NGV coder may operate so that prediction mode and/or reference type data may be defined using symbol-run coding or a codebook or the like.
• the prediction mode and/or reference type data may be transform encoded using content adaptive or discrete transform in various examples to generate transform coefficients.
• data associated with partitions e.g., the transform coefficients or quantized transform coefficients
• overhead data e.g., indicators as discussed herein for transform type, adaptive transform direction, and/or a transform mode
• data defining the partitions and so on may be encoded (e.g., via an entropy encoder) into a bitstream.
• the bitstream may be communicated to a decoder, which may use the encoded bitstream to decode video frames for display.
• a decoder On a local basis (such as block-by-block within a macroblock or a tile, or on a partition-by-partition within a tile or a prediction unit, or fragments within a superfragment or region) the best mode may be selected for instance based at least in part on Rate Distortion Optimization (RDO) or based at least in part on pre-analysis of video, and the identifier for the mode and needed references may be encoded within the bitstream for use by the decoder.
• RDO Rate Distortion Optimization
• Prediction modes and reference types analyzer 125 may allow for selection of prediction modes from among, “skip”, “auto”, “inter”, “multi”, and “intra” as mentioned above, and for each partition of a tile, all of which may apply to P- and F-pictures; this is shown in Table 1 below.
• prediction modes it also allows for selection of reference types that can be different depending on "inter” or “multi” mode, as well as for P- and F-pictures; the detailed list of ref types is shown in Tables 2(a) and 2(b) for P-pictures, and Tables 3(a), 3(b), 3(c), and 3(d) for F-pictures.
• Tables 1 through 3(d), shown below, illustrate one example of codebook entries for a current frame (curr_pic) being, or that will be, reconstructed.
• a full codebook of entries may provide a full or substantially full listing of all possible entries and coding thereof.
• the codebook may take into account constraints as described above.
• data associated with a codebook entry for prediction modes and/or reference types may be encoded in a bitstream for use at a decoder as discussed herein.
• table 2(b) is directed to a specific combination of references including a past reference without parameters and one of the references on the table as indicated by the table heading.
• proj F refers to PI
• line 8 by one example, includes two optional references.
• Dir refers to a sub-mode that is a fixed, or partially fixed, combination of references for multi-mode for F-frames, such that Dir 0 above, and Dir 1 and Dir 2 below, each refer to a combination of references.
• Dir 0 may refer to a combination of a past reference (which may be a particular reference at a particular time (reference 3 at n+2 for example) and combined with one of the references from the table.
• Dir on the tables below are similar and as explained in the heading of the table.
• inter mode of P-pictures supports a reference type called MROd (morphed reference 0 dominant motion), and for “inter” mode of F-pictures, supported reference types include MROd and MR3d (morphed reference 3 dominant motion). These codes are explained further below. Further, in “multi” mode, MR3d is supported as one of the two references used for a single current frame. Besides “inter” and “multi”, DMC also may be used in "auto" mode of NGV. A summary of modes and reference type combinations where DMC is invoked are as follows.
• FIG. 6 is an illustrative diagram of an example encoder prediction subsystem 330 for performing characteristics and motion compensated prediction, arranged in accordance with at least some implementations of the present disclosure.
• encoder prediction subsystem 330 of encoder 600 may include decoded picture buffer 119, morphing analyzer and generation module 120, synthesizing analyzer and generation module 121, motion estimator module 122, and/or characteristics and motion compensated precision adaptive filtering predictor module 123.
• the output of quality analyzer and quality restoration filtering may be transmitted to decoded picture buffer 119.
• the output of quality analyzer and quality restoration filtering may be a final reconstructed frame that may be used for prediction for coding other frames (e.g., the final reconstructed frame may be a reference frame or the like).
• prediction operations may include inter- and/or intra- prediction. As shown in FIG. 6, inter-prediction may be performed by one or more modules including morphing analyzer and generation module 120, synthesizing analyzer and generation module 121, and/or characteristics and motion compensated precision adaptive filtering predictor module 123.
• Morphing analyzer and generation module 120 may include a morphing types analyzer (MTA) and a morphed pictures generator (MPG) 610 as well as a morphed prediction reference (MPR) buffer 620.
• Morphing types analyzer (MTA) and a morphed pictures generator (MPG) 610 may analyze a current picture to determine parameters for changes in gain, changes in dominant motion, changes in registration, and changes in blur with respect to a reference frame or frames with which it is to be coded.
• the determined morphing parameters may be quantized/de-quantized and used (e.g., by morphing analyzer and generation module 120) to generate morphed reference frames.
• Such generated morphed reference frames may be stored in morphed prediction reference (MPR) buffer 620 and may be used by motion estimator module 122 for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• Synthesizing analyzer and generation module 121 may include a synthesis types analyzer (STA) and synthesized pictures generator (SPG) 630 as well as a synthesized prediction reference (SPR) buffer 640.
• Synthesis types analyzer (STA) and synthesized pictures generator 630 may generate super resolution (SR) pictures and projected interpolation (PI) pictures or the like for determining motion vectors for efficient motion compensated prediction in these frames.
• Such generated synthesized reference frames may be stored in synthesized prediction reference (SPR) buffer 640 and may be used by motion estimator module 122 for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• Motion estimator module 122 may generate motion vector data based at least in part on morphed reference frame(s) and/or super resolution (SR) pictures and projected interpolation (PI) pictures along with the current frame.
• motion estimator module 122 may be considered an inter-prediction module.
• the motion vector data may be used for inter-prediction. If inter-prediction is applied, characteristics and motion filtering predictor module 123 may apply motion compensation as part of the local decode loop as discussed.
• FIG. 7 is an illustrative diagram of an example decoder prediction subsystem 701 for performing characteristics and motion compensated prediction, arranged in accordance with at least some implementations of the present disclosure.
• decoder prediction subsystem 701 of decoder 700 may include decoded picture buffer 210, morphing analyzer and generation module 211, synthesizing analyzer and generation module 212, and/or characteristics and motion compensated precision adaptive filtering predictor module 213.
• the output of quality restoration filtering module may be transmitted to decoded picture buffer (or frame selector control) 210.
• the output of quality restoration filtering module may be a final reconstructed frame that may be used for prediction for coding other frames (e.g., the final reconstructed frame may be a reference frame or the like).
• compensation due to prediction operations may include inter- and/or intra- prediction compensation.
• inter-prediction compensation may be performed by one or more modules including morphing analyzer and generation module 211, synthesizing analyzer and generation module 212, and/or characteristics and motion compensated precision adaptive filtering predictor module 213.
• Morphing analyzer and generation module 211 may include a morphed pictures generator (MPG) 710 as well as a morphed prediction reference (MPR) buffer 720.
• Morphed pictures generator (MPG) 710 may use de-quantized morphing parameters (e.g., determined from input bitstream) to generate morphed reference frames.
• Such generated morphed reference frames may be stored in morphed prediction reference (MPR) buffer 720 and may be used by characteristics and motion compensated precision adaptive filtering predictor module 213.
• Synthesizing analyzer and generation module 212 may include a synthesized pictures generator (SPG) 730 as well as a synthesized prediction reference (SPR) buffer 740.
• Synthesized pictures generator 730 may be configured to generate one or more types of synthesized prediction reference pictures such as super resolution (SR) pictures and projected interpolation (PI) pictures or the like based at least in part on parameters determined from input bitstream 201.
• Such generated synthesized reference frames may be stored in synthesized prediction reference (SPR) buffer 740 and may be used by motion compensated filtering predictor module 213.
• characteristics and motion compensated filtering predictor module 213 may apply motion compensation based at least in part on morphed reference frame(s) and/or super resolution (SR) pictures and projected interpolation (PI) pictures along with the current frame.
• SR super resolution
• PI projected interpolation
• encoder prediction subsystem 330 of encoder 800 may include decoded picture buffer 119, morphing analyzer and generation module 120, synthesizing analyzer and generation module 121, motion estimator module 122, and/or characteristics and motion compensated precision adaptive filtering predictor module 123.
• the output of quality analyzer and quality restoration filtering may be transmitted to decoded picture buffer 119.
• the output of quality analyzer and quality restoration filtering may be a final reconstructed frame that may be used for prediction for coding other frames (e.g., the final reconstructed frame may be a reference frame or the like).
• prediction operations may include inter- and/or intra- prediction. As shown in FIG. 8, inter-prediction may be performed by one or more modules including morphing analyzer and generation module 120, synthesizing analyzer and generation module 121, and/or characteristics and motion compensated precision adaptive filtering predictor module 123.
• Morphing analyzer and generation module 120 may include a morphing types analyzer (MTA) and a morphed pictures generator (MPG) 610 as well as a morphed prediction reference (MPR) buffer 620. Morphing types analyzer (MTA) and a morphed pictures generator (MPG) 610 may be configured to analyze and/or generate one or more types of modified prediction reference pictures.
• MTA morphing types analyzer
• MPG morphed pictures generator
• MPR morphed prediction reference
• Morphing types analyzer (MTA) and a morphed pictures generator (MPG) 610 may be configured to analyze and/or generate one or more types of modified prediction reference pictures.
• morphing types analyzer (MTA) and a morphed pictures generator (MPG) 610 may include Gain Estimator and Compensated Prediction Generator 805, Blur Estimator and Compensated Prediction Generator 810, Dominant Motion Estimator and Compensated Prediction Generator 815, Registration Estimator and Compensated Prediction Generator 820, the like and/or combinations thereof.
• Gain Estimator and Compensated Prediction Generator 805 may be configured to analyze and/or generate morphed prediction reference pictures that are adapted to address changes in gain.
• Blur Estimator and Compensated Prediction Generator 810 may be configured to analyze and/or generate morphed prediction reference pictures that are adapted to address changes in blur.
• Global Motion Estimator and Compensated Prediction Generator 815 may be configured to analyze and/or generate morphed prediction reference pictures that are adapted to address changes in dominant motion. Specifically, the global motion estimator and compensated prediction generator 815 is used for computation of global motion parameters (dp) and applying them on a picture from the DPR buffers 119 to generate a GMC Morphed Reference Picture that is stored in one of the MPR Picture Buffers (Local/Picture Buffers for Dominant Motion Compensated Prediction). The output of that is used for block motion estimation and compensation.
• Registration Estimator and Compensated Prediction Generator 820 may be configured to analyze and/or generate morphed prediction reference pictures that are adapted to address changes in registration.
• Morphing types analyzer (MTA) and a morphed pictures generator (MPG) 610 may store such generated morphed reference frames in morphed prediction reference (MPR) buffer 620.
• morphed prediction reference (MPR) buffer 620 may include Gain Compensated (GC) Picture/s Buffer 825, Blur Compensated (BC) Picture/s Buffer 830, Dominant Motion Compensated (DC) Picture/s Buffer 835, Registration Compensated (RC) Picture/s Buffer 840, the like and/or combinations thereof.
• Gain Compensated (GC) Picture/s Buffer 825 may be configured to store morphed reference frames that are adapted to address changes in gain.
• Blur Compensated (BC) Picture/s Buffer 830 may be configured to store morphed reference frames that are adapted to address changes in blur.
• Dominant Motion Compensated (DC) Picture/s Buffer 835 may be configured to store morphed reference frames that are adapted to address changes in dominant motion.
• Registration Compensated (RC) Picture/s Buffer 840 may be configured to store morphed reference frames that are adapted to address changes in registration.
• Synthesizing analyzer and generation module 121 may include a synthesis types analyzer (ST A) and synthesized pictures generator 630 as well as a synthesized prediction reference (SPR) buffer 640.
• Synthesis types analyzer (ST A) and synthesized pictures generator 530 may be configured to analyze and/or generate one or more types of synthesized prediction reference pictures.
• synthesis types analyzer (STA) and synthesized pictures generator 630 may include Super Resolution Filter Selector & Prediction Generator 845, Projection Trajectory Analyzer & Prediction Generator 850, the like and/or combinations thereof.
• Super Resolution Filter Selector & Prediction Generator 845 may be configured to analyze and/or generate a super resolution (SR) type of synthesized prediction reference pictures.
• Projection Trajectory Analyzer & Prediction Generator 850 may be configured to analyze and/or generate a projected interpolation (PI) type of synthesized prediction reference pictures.
• Synthesis types analyzer (STA) and synthesized pictures generator 630 may generate super resolution (SR) pictures and projected interpolation (PI) pictures or the like for efficient motion compensated prediction in these frames.
• Such generated synthesized reference frames may be stored in synthesized prediction reference (SPR) buffer 640 and may be used by motion estimator module 122 for computing motion vectors for efficient motion (and characteristics) compensated prediction of a current frame.
• SPR synthesized prediction reference
• synthesized prediction reference (SPR) buffer 640 may include Super Resolution (SR) Picture Buffer 855, Projected Interpolation (PI) Picture Buffer 860, the like and/or combinations thereof.
• Super Resolution (SR) Picture Buffer 855 may be configured to store synthesized reference frames that are generated for super resolution (SR) pictures.
• Projected Interpolation (PI) Picture Buffer 860 may be configured to store synthesized reference frames that are generated for projected interpolation (PI) pictures.
• Motion estimator module 122 may generate motion vector data based on morphed reference frame(s) and/or super resolution (SR) pictures and projected interpolation (PI) pictures along with the current frame.
• motion estimator module 122 may be considered an inter-prediction module.
• the motion vector data may be used for inter-prediction. If inter-prediction is applied, characteristics and motion filtering predictor module 123 may apply motion compensation as part of the local decode loop as discussed.
• the prediction mode analyzer 125 (or Pred Modes & Ref Types Analyzer & Selector), and as explained above, chooses on a local (block, tile or partition) basis the best prediction from among various type of inter modes and intra mode.
• inter is being used in generality and includes 'inter' mode, 'multi' mode, 'auto' mode and 'skip' modes.
• the chosen mode (and sub-mode if applicable), morphing or synthesis parameters ⁇ dp, gp, rp, sp, pp), reference info, and motion (mv, ⁇ ) and other data is entropy coded as explained above and sent as part of an encoded bitstream to the decoder.
• FIG. 9 is an illustrative diagram of another example decoder prediction subsystem 701 for performing characteristics and motion compensated prediction, arranged in accordance with at least some implementations of the present disclosure.
• decoder prediction subsystem 701 may include decoded picture buffer 210, morphing analyzer and generation module 21 1 , synthesizing analyzer and generation module 212, and/or characteristics and motion compensated precision adaptive filtering predictor module 213.
• the output of quality restoration filtering module may be transmitted to decoded picture buffer 210.
• the output of quality restoration filtering module may be a final reconstructed frame that may be used for prediction for coding other frames (e.g., the final reconstructed frame may be a reference frame or the like).
• compensation due to prediction operations may include inter- and/or intra-prediction compensation.
• inter-prediction compensation may be performed by one or more modules including morphing analyzer and generation module 21 1 , synthesizing analyzer and generation module 212, and/or characteristics and motion compensated precision adaptive filtering predictor module 213.
• Morphing generation module 212 may include a morphed pictures generator (MPG) 710 as well as a morphed prediction reference (MPR) buffer 720.
• Morphed pictures generator (MPG) 710 may use de-quantized morphing parameters (e.g., determined from input bitstream) to generate morphed reference frames.
• morphed pictures generator (MPG) 710 may include Gain Compensated Prediction Generator 905, Blur Compensated Prediction Generator 910, Dominant Motion Compensated Prediction Generator 915, Registration Compensated Prediction Generator 920, the like and/or combinations thereof.
• Gain Compensated Prediction Generator 905 may be configured to generate morphed prediction reference pictures that are adapted to address changes in gain as described in greater detail below.
• Blur Compensated Prediction Generator 910 may be configured to generate morphed prediction reference pictures that are adapted to address changes in blur.
• Dominant Motion Compensated Prediction Generator 915 may be configured to generate morphed prediction reference pictures that are adapted to address changes in dominant motion.
• Registration Compensated Prediction Generator 920 may be configured to generate morphed prediction reference pictures that are adapted to address changes in registration.
• Morphed pictures generator (MPG) 710 may store such generated morphed reference frames in morphed prediction reference (MPR) buffer 720.
• MPR morphed prediction reference
• morphed prediction reference (MPR) buffer 720 may include Gain Compensated (GC) Picture/s Buffer 925, Blur
• BC Picture/s Buffer 930
• Dominant Motion Compensated (DC) Picture/s Buffer 935 Registration Compensated (RC) Picture/s Buffer 940, the like and/or combinations thereof.
• Gain Compensated (GC) Picture/s Buffer 925 may be configured to store morphed reference frames that are adapted to address changes in gain.
• Blur Compensated (BC) Picture/s Buffer 930 may be configured to store morphed reference frames that are adapted to address changes in blur.
• Dominant Motion Compensated (DC) Picture/s Buffer 935 may be configured to store morphed reference frames that are adapted to address changes in dominant motion.
• Registration Compensated (RC) Picture/s Buffer 940 may be configured to store morphed reference frames that are adapted to address changes in registration.
• Synthesizing generation module 212 may include a synthesized pictures generator 630 as well as a synthesized prediction reference (MPR) buffer 740.
• Synthesized pictures generator 730 may be configured to generate one or more types of synthesized prediction reference pictures such as super resolution (SR) pictures and projected interpolation (PI) pictures or the like based on parameters determined from input bitstream 201.
• Such generated synthesized reference frames may be stored in synthesized prediction reference (SPR) buffer 740 and may be used by motion compensated filtering predictor module 213.
• synthesized pictures generator 730 may include Super Resolution Picture Generator 945, Projection Trajectory Picture Generator 950, the like and/or combinations thereof.
• Super Resolution Picture Generator 945 may be configured to generate a super resolution (SR) type of synthesized prediction reference pictures.
• Projection Trajectory Picture Generator 950 may be configured to generate a projected interpolation (PI) type of synthesized prediction reference pictures.
• Synthesized pictures generator 730 may generate super resolution (SR) pictures and projected interpolation (PI) pictures or the like for efficient motion compensated prediction in these frames.
• Such generated synthesized reference frames may be stored in synthesized prediction reference (SPR) buffer 740 and may be used by characteristics and motion compensated filtering predictor module 213 for efficient motion (and characteristics) compensated prediction of a current frame.
• SPR synthesized prediction reference
• synthesized prediction reference (SPR) buffer 740 may include Super Resolution (SR) Picture Buffer 955, Projected Interpolation (PI) Picture Buffer 960, the like and/or combinations thereof.
• Super Resolution (SR) Picture Buffer 955 may be configured to store synthesized reference frames that are generated for super resolution (SR) pictures.
• Projected Interpolation (PI) Picture Buffer 960 may be configured to store synthesized reference frames that are generated for projected interpolation (PI) pictures.
• characteristics and motion compensated filtering predictor module 213 may apply motion compensation based on morphed reference frame(s) and/or super resolution (SR) pictures and projected interpolation (PI) pictures along with the current frame.
• SR super resolution
• PI projected interpolation
• NGV video coding addresses limitations of the current state of the art by novel approaches to content partitioning, content adaptive prediction, and content adaptive transform coding.
• content adaptive prediction it includes a more sophisticated approach to global motion compensation as compared to the MPEG-4, part2 standard based technique discussed earlier.
• GMC global motion compensation
• an original or current video picture 1002 to be coded (on the right) has a foreground object 1004 (a large star shaped object) and a background 1006.
• a dominant motion compensated (DMC) picture 1000 (on the left and also referred to as the decoded reference frame) is first created by forming a GMC morphed picture 1008 rounded or fit within a rectangle 1010 as explained below.
• a delta correction motion vector (Amvx, Amvy) 1016 then may 'fine tune' an adjusted (or morphed or warped) position of the foreground object 1012 to a final position 1014.
• the motion vectors shown herein point from the current frame 1002 to the corresponding position on the reference frame to show where the region, portion, or block comes from as per usual coding diagrams.
• a single delta correction motion vector 1016 of the foreground star shaped object 1014 is shown, in reality there may be at least two delta correction motion vectors at work since one motion vector could be used for the background 1018 (including a zero delta motion vector), and another motion vector used for the foreground (such as the shown delta motion vector).
• a single motion vector may be used on a block (such as a macroblock or larger), on a tile (such as a 64 x 64 rectangle or larger), or other partition or portion of a frame that may or may not be formed by grouping blocks, tiles, or other units together.
• an example process 1100 is a computer implemented method of video coding, and specifically, to perform dominant motion compensation.
• the process 1100 is arranged in accordance with at least some implementations of the present disclosure.
• Process 1100 may include one or more operations, functions or actions as illustrated by one or more operations.
• Process 1100 may form at least part of a next generation video coding process.
• process 1100 may form at least part of a next generation video encoding process as undertaken by coder system 100 and 200 of FIGS. 1-2 or dominant motion compensation coder sub-systems 1800 and 1900 of FIGS. 18-19, and/or any other coder system or subsystems described herein.
• Process 1100 may begin with "obtaining frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame” 1102. Thereafter, the process 1100 may comprise "forming a warped global compensated reference frame by displacing at least one portion of the decoded reference frame by using global motion trajectories". This is explained in detail below.
• the at least one portion may refer to single portion of the frame, many portions, or the entire frame.
• the portion may be a block, a tile such as a coding tree block, and/or a region or partition of the frame.
• the region may or may not be associated with an object in the frame (or in other words, an object shown on the image the frame provides), and may or may not have a boundary that is shaped like the object.
• the process 1100 may also comprise "determining a motion vector indicating the motion of the at least one portion and motion from a position based on the warped global compensated reference frame to a position at the current frame". This may be performed by motion estimation calculations.
• the process 1100 may also include "forming a prediction portion based, at least in part, on the motion vectors and corresponding to a portion on the current frame".
• motion vectors may be applied to adjust the position of the block, tile, region, or object, before the pixel values are used in that portion to form a prediction that may be compared to the corresponding area of an original frame to determine if there is any residual that warrants coding.
• dominant motion compensation includes performing local global motion compensation on portions that are less than an entire frame before using the pixel values as predictions and, in one approach, without determining motion vectors at least for that portion.
• an example process 1200 is arranged in accordance with at least some implementations of the present disclosure.
• Process 1200 may include one or more operations, functions or actions as illustrated by one or more operations.
• Process 1200 may form at least part of a next generation video coding process.
• process 1200 may form at least part of a next generation video encoding process as undertaken by coder system 100 and 200 of FIGS. 1-2, or dominant motion compensation coder sub-systems 2400 and 2500 of FIGS.
• the process 1200 may be a computer-implemented method for video coding, and comprises "obtaining frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame" 1202.
• the process 1200 also may include "dividing the reference frame into a plurality of portions that are less than the area of the entire frame" 1204.
• the frame may be divided into portions that are a uniform unit such as a block or a tile such as a coding tree block, and so forth.
• the portion may be object based, such as a foreground, a background, a moving object in the frame, or any other object in the frame.
• the process 1200 also may include "performing dominant motion compensation comprising applying local global motion compensation on at least one of the portions by displacing the at least one portion of the decoded reference frame by using global motion trajectories at a boundary of the portion" 1206.
• global motion trajectories may be placed at corners of each, or selected, portions on the frame.
• the process 1200 also may include "form a prediction portion that corresponds to a portion on the current frame, and by using the pixel values of the displaced portion" 1208.
• the pixel values may be used directly from the warped GMC picture without the use of motion vectors.
• the local GMC then provides greater accuracy than applying GMC once to the entire frame.
• Process 1300 may include one or more operations, functions or actions as illustrated by one or more operations 1302 to 1328 numbered evenly.
• Process 1300 may form at least part of a next generation video coding process.
• process 1300 may form at least part of a next generation video encoding process as undertaken by coder system 100 or 200 of FIGS. 1- 2 or gain compensation coder sub-systems 1800 or 1900 of FIGS. 18-19, and/or any other coder system or subsystems described herein.
• Process 1300 first may include first obtaining frames of pixel data and having a current frame and a decoded reference frame 1302.
• a video stream may be provided to an encoder that has a decoding loop 135 in order to find residuals and provide the quantized residuals to a decoder.
• frames may be decoded, and used as decoded reference frames to predict yet other frames.
• a morphing unit such as unit 120 may be used to determine which frames are to be modified or morphed by dominant motion compensation. These frames may already be divided into units such as macroblocks, prediction blocks, and so forth.
• process 1300 may comprise create global motion compensation (GMC) warped frames 1304.
• GMC global motion compensation
• One form of the principle of generation of a GMC (morphed) picture given a decoded reference picture 1400 and global motion trajectories is as follows.
• GMC using an affine model involves use of six parameters that are encoded as three motion trajectories 1404, 1406, and 1408 with one corresponding to each of the three corners of the reference picture 1400, and the fourth corner treated as unconstrained.
• Motion trajectories may be created 1306 by processes that are well understood.
• the trajectories may be applied 1308 also by processes as understood or by using the equations provided herein as explained below.
• the resulting GMC warped frame or picture 1402 appears warped compared to the reference picture 1400.
• a Ref Picture 'rectangle' results in a quadrilateral GMC morphed or warped picture 1402 as shown when applying the GMC parameter equations.
• the quadrilateral itself is not referred to as a reference frame yet.
• a GMC morphed reference picture or frame 1500 may be formed 13 10 from the GMC morphed picture 1402. This is performed to provide a frame size for ease of computations and comparisons to the original frame. This may include creating a larger padded rectangle 1500 encompassing the GMC morphed picture (a trapezoid) 1402 using the top left coordinate as the reference point (or starting point or connection point). The area outside the quadrilateral 1402 but inside the rectangle 1500 may be filled by padding 1506 which consists of simply copying pixels from the right, top and bottom edges of the quadrilateral 1402, except for corner pixels (areas of overlap) that may be filled by extending both horizontally and vertically and averaging pixels.
• padding 1506 which consists of simply copying pixels from the right, top and bottom edges of the quadrilateral 1402, except for corner pixels (areas of overlap) that may be filled by extending both horizontally and vertically and averaging pixels.
• this GMC morphed reference frame is used for motion compensation going forward. It will be understood that the rectangle formed based on the warped picture itself also may be referred to herein as the warped reference frame since it includes warped pixel locations of an image.
• a virtual GMC morphed picture 1600 may optionally be formed 1312 to proceed with the motion compensation. This may be provided when the system has sufficient compute power to handle division efficiently since such further warping results in significant computation load. Otherwise, the motion compensation may continue with the warped reference rectangle 1500 as explained above.
• the reference picture may be extended to generate virtual reference picture 1600 such that the width and height becomes a power of two. For example if the reference picture 1500 has a width of 720 and height of 480, then the virtual ref picture would have a width of 1024 and height of 512.
• motion trajectories 1602, 1604, and 1606 may be computed for each of the three vertices (with the fourth vertex being unconstrained), and applied to the vertices of the virtual reference picture 1600 (rather than being applied to the reference picture 1400 as done before).
• the resulting warped quadrilateral is also shown and referred to as the virtual GMC morphed picture 1600.
• affine transform process is described by the following equations that use affine parameters a, b, c, d, e,f to map a set of points (x, y) in a previous picture to a modified set of points ( ⁇ ', y').
• x.' a ⁇ x. + b ⁇ v. + c (10)
• y i ' d ⁇ Xi + e - y i +f (11)
• Equations (10) and (11) effectively modify or morph the reference frame so it can then be used for more efficient motion compensation for a current frame being analyzed.
• This model is transmitted as three motion trajectories, one for top-left corner of the picture, one for top-right corner of the picture, and one for bottom-left corner of the picture.
• Affine parameters are calculated (fixed point arithmetic) for a virtual picture which is assumed to be of width and height of nearest power of 2 number which is greater than the coded picture. This removes division operations at the decoder.
• the affine parameters A, B, C, D, E, F can then be calculated as follows.
• Process 1300 also may include define frame portions for motion vectors 1304.
• motion vectors are provided on a block-by-block basis, and may be defined prediction blocks 1306 such as macroblocks or other prediction or coding units.
• the frame may be divided into tiles, which may be blocks of 64 x 64 pixels or more.
• the tiles may be grouped into regions associated with an object such as being part of the background or foreground.
• a motion vector is then determined for each region.
• the regions may be defined directly without any initial division of large tiles or blocks although the boundary to such regions may be defined to fit small blocks (such as 4 x 4 pixels).
• block-based, dominant motion compensation of where each block uses a delta motion vector (MV) correction with respect to an affine GMC reference picture includes first using 1316 or obtaining blocks such as prediction macrob locks that may be 16 x 16 pixels, or other size block units of a current frame that are larger as shown, and that are displaced on a warped reference frame.
• MV delta motion vector
• a current frame 1702 with a star shaped object 1704 to be coded may be divided into blocks A2, B2, C2 for coding, and a GMC Morphed Reference frame 1700 may be derived using a past decoded reference frame and GMC motion trajectories that formed the warped quadrilateral 1708.
• the blocks of pixels A2, B2, C2 from the current frame 1702 are matched during motion estimation to find closest matches at Al, Bl, and CI respectively.
• the first block match is offset by delta motion vector Amvl (1714)
• the second block match is offset by delta motion vector Amv2 (1716)
• the third block match is offset by delta motion vector Amv3 (1718) in the warped and padded GMC Reference picture 1700.
• blocks While only three blocks are shown, it is assumed that the entire picture may be divided into blocks on a block grid, and for each block, a delta motion vector can be computed that provides the best match in the GMC reference picture 1700. Also, while blocks are shown to be of medium size, generally the blocks can be large, small, or each block can be one of a few permitted sizes, and whatever size may be needed to provide the right tradeoff between the reduction in Motion Compensated Prediction error versus the cost of extra delta motion vector information that needs to be coded and transmitted.
• motion estimation may be performed 1326 to determine delta motion vectors based on the warped position of the portion of the frame, which is a block in this case.
• different blocks may use different coding modes such as DMC, Gain, Register, SR, PI, reference no parameters, or intra as described above, and that maximize the reduction in prediction error with regard to coding cost.
• the delta motion vectors may be kept at 1 ⁇ 4 or 1/8 th pel precision.
• the motion vectors identified herein as (Amv) can be coded efficiently with prediction following the method similar to coding of normal motion vectors.
• the process 1300 used with blocks as described herein may not be referred to as GMC prediction since it uses delta motion vectors pointed to source blocks at a GMC reference picture 1700. Rather, this is considered a type of dominant motion compensation (DMC), and may be referred to motion vector DMC.
• DMC dominant motion compensation
• Other forms of DMC exist as described below. This difference (between GMC and DMC) here, however, is not minor. It forms an adjustment to the pixel locations that may significantly decrease prediction error over known GMC providing more efficient coding.
• warped reference frame 1700 does not require separate knowledge of a foreground or a background object, while the process illustrated for a warped reference frame 1700 still extends the principle of GMC to DMC.
• the process 1300 may continue with form prediction 1328 (or specifically a prediction portion or in this case a prediction block) using the portion identified by the motion vector.
• form prediction 1328 or specifically a prediction portion or in this case a prediction block
• a simple technique such as bilinear interpolation may be used for generating the necessary DMC prediction block. More sophisticated methods can also be used as follows:
• A, B, C, D, E, & F are affine parameters calculated from the three motion trajectories transmitted.
• x (A*j + B*i + C « r) » r (20)
• y (D*j + E*i + F « s) » s (21)
• (j, i) is current pixel location (on the current frame being analyzed)
• « and » are left and right bitwise shifts
• (x, y) is the reference pixel coordinate in l/8 th Pel accuracy on the morphed or modified reference frame.
• M ef[i]£j] ((8-p x )*(8-p y )*RefIy 0 ][xo] + p x *(8-p y )*RefIy 0 ][xo+l] + p y *(8-p x )*Ref[y 0 +l][x 0 ] + p y *p x *RefIy 0 +l][xo+l] + 31) » 6 (26) where MRef is the morphed reference frame, and recited in a different form:
• (iMVx, iMVy) is the transmitted motion vector in Sub-Pel Unit (fs) for a block at (j, i) of size (W b x H b ).
• A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted.
• fs is the Sub-Pel Factor (e.g.
• MRef is the morphed reference frame
• tPred h is the intermediate Horizontal Interpolation
• Pred j i is the final Motion Compensated Morphed Reference Prediction.
• A, B, C, D, E, & F are affine parameters calculated from the three Motion trajectories transmitted.
• x and y are reference pixel coordinates in l/8 th Pel accuracy.
• (iMVx, iMVy) is the transmitted Motion Vector in Sub-Pel Units (fs) for a block at (j,i) of size (W b x H b ).
• A, B, C, D, E, & F are affine parameters calculated from the three motion trajectories transmitted.
• i' (i + u*H s )*fs + iMVx (72)
• j' (j+v*W s )*fs + iMVy (73)
• (j, i) is the current block pixel location
• (u, v) is the index of every (W s x H s ) sub-block within given current block of (W b x H b )
• (W s x H s ) sub-block is typically 4 x 4, 8 x 4, or 8 x 8.
• i',j' is motion adjusted current pixel location in fs sub-pel accuracy.
• y 0 y&(fs-l) (76)
• p x x & (fs-1) (77)
• y 0 y / fs (78) x/fs (79)
• y 0 , x 0 is the integer pel location in Ref Image, px, py is the l/8 th pel phase
• n [0,W s -l]
• the best prediction is chosen among the alternatives that are calculated for a particular portion or block, if any.
• the pixel values of the best prediction are compared to those corresponding pixel values of the original frame, and the difference, if any, is a residual that is coded and transmitted to the decoder.
• a portion or sub-system of a NGV or modified HEVC encoder 1800 uses the block-based type of DMC prediction 1316 using delta motion vectors as described above. While HEVC standard does not support Global (or Dominant) Motion Compensation, or Morphed or Synthesized references, it does support plain references and can be modified to the encoder sub-system 1800.
• encoder 1800 provides a simplified representation that focuses on DMC prediction.
• decoded frames obtained from deblock filtering and QR filtering are stored in Decoded Prediction Reference (DPR) Picture Buffers 1802 (also referred to as multi reference frame store 119 of FIG. 1) for use by morphed reference analysis and generation logic or synthesized reference analysis and generation logic.
• DPR Decoded Prediction Reference
• the details of other components employed for morphing are omitted so that here the morphed reference generation logic is simply labeled as Other Morph Analyzer, Generator & Picture/s Buffer 1808.
• synthesized reference analysis and generation are hidden so the synthesized reference analysis and generation logic is labeled as Other Synth Analyzer, Generator & Picture/s Buffer 1810.
• GMC/DMC components are separated and shown explicitly as compared to other morphing related components and logic.
• the main components are Global or Dominant Motion Estimator & Compensated Prediction Picture/Tile/Partition Generator 1804, Dominant Motion Compensation Local/Picture Buffer 1806, and an interpolation subset 1815 of a Motion Compensated Filtering Predictor 1814, whereas the control logic includes global motion trajectory information gmt and delta motion vectors Amvs.
• components of other alternative (non-block based) processes for defining frame portions to be used with motion vectors are shown in dashed line and are described later below.
• decoded and filtered picture/s stored in DPR buffer 1802 are input to the Global or Dominant Motion Estimator & Compensated Prediction Picture/Tile/Part.
• Generator 1804 that, in the present block-based alternative, performs global motion estimation (GME) producing global motion parameters (represented as gmt trajectory) for an entire frame (where the trajectories are located at the corners of the frame) and generating GMC Ref Picture(s) or regions that are stored in Local/Picture Buffer 1806 for Dominant Motion Compensation.
• GME global motion estimation
• the block motion estimator and partitions motion assembler 1812 performs block based motion estimation resulting in motion vectors or delta motion vectors of blocks (or partitions), and these motion vectors are used by motion compensated predictor 1814 (referred to here as Bi-Tree Partitions Char and Motion Compensated Adaptive Precision Filtering Predictor) that generates prediction blocks by sub-pixel interpolation using interpolation unit 1815.
• motion compensated predictor 1814 referred to here as Bi-Tree Partitions Char and Motion Compensated Adaptive Precision Filtering Predictor
• the alternative choices of various sub-pixel interpolators are explained above (alternatives 1 to 4).
• the output prediction blocks (partitions) from motion compensated predictor 1814 are fed to the Prediction Modes and Reference Types Analyzer 1816.
• intra prediction blocks are input to the Prediction Modes and reference types analyzer 1816 from an intra directional prediction analyzer and generator 1818.
• the Pred Modes & Ref Types Analyzer 1816 determines the best prediction block from various choices (e.g. DMC prediction is one of the many choices available) and outputs it to a differencer (in portion of the circuit not shown here) that generates a prediction error for coding.
• the entropy coder 1820 also referred to as the Entropy Encoder Morphing and Synthesis Parameters & MVs, encodes GMC/DMC parameters and data, i.e., gmt, and Amvs.
• modes and reference information that identifies if a block (partition) uses DMC or some other morphed or synthesized prediction type (or intra type) and further, the picture to use as reference is not shown.
• the encoder sub-system 1800 can be relatively extended or modified for other video compression techniques.
• the DMC approach discussed here can also be made to work with by first extending current H.264, and the upcoming HEVC standards.
• the HEVC standard does not support GMC or any of the morphing or synthesized prediction modes of NGV, but it does support multiple reference based prediction so before DMC based improved delta mvs can be added, GMC would need to be added first to the HEVC standard.
• NGV codec uses Tiles and Bi-Tree partitioning for motion compensation, but HEVC uses the concept of Coding Tree Blocks (CTB) or Largest Coding Unit (LCU), and quadtree partitioning into Coding Units (CUs) and small codebook of 8, based partitioning into Prediction Units (PUs), that are functionally similar although they have outwardly different processing structures.
• CTB Coding Tree Blocks
• LCU Largest Coding Unit
• CUs Coding Unit
• PUs Prediction Units
• Mode and ref type information is available in HEVC but would need to be extended to support GMC/DMC extensions.
• a portion or subsystem of NGV or modified HEVC decoder 1900 may be complementary to the related components of the encoder 1800. As compared to decoder 200 with which it essentiality shares the same or similar functionality, decoder 1900 shows a simpler representation that focuses on DMC prediction. Decoder 1900 uses Dominant Motion Compensation of Blocks and by using delta MV's correction. HEVC standard does not support Global (or Dominant) Motion Compensation, or Morphed or Synthesized references, but does support plain references.
• the entropy decoder 1901 (also called Entropy Decoder Morphing and Synthesis Params & MVs) first decodes morphing and synthesis parameters, motion vectors, delta motion vectors (shown), and modes and reference type decisions (not shown). Decoded and filtered picture/s may be stored in DPR buffer 1902 and are input to Global or Dominant Motion Compensated Prediction Picture/Tile/Partition Generator 1904 that uses decoded gmt parameters to generate GMC Reference Picture(s) or regions that are then stored in a Dominant Motion Compensated Local/Picture Buffer 1906.
• a motion compensated prediction unit 1912 also called Bi-Tree Partitions Char and Motion Compensated Adaptive Precision Filtering Predictor, is specific to NGV coding, but in general any type of motion compensated predictor could have been used.
• the set of prediction- interpolation alternatives (1 to 4) described above may be used by an interpolator unit or subsystem 1915 of the prediction unit 1912.
• the motion compensated prediction unit 1912 Upon decoding, use of the delta motion vectors by the motion compensated prediction unit 1912 results in dominant motion compensated blocks (partitions) that are sent to the Prediction Mode Selector 1914, along with predictions from an intra directional prediction generator 1916, which uses decoded reference type and mode info to output, for each or multiple block (partition), the best prediction block (partition). This is identical, or similar, to the prediction process in the local prediction loop at the encoder (the encoder also uses analysis/estimation process that is not part of the decoder).
• decoder 1900 Similar to encoder 1800, decoder 1900 also has a simplified view of other morphing components. Thus, Other Morph. Generator & Picture/s Buffer 1908) and parameters (mop), and synthesis components Other Synth Generator & Picture/s Buffer 1910 and parameters (syp) are not elaborated upon here. Further, as with encoder 1800, the portion of decoder 1900 shown can be adapted to work with extended H.264, or HEVC video standards. As mentioned earlier, the HEVC standard does not include morphing and synthesis types of the NGV codec and only shares the commonality of multiple reference frame prediction, so the HEVC standard would have to be extended to support the DMC prediction modes. The NGV and the HEVC standard also differ in partitionings employed. That difference, however, is mainly superficial since at a functional level the processes employed are similar.
• the process 1300 may alternatively include divide 1318 the frame into tiles, and then group 1322 the tiles into regions, before applying the delta motion vectors (note herein motion vectors may mean delta motion vectors depending on the context).
• motion vectors may mean delta motion vectors depending on the context.
• block based delta motion vector type of DMC approach discussed via warped reference frame 1700 may be simple, it is not as efficient as it could be regarding overhead management since delta motion vectors are provided on a block basis in addition to the cost of coding global GMC trajectory (gmt) parameters.
• warped reference frames 2000, 2100, and 2200 Three variations of warped reference frames 2000, 2100, and 2200, are provided to show the use of delta motion vectors for DMC on other portions of a frame rather than, or in addition to, a block-by-block basis.
• a warped reference frame 2000 (FIG. 20) may be used to provide dominant motion compensation of approximate region-layers having a group of merged Tiles or CTBs, where each region uses a delta MV correction with respect to an affine GMC reference picture.
• a significant reduction in overhead may be obtained by using larger tiles instead of blocks, and more importantly, grouping of tiles together to further reduce the number of delta motion vectors that need to be sent.
• a warped reference 2000 may have been formed from a warped quadrilateral 2008 based on a decoded reference frame, and is used as a reference for a current frame 2002 to be coded.
• Both the current frame 2002 and the warped reference frame 2000 may be divided into tiles, and here nine tiles are shown, but the number of tiles may be different.
• the tiles may be a 64 x 64 array of luma pixels and corresponding chroma pixels.
• the tiles may be larger as well.
• the tiles may grouped as foreground tiles 2004 and background tiles 2006 on the current frame.
• the position of the tiles at the warped reference frame are still adjusted by the delta motion vector tile-by-tile such as for tile 2010 or 2012, except now, the same delta motion vector is used for each tile in the group or region.
• all foreground tiles have the same vector and all the background tiles have the same vector. This may depend on how the computations proceed and very well may be the same as, or stated as, moving the entire region together with the single motion vector.
• the approach discussed thus far also applies to the case of more than two regions (multi-object segmentation) and requires sending multiple exclusive tile-to-object/region maps instead of a single FG/BG map.
• the portions, regions or region-layers may be groups of tiles that have an association to the same object whether that object is a background, a foreground, a moving object, or any other object. It is contemplated that the tiles may be grouped by other common associations such as to size or position on the frame regardless of the image displayed by the frame, and so forth. This applies to any of the groupings discussed herein. While the term tiles from NGV has been used thus far, if DMC is used in conjunction with HEVC, the same principle applies with the term CTB/LCU replacing the term tile.
• warped reference frame 2000 shows a first variation of delta mv type of DMC with increased reduction of overhead
• the approximate FG/BG region classification based on a tile map can be rather coarse, which in turn may not result in sufficient DMC prediction (measured by reduction in DMC prediction error).
• a warped reference frame 2100 may be used to perform dominant motion compensation of approximate region-layers of merged Bi-tree partitions of tiles or CTBs with each or multiple region-layers using delta MV correction with respect to an affine GMC reference frame in order to further reduce overhead.
• a warped reference frame 2100 may be used as a reference to code current frame 2102.
• Current frame 2102 has foreground (FG) tiles 2108 and background (BG) tiles 2110.
• frame 2102 has a variation of tile sizes and permits tiles to be split into two horizontally or vertically, allowing for border tiles 2104 and 2106 to be more accurate, and thus the approximate region boundary to be more accurate, and hence the prediction to be more accurate (resulting in further reduction of prediction error).
• current frame 2102 has some tiles 2108 and 2110 that are complete, be it in FG or BG regions, while many tiles in the FG/BG border are either horizontally (2104) or vertically (2106) by bi-tree partition splits allowing improved accuracy with FG/BG approximate region-layers to be constructed.
• warped reference frame 2100 generated by first estimating global gmt parameters for the entire current picture with regard to a reference frame, and then warping the reference frame using the computed parameters to compute a warped GMC frame 2116, and then padding it with a boundary extension to create the rectangular warped GMC Reference frame 2100.
• this GMC Reference frame 2100 two delta motion vectors 2122 and 2124 are then computed, one for approximate FG (full tiles and half tiles that form the approximate foreground region), and the other for approximate BG
• warped and adjusted tile 2126 is used to position current frame tile 2110, while warped and adjusted tile halves 2120 and 2122 respectfully are references for current frame tile halves
• a warped reference frame 2200 may provide dominant motion compensation of approximate region-layers of merged quad-tree partitions of tiles or CTBs with each region-layer using delta MV correction with respect to affine GMC Reference frame.
• current frame 2202 to be coded is permitted to have tiles split into quads 2206 and 2208 (1/2 both horizontally and vertically), allowing for border tiles to be more accurate, and thus the approximate region boundary to be more accurate, and hence the prediction to be more accurate (resulting in further reduction of prediction error).
• current frame 2202 has some tiles 2204 and 2210 that are complete, be it in FG or BG regions, while many tiles in the FG/BG border are both horizontally and vertically (quadtree partitions) split allowing improved accuracy FG/BG approximate region-layers to be constructed.
• GMC Reference frame 2200 generated by first estimating global gmt parameters for the entire current frame picture with regard to a reference picture, and then warping the reference picture using the computed parameters to compute warped GMC Picture 2114, and then padding it with a boundary extension to create rectangular GMC Reference
• encoder 1800 and decoder 1900 may be slightly modified to provide components to perform approximate region-layer delta mv based DMC approach for frames 2000, 2100, and 2200 discussed above.
• the region-layers may be merged tiles or CTBs, or merged (Bi-/Quad-) tree partitions of Tiles/CTBs) each using delta MV's correction.
• HEVC standard does not support Global (or Dominant) Motion Compensation, or Morphed or Synthesized references, but does support plain references.
• the decoder 1900 may use a Content Blocks Props Analyzer & Approximate Region Segmenter 1822 that analyzes each input picture of a video sequence and segments it into approximate region-layers.
• a merge map may be provided and has data that carries the mapping of tiles (and partitions of tiles) into one of the two categories.
• a mapper 1824 also referred to as an approximate regions to tile/CTB and partitions mapper receives boundary data or parameters (also referred to as Tile/CTB & Bi/Quadtree Partitions Boundary) in order to construct the merge map.
• Both the current picture (being processed) as well as the past decoded reference picture are input to the Global or Dominant Motion Estimator & Compensated Prediction Picture Generator 1804 so that delta mv DMC parameters can be computed for each of the two approximate regions and by using the DMC Reference Picture.
• the merge map is used by the motion compensated predictor (also called the Bi-Tree Partitions Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor) and by the block motion estimator and partitions motion assembler 1812 (also called the 4x4 Block Motion Estimator 1 ⁇ 4 & 1/8 pel Accuracy and Partitions Motion Assembler), of which the latter is used to compute delta motion vectors and the former is used to compute actual motion compensated DMC prediction using these delta mvs.
• the result is a complete DMC Ref Picture.
• the DMC approximate region based predictions using DMC Ref Pictures are fed to the Pred Modes & Ref Types Analyzer 1816, along with the input from the intra directional predictor 1818, and the process proceeds as described above on a Tile or partition basis where the Pred Modes & Ref Types Analyzer 1816 determines the best prediction from various choices.
• the entropy coder 1820 encodes DMC data, such as gmt parameters, Amvs and merge map for this alternative, along with other data such as mvs, mop (morphing parameters) and syp (synthesis parameters) and mode info, and so forth.
• decoder 1900 may be modified to perform Dominant Motion Compensation of approximate Region-layers (of merged Tiles/CTBs, or merged (Bi-/Quad-) tree partitions of Tiles/CTBs) each using delta MV's correction.
• An entropy decoder 1901 decodes DMC data such as gmt parameters, delta mvs, and merge map, as well other data such as mvs, mops (morphing parameters), and syp (synthesis parameters), and mode info (not shown).
• the decoded merge map data is used by the motion compensation predictor 1912 (also called the Bi-Tree Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor) to keep track of the type of (FG/BG) region a tile or partitioned tile being processed belongs to.
• a delta motion vector for approximate FG region is then applied to determine a DMC predicted approximate FG region consisting of tiles and partitions. Likewise, a delta motion vector for approximate
• BG region is then applied to determine DMC predicted approximate BG region consisting of tiles and partitions.
• the averaged pixels of an overlapping area are used as the reconstruction.
• neighboring background region boundary and foreground region boundary pixels are averaged or extended to fill these holes. The result is a complete DMC
• the Prediction Mode Selector 1914 uses the tiles or partitions based mode information sent by the encoder via the bitstream to use tiles/partitions of either the DMC approximate FG or BG regions from the DMC Reference Picture, or one of the several available morphed predictors, or synthesized predictors, or intra predictors. The resulting prediction is then added back (this portion of decoder is external) to a decoded quantized prediction error decoded at the decoder to reconstruct a final decoded video picture.
• process 1300 may also include define object associated regions 1324, rather than defining tiles and then grouping them into regions, and rather than using blocks for motion vector based DMC.
• dominant motion compensation of segmented region-layers is performed with each region layer using a delta MV correction with respect to an affine GMC reference frame with the goal of further improving prediction accuracy/overhead tradeoff.
• This variation uses regions (more precisely a collection of regions belonging to an object), and called a region-layer (RL) explicitly.
• a current frame 2300 to be coded is explicitly segmented into a foreground (FG) region-layer 2304 and a background (BG) region-layer 2306, with the FG region-layer 2304 comprising a head and shoulder view 2302 of a person, and the background containing the rest of the picture including a star 2308.
• GMC parameters gmt are first computed and are used to generate a GMC Reference frame or Picture (or warped reference frame) 2310 as described previously.
• the location of the FG region-layer 2304 is determined in the GMC Reference frame 2322 and a single correction delta mv (Amvf) 2328 is computed for the FG region adjusting the position of the region-layer from a warped position 2326 to an adjusted position 2330 and by the delta motion vector, such that it reduces the DMC prediction error for the FG region-layer.
• a single correction delta mv (Amvb) 2318 is computed for the BG region such that the delta motion vector adjusts the position of the star from a warped position 2314 to an adjusted position 2316, which also reduces the DMC prediction error for the background.
• This variation of delta mv type DMC incurs additional cost in region-layer representation in terms of a differential with regard to a simple block-based (partition-based) merge map, but in return allows for further reduction of DMC prediction error with regard to the previous tile-based technique while requiring the same number (two) of delta mvs.
• all three variations block, grouped tile, and whole region offer different tradeoffs in terms of complexity, overhead, and DMC prediction error reduction.
• the block-based variation of warped reference frame 1700 (FIG. 17) is the simplest in complexity, the tile-grouping variation/s (FIGS. 20-22) offer a middle ground in terms of approximate region-layer boundary, while the whole region variation (FIG. 23) offers the opportunity for more accurate reduction of DMC prediction error.
• an encoder 2400 is provided with components to perform DMC of segmented region-layers where each region-layer uses a delta mv correction.
• the encoder 2400 has a Content Analyzer and Region Segmenter 2426 that analyzes each input picture of video sequence and segments it into region-layers. For the purpose of explanation, assuming each picture is segmented into two region-layers, a foreground (FG) region-layer and a background (BG) region-layer, a region boundary (region bndry) has data that carries the foreground boundary shape (the remaining being the background).
• Both the current picture (being processed) as well as the past decoded reference picture are input to the Global or Dominant Motion Estimator & Compensated Prediction Picture Generator 2404 so that delta mv DMC parameters can be computed for each of the two regions and to generate the DMC Reference Picture.
• the region bndry map is used by the motion compensated predictor 2414 (also called the Regions Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor) and by the regions motion estimator 2412 (also called the Regions Motion Estimator 1 ⁇ 4 & 1/8 pel Accuracy) of which the latter is used to compute delta motion vectors while the former is used to compute actual motion compensated DMC prediction using these delta mvs.
• the DMC region based predictions of FG and BG regions are generated using computed motion vectors for offsetting in GMC Reference frames, and resulting predictions are fed to the Prediction Modes & Ref Types Analyzer 2416.
• the intra data from an intra directional prediction analyzer and generator 2418 as well as morphed and synthesized predictions may be provided to the analyzer 2416.
• the Prediction Modes & Ref Types Analyzer 2416 may determine the best prediction from various choices (e.g. DMC prediction is one of the many choices available), and outputs it to a differencer (in a portion of the circuit not shown here) that generates prediction error for coding.
• the entropy coder 2420 (called the Entropy Encoder Morphing and Synthesis Parameters & MVs) encodes DMC data, such as gmt parameters, delta mvs and region bndry, along with other data such as mvs, mop (morphing parameters) and syp (synthesis parameters) and mode info (not shown).
• DMC data such as gmt parameters, delta mvs and region bndry
• mvs mop (morphing parameters) and syp (synthesis parameters) and mode info (not shown).
• this type of region based DMC makes best sense in context of overall Region Based Video Coder where a picture is divided into region-layers, and for ease of processing each region may be divided into sub- regions.
• a portion or sub-system of a region based decoder 2500 is provided to perform Dominant Motion Compensation of segmented Region-Layers each region-layer using delta MV's correction.
• the decoder 2500 may have an entropy decoder 2518 (also referred to as an Entropy Decoder Morphing & Synthesis Params) and that decodes DMC data such as gmt parameters, delta motion vectors, and region bndry, as well other data such as mvs, mops (morphing parameters), and syp (synthesis parameters), and mode info (not shown).
• the decoded region bndry data is used by a Regions Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor 2512 to determine if a sub-region is part of the FG or BG region.
• the Global Motion Compensated Prediction Pictures Generator 2504 uses the decoded gmt parameters on decoded picture/s from DPR Picture Buffer 2502 to first generate a warped GMC picture that is then padded to generate a rectangular GMC Reference Picture. Then using decoded region bndry and delta motion vectors, DMC prediction of an FG region as well as DMC prediction of a BG region is generated.
• the Pred Mode Selector 2514 uses the sub-region based mode information sent by the encoder via the bitstream to use sub-regions of either DMC predicted FG or BG regions from the GMC Reference Picture, or sub-regions of one of the several available morphed predictors, or synthesized predictors, or intra predictors. The resulting prediction is then added back (this portion of decoder is external) to decoded quantized prediction error decoded at the decoder to reconstruct a final decoded video picture. The remaining components are similar to those described before.
• a second type of DMC that is referred to as local global motion trajectory (gmt) type of DMC, or really global motion compensation applied locally, but simply called local global motion compensation for short.
• This may be performed by computer implemented example process 2600 for local global motion compensation.
• Example process 2600 is arranged in accordance with at least some implementations of the present disclosure.
• Process 2600 may include one or more operations, functions or actions as illustrated by one or more operations 2602 to 2626 numbered evenly.
• Process 2600 may form at least part of a next generation video coding process.
• process 2600 may form at least part of a next generation video encoding process as undertaken by coder system 100 or 200 of FIGS. 1-2 or gain compensation coder sub-systems 1800 or 1900 of FIGS. 18-19, and/or any other coder system or subsystems described herein.
• Process 2600 first may include obtaining frames of pixel data and having a current frame and a decoded reference frame 2602 as described previously. Process 2600 may then include define frame portions for local global motion compensation 2604. Three alternative ways to divide a frame into portions is provided and generally look similar to the three divisions used with delta motion vectors, except here some key differences exist. Here, the process 2600 may continue with divide the frame into tiles 2606 instead of blocks and where local GMC is applied to each tile. By another alternative, the process 2600 may include divide the frame into tiles 2608 (and/or sub-tiles 2610) and then group the tiles into regions 2612 so that the same local global motion trajectories are applied to each tile in the same region.
• process 2600 may include define object-associated regions 2614 so that local GMC will be applied to each region.
• object-associated regions 2614 are explained in detail below. Note on both process FIGS. 13 and 26, tile is meant in the general sense to mean very large block and includes CTBs.
• dominant motion compensation of tiles or CTBs may be performed by using affine motion parameters for generation of affine local motion compensated (LMC) reference for each or multiple tiles or CTBs.
• LMC affine local motion compensated
• a current frame 2702 may be divided into tiles (or rectangular regions) 2704.
• the process 2600 may then continue with create dominant motion compensated warped portion.
• a reference frame 2700 has individual local gmt GMC morphed tiles (or rectangular regions) that may be reconstructed.
• the process 2600 may continue with determine local global motion trajectories 2618 which also may be referred to as dominant motion trajectories (dmts).
• each tile (or rectangular region) is allowed to have its own independent set of gmt DMC (or dmts) parameters (each of three vertices of the tile can be provided independent motion with a fourth corner being dependent) instead of a single set of gmt GMC parameters for the entire frame.
• reference frame 2700 may have a tile 2706 that is a reference for tile 2704 on the current frame 2702.
• the tile may be displaced from a position on the initial reference frame by global motion trajectories (gmt) 2712a, 2712b, and 2712c.
• gmt global motion trajectories
• a tile 2708 may be displaced by global motion trajectories 2714a, 2714b, and 2714c
• tile 2710 may be displaced by global motion trajectories 2716a, 2716b, and 2716c, such that each tile or region has its own gmt set (or dmts set).
• the dmts or local gmt may be obtained by known processes with the affine methods.
• averaging of pixels in overlapping areas of warped neighboring tiles may be performed, followed by filling in holes also using averaging of values from available nearby pixels to form the region rectangle 2622.
• This is more sophisticated than the rather simplistic boundary extension performed earlier to create a rectangular morphed GMC reference frame.
• extensions could be used instead to form tile or region rectangles.
• the resulting filled picture is then the DMC Ref Picture for this type of DMC and can be used for tile (or rectangular region) based DMC motion compensation.
• a further virtual region may be formed 2624 as with the virtual reference frame.
• the process 2600 may then include form portion predictions using the pixels from the warped portions (or tiles or regions) 2626.
• encoder 1800 may be modified to be a portion or sub-system of NGV/HEVC Extension-2a Encoder and to have components to perform an approximate region-layer gmt based DMC approach used with reference frame 2700 as discussed above.
• the two inputs to the global or Dominant Motion Estimator & Compensated Prediction Pictures/Tiles/Partitions Generator 1804 may include the current picture (being processed) as well as the past decoded reference picture (one of the pictures from DPR Picture Buffer 1802) so that gmt DMC parameters can be computed for each tile, and a DMC Reference Picture can be generated.
• tile -based DMC parameters can result in areas of overlap that are resolved by averaging pixels in overlapped areas, and holes that are resolved by averaging and filling in a boundary from neighboring tiles.
• the tile-based DMC predictions using DMC Reference Pictures may be fed directly from the dominant motion compensated prediction local/picture buffer 1806 to the Prediction Modes & Ref Types Analyzer 1816 (in addition to intra prediction input from the intra directional prediction analyzer and generator 1818 as well as morphed and synthesized predictions.
• the Prediction Modes & Ref Types Analyzer 1816 determines the best prediction from various choices (e.g.
• DMC prediction is one of the many choices available) and outputs it to a differencer (in portion of circuit not shown here) that generates prediction error for coding.
• the entropy coder 1820 encodes DMC data, such as dmts parameters along with other data such as mvs, mop (morphing parameters) and syp (synthesis parameters) and mode info (not shown).
• the decoder 1900 may be modified as a portion of NGV/HEVC Extension-2a Decoder with Dominant Motion Compensation of Tiles or CTBs each using affine motion parameters.
• an entropy decoder 1901 decodes DMC data such as dmts parameters as well other data such as mvs, mops (morphing parameters), and syp (synthesis parameters), and mode info (not shown).
• the Global or Dominant Motion Compensated Prediction Pictures/Tile/Partitions Generator 1904 uses the decoded dmts parameters to generate tile based warped DMC predictions during which pixels in the areas of overlap are reconstructed as average of pixels from overlapping tiles, and while filling any holes (average of nearest boundary pixels of tiles, or boundary extension on borders of picture) to create a complete DMC Reference picture for prediction.
• the Prediction Mode Selector 1914 may directly, or otherwise, receive the DMC reference picture (or the warped tiles) from the dominant motion compensated prediction local picture buffer 1906, and uses the tiles based mode information sent by the encoder via the bitstream to use the warped tiles of either the DMC Reference Picture, or one of the several available morphed predictors, or synthesized predictors, or intra predictors (intra directional prediction generator 1916). The resulting prediction is then added back (this portion of the decoder is not shown) to decoded quantized prediction error decoded at the decoder to reconstruct final decoded video picture. Otherwise, the description of the components for this alternative are as described above with the other alternatives. Referring to FIGS.
• process 2600 may continue with divide the frame into tiles 2608, and when available, into sub-tiles 2610.
• the tiles may then be grouped into regions 2612 as with the delta motion vector option.
• the tiles instead of a motion vector for each region, the tiles here are each warped with its own set of dmts as explained previously to create warped portions 2616, and the pixel values of the warped tiles are used as predictions.
• this option provides Dominant Motion Compensation with approximate region-layers of merged Tiles or CTBs each using affine motion parameters for generation of an affine Semi-global Motion Compensated (SMC) Reference.
• SMC Semi-global Motion Compensated
• a current frame 2802 may be divided into tiles (or rectangular regions).
• a group of tiles (or approximate region-layers) may have their own independent set of gmt, or more specifically dmt, GMC parameters (each of three vertices of each tile in the group of tiles can be provided the same independent motion with fourth vertex of each tile being dependent).
• GMC parameters each of three vertices of each tile in the group of tiles can be provided the same independent motion with fourth vertex of each tile being dependent.
• a frame may be divided into two groups of tiles (or approximate region-layers), one group corresponding to the background (BG), and the other group corresponding to foreground (FG).
• two sets of gmt GMC parameters one for FG and the other for BG is sent, along with a tile (rectangular region) based FG/BG map ⁇ merge map) via the bitstream to the decoder.
• the operation may not be limited to division of a picture into two groups of tiles (or approximate region layers), and can the frame may be divided into three, four, or more approximate region-layers where each approximate region-layer has one gmt GMC parameter set that is used and sent to the decoder.
• a single gmt GMC parameter is computed, and likewise for the complimentary tiles that are part of the BG group of tiles (approximate region-layer), a different gmt GMC parameter is computed.
• FIG. 28 This is illustrated by FIG. 28 where the current frame 2802 is divided into foreground tiles 2804 and background tiles 2806.
• One background tile 2806 corresponds to a background tile 2808 of a reference frame 2800.
• the background tile 2808 is displaced by dominant motion trajectories 2814, while an adjacent background tile 2810 is warped by trajectories 2816 that are the same or similar to trajectories 2814 since both of these tiles are in a background region.
• One foreground tile 2804 corresponds to a warped tile 2812 on the reference frame 2800 and uses foreground trajectories 2818 that are different from the background trajectories 2814 and 2816.
• a reference picture is read from DPR picture buffers, and the collocated tile corresponding to each tile of an FG region layer is warped at each of the three vertices (the fourth being free) by using FG gmt parameters.
• the process is repeated for each BG tile of approximate a BG region-layer using the same picture and using BG gmt parameters.
• an averaging process is used to reconstruct the final pixels.
• hole filling is employed using averaged prediction from closest neighboring boundary of approximate FG and BG region-layers.
• the resulting filled picture is then the DMC Reference Picture, and can be used for tile (or rectangular region) based DMC motion compensation.
• the technique can easily be applied to more than two approximate region- layers. Since in this technique there will always be two or more approximate region-layers, instead of the term gmt, the term dmt (dominant motion trajectory) parameters is used as mentioned above
• Dominant Motion Compensation is performed with approximate region-layers of merged Bi-tree Partitioned Tiles or CTBs each using affine motion parameters for generation of an affine SMC reference frame.
• a modification of the tile-based process for reference frame 2800 is provided to improve the accuracy of approximate FG/BG region-layer classification by horizontally or vertically splitting the full tiles into half-tiles (such as that from Bi-Tree partitioning), which can be used in addition to the full tiles.
• a current frame 2902 to be coded may have foreground full tiles 2904 and half- tiles 2906.
• the FG group of tiles (or approximate region-layer) may include mostly full tiles but several horizontal or vertical half-tiles as well.
• the background also may have full tiles 2910 as well as horizontal or vertical half-tiles 2912 and 2914.
• a reference frame 2900 may include both full and half-tiles 2916, 2922, and 2926.
• the background full tile 2916 is shown to be warped from an original position 2920 by trajectories 3018 where each of the background tiles or half-tiles use the same trajectories 2918.
• the foreground tiles and half-tiles 2924, 2928, and 2930 all use the same warping trajectories 2924.
• the FG/BG segmentation (merge) map may require slightly higher bit totals due to higher accuracy in FG/BG approximation. Overall, there will be only one FG and one BG set of dmts motion parameters still. In the present example, the overall process of DMC Reference Picture generation provides improved prediction due to higher FG/BG accuracy.
• Dominant Motion Compensation is performed with approximate region-layers of merged quad-tree partitioned tiles or CTBs each using affine motion parameters for generation of an affine SMC reference. Specifically, a modification of the tile- based process for reference frame 2800 is provided to improve the accuracy of approximate
• FG/BG region-layer classification by horizontally and vertically splitting the tiles into quarter- tiles can be used in addition to full tiles.
• the current frame 3002 may have an FG group of tiles (or approximate region-layer) and may include mostly full tiles 3004, but several quarter tiles 3006 and 3008 as well.
• the background may also include full tiles 3010 as well quarter tiles 3012.
• a reference frame 3000 may include a corresponding full background tile 3018 that is shifted to a warped tile or rectangle 3014 by trajectories 3020, while trajectories 3022 warp a quarter-tile 3024 to a shifted position 3016 as well.
• the FG/BG segmentation (or merge) map may require a slightly higher bit total due to higher accuracy in the FG/BG approximation. Overall, there will be only one FG and one BG set of dmts motion parameters still. In the present example, the overall process of DMC Reference Picture generation provides improved prediction due to higher FG/BG accuracy.
• encoder 1800 may be modified to form a subsystem or portion of NGV/HEVC Extension-2b encoder to perform Dominant Motion Compensation with approximate region-layers of tiles, merged Bi-/Quad- tree Partitioned Tiles or CTBs each using affine motion parameters.
• the encoder 1800 may be modified to perform approximate region-layer gmt based DMC approach as discussed with reference frames 2800, 2900, and 3000 (FIGS. 28-30), and similar to that already described above with reference frames 2000, 2100, and 2200 that groups tiles into regions albeit for delta motion vectors.
• the encoder 1800 may have a Content Blocks Props Analyzer & Approximate Region Segmenter
• each picture is segmented into two region-layers, an approximate foreground (FG) region—layer and an approximate background (BG) region-layer and a merge map is provided to carry data mapping of tiles (and partitions of tiles) into one of the two categories.
• FG foreground
• BG approximate background
• the merge map information is then input to the Global or Dominant Motion Estimator & Compensated Prediction Pictures/Tiles/Partitions Generator 1804, the other inputs of which include both the current picture (being processed) as well as the past decoded reference picture (one of the pictures from DPR Picture Buffer 1802) so that gmt DMC parameters can be computed for each of the two approximate regions so that the DMC Reference Picture can be generated.
• the DMC approximate region based predictions using DMC Reference Pictures are fed directly to the Prediction Modes & Ref Types Analyzer 1816 along with other inputs such as intra prediction and morphed and synthesized predictions.
• the Pred Modes & Ref Types Analyzer 1816 determines the best prediction from various choices (e.g. DMC prediction is one of the many choices available) and outputs it to a differencer (in a portion of the circuit not shown here) that generates prediction error for coding. Further, the entropy coder 1820 encodes DMC data, such as dmts parameters and the merge map, along with other data such as mvs, mop (morphing parameters) and syp (synthesis parameters) and mode info (not shown). The other components of FIG. 18 that are not mentioned here are described above with other implementations.
• a decoder 1900 may be modified to be part of a sub-system or portion of NGV/HEVC Extension-2b decoder to perform Dominant Motion Compensation with approximate region-layers of Tiles, merged Bi-/Quad- tree Partitioned Tiles or CTBs each using affine motion parameters.
• the entropy decoder decodes DMC data such as dmts parameters, and merge map, as well other data such as mvs, mops (morphing parameters), and syp (synthesis parameters), and mode info (not shown).
• the decoded merge map data is used by DMC predictor 1904 (Global or Dominant Motion Compensated Prediction Pictures/Tiles/Partitions Generator), and MC predictor 1912 (Bi-Tree Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor).
• DMC predictor 1904 Global or Dominant Motion Compensated Prediction Pictures/Tiles/Partitions Generator
• MC predictor 1912 Bi-Tree Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor.
• the Dominant Motion Compensated Prediction Pictures/Tile/Partitions Generator 1904 also uses the decoded dmts parameters on decoded picture/s from DPR Picture Buffer 1902 to first generate DMC approximate FG and BG regions.
• the DMC predictor 1904 also generates reconstructed pixels in the area of overlap as an average of pixels from the two approximate regions, and fills any holes (average of nearest boundary pixels of two regions, or boundary extension on borders of picture) to create a complete DMC Reference picture for prediction.
• the Pred Mode Selector 1914 uses the tiles/partitions-based mode information sent by the encoder via the bitstream to use tiles/partitions of either DMC approx. FG or BG regions from DMC Reference Picture, or one of the several available morphed predictors, or synthesized predictors, or intra predictors.
• a current frame 3100 is segmented into two or more region- layers.
• the current frame 3100 may be segmented into a first region-layer 3104 corresponding to the foreground (FG) region-layer while the remaining portions of the picture may be referred to as the background (BG) region-layer 3102.
• the FG region-layer is enclosed in the tightest fitting bounding box or rectangle, or any other convenient shape, and gmt DMC parameters are calculated for either the entire bounding box or the bounding box with masked background, with, for example, a past decoded reference frame 3108 from a DPR picture buffer as a reference.
• the boundary may be set at a certain distance from the object rather than closest fit. Note other alternatives may include having a boundary that matches or corresponds with (aligns with) the shape of the object in the foreground, here a head and shoulders 3106.
• the gmt (or dmi) DMC parameters 3120a, 3120b, and 3120c are applied to the vertices of the boundary, and specifically the bounding box region in the reference frame 3108 to form a warped bounding box region 3016 from the unwarped position 3014, and that represents the warped FG region- layer providing a warped head and shoulder (object) position 3118.
• a set of gmt or dmt DMC parameters 3112a, 3112b, and 3112c are computed for the BG region-layer 3124 by using the frame rectangle with FG the region-layer 3116 masked, and the computed gmt DMC parameters 3112a, 3112b, and 3112c are then applied to vertices of the entire frame 3126, resulting in a warped BG region-layer 3110. Since it is possible for the two morphed region- layers 31 10 and 3116 to overlap, the area of overlap 3128 can be reconstructed by averaging overlapping pixels from the two regions. Further, since the two warped region-layers 3110 and
• any unfilled area close to the frame border is boundary extended as before.
• two sets of gmt DMC trajectories one for FG region-layer and the other for BG region-layer
• FG/BG segmentation boundary map is sent via bitstream to the decoder. This particular variation is best used in context of the encoder that already uses region based coding.
• the encoder 2400 may be modified to be a sub-system or portion of an advanced region based encoder to perform Dominant Motion Compensation with segmented region-layers each using affine motion parameters.
• the encoder 2400 may have a Content Analyzer and Region Segmenter 2426 that analyzes each input picture of video sequence and segments it into region-layers.
• region layers such as a two region-layer structure with a foreground (FG) region-layer and a background (BG) region-layer to create a region bndry with data that carries the foreground boundary shape (the remaining being the background or example).
• FG foreground
• BG background
• the region bndry shape information is then input to the global or Dominant Motion Estimator &
• Compensated Prediction Pictures/Regions Generator 2404 the other inputs of which include both the current picture (being processed) as well as the past decoded reference picture (one of the pictures from DPR Picture Buffer 2402) so that gmt DMC parameters can be computed for each of the two regions so that the DMC Reference Picture can be generated.
• the DMC reference pictures are place in the dominant motion compensated prediction local/picture buffer 2406, and DMC region based predictions formed by using the DMC Reference Pictures are fed directly to the Prediction Modes & Ref Types Analyzer 2416 (as well as other inputs such as intra prediction and morphed and synthesized predictions).
• the Prediction Modes & Ref Types Analyzer 2416 determines the best prediction from various choices (e.g. DMC prediction is one of the many choices available) and outputs it to a differencer (in a portion of the circuit not shown here) that generates prediction error for coding. Further, the entropy coder 2420 encodes DMC data, such as dmts, parameters and region bndry, along with other mop (morphing parameters) and syp (synthesis parameters) and mode info (not shown). The other components of FIG. 24 that are not mentioned here are described above with other implementations.
• region based DMC makes best sense in context of overall region based video encoder coder where a frame is divided into region-layers, and for ease of processing, each region is divided into sub-regions. While it is possible for sub-regions to be tiles or blocks, they could also be arbitrary in shape at some precision (say 4 x 4 block accuracy). This applies whenever sub-region processing is mentioned herein.
• the decoder 2500 may be modified to be part of a sub-system or portion of Advanced Region-based Decoder to perform Dominant Motion Compensation with segmented region-layers using affine motion parameters.
• the decoder 2500 may have an entropy decoder 2518 (Entropy Decoder Morphing & Synthesis Params) that decodes DMC data such as dmts parameters, and region bndry, as well other data, such as mvs, mops (morphing parameters), and syp (synthesis parameters), and mode info (not shown).
• the decoded region bndry data is used by the DMC predictor 2504 (Global or Dominant Motion Compensated Prediction Pictures/Regions Generator), and MC predictor 2512 (Regions Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor).
• DMC predictor 2504 Global or Dominant Motion Compensated Prediction Pictures/Regions Generator
• MC predictor 2512 Regions Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor
• the Global or Dominant Motion Compensated Prediction Pictures/Regions Generator 2504 also uses the decoded dmts parameters on decoded picture(s) from the DPR Picture Buffer 2502 to first generate DMC FG and BG regions, and in case of overlap of regions, generating reconstructed pixels in an area of overlap as an average of pixels from the two regions, and filling holes (average of nearest boundary pixels of two regions, or boundary extension on borders of a picture) to create a complete DMC Reference picture for prediction.
• the Prediction Mode Selector 2514 uses the sub-region based mode information sent by the encoder via the bitstream to use sub-regions of either DMC FG or BG regions from DMC Reference Picture, or sub-regions of one of the several available morphed predictors, or synthesized predictors, or intra predictors. The resulting prediction is then added back (this portion of decoder is external or not shown here) to decoded quantized prediction error decoded at the decoder to reconstruct final decoded video picture.
• the components of FIG. 25 that are not mentioned here are described above with other implementations. Referring to FIGS. 32-33, by another implementation, the two main DMC types: (1) delta motion vector-based DMC, and (2) local global motion compensation DMC, are combined.
• a simple approach would be to use block- based delta mv type DMC on a tile basis, and also use local gmt type of DMC on a tile basis, and based on a reduction of DMC prediction error, choose the best DMC mode.
• block-based delta mv type DMC for tiles that use block-based delta mv type DMC for example, block-based delta mvs and frame-based GMC gmt parameters would be sent via the bitstream to the decoder, and for tiles that use local gmt type of DMC, tile based gmt parameters would be sent in the bitstream.
• an encoder 3200 may be part of a sub-system or portion of a NGV/HEVC Extension-3 Encoder used to perform Dominant Motion Compensation of blocks each using delta MV correction as well as tiles or CTBs each using affine motion parameters.
• Encoder 3200 may store decoded and filtered frames in a DPR picture buffer 3202 for use by Global Motion Estimator & Compensated Prediction Pictures Generator 3204, and Dominant Motion Estimator & Compensated Prediction Picture Generator 3220, and as well as by Other Morph Analyzer Generator & Picture/s Buffer 3208, and Synth Analyzer, Generator & Picture/s Buffer 3210.
• the high level operation of the two DMC operations was discussed earlier and will not be repeated here.
• components of the other encoders described herein that are similar to components of the encoder 3200 operate similarly.
• the operation of DMC components related to the dual operation of MV based DMC and local gmt based DMC are described below.
• each or multiple tiles of a frame from DPR buffer 3202 is input to the Dominant Motion Estimator & Compensated Prediction Picture Generator 3220 for computation of an independent set of gmt DMC parameters (each of three vertices of the tile can be provided independent motion with a fourth one being dependent) instead of a single set of gmt GMC parameters for the entire picture.
• each tile of a previous reference frame (from DPR buffer 3202) is warped to generate individual local gmt GMC morphed tiles such that some warped tiles result in overlapped pixels and others result in holes not covered by any of the tiles.
• dmt (or its plural dmts) may be used to refer to local tile based gmt DMC parameters to differentiate themselves from gmt itself.
• the same or a different reference frame from DPR buffer 3202 is input to Global Motion Estimator & Compensated Prediction Picture Generator 3204 that performs global motion estimation (GME) producing global motion parameters (represented as gmt trajectories) and generating a GMC reference frame that is stored in Dominant Motion Compensation Prediction Local/Picture Buffer 3206.
• GME global motion estimation
• a block motion estimation and partitions motion assembler 3212 may perform tile based motion estimation resulting in delta motion vectors of tiles that can be used for correction or motion, and that are used by motion compensated predictor 3214 (referred to here as (Bi-Tree Partitions) Char and Motion Compensated Adaptive Precision Filtering Predictor) and that generates prediction tiles by sub-pixel interpolation using the GMC Reference frame.
• motion compensated predictor 3214 referred to here as (Bi-Tree Partitions) Char and Motion Compensated Adaptive Precision Filtering Predictor
• the output prediction tiles from the delta mv based DMC and local gmt based DMC are fed to the Prediction Modes & Reference Types Analyzer 3216 as well as inputs such as intra predicted blocks/tiles from the intra directional prediction analyzer and generator 3218.
• the Prediction Modes & Reference Types Analyzer 3216 determines the best prediction block or tile from various choices.
• the available choices may include DMC prediction as one of the many morphing choices available including one choice that is local gmt DMC based and another choice that is delta motion vector DMC based.
• the analyzer 3216 outputs the best prediction to a differencer (in portion of circuit not shown here) that generates prediction error for coding.
• the analyzer 3216 also outputs a map of the DMC mode selection information (dmsi) on a portion (tile or region or other partition) basis that was used in the processing.
• the entropy coder 3224 also called Entropy Encoder Morphing and Synthesis Parameters & MVs
• GMC/DMC parameters and data such as the gmt, Amvs, dmts, and dmsi.
• a decoder 3300 may be part of a sub-system or portion of a NGV/HEVC Extension-3 decoder use to perform Dominant Motion Compensation of blocks each using delta MV corrections as well as tiles or CTBs each using affine motion parameters.
• the decoder 3300 may enter and store decoded and filtered frames in DPR picture buffer 3302 for use by Global Motion Compensated Prediction Picture Generator 3204, Dominant Motion Compensated Prediction Picture Generator 3218, Other Morph Generator & Pictures Buffer 3208, and Synth Generator & Pictures Buffer 3210.
• the high level operation of the two processes was explained earlier with other implementations, and will not be repeated here. Thus, components in the other implementations that are similar to the components of decoder 3300 operate similarly.
• Global Motion Compensated Prediction Picture Generator 3204 along with block motion compensated predictor 3212 (shown here as (Bi-Tree Partitions) Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor), or Dominant Motion Compensated Prediction Pictures/Tiles Generator 3218 is deployed to generate the appropriate DMC motion compensation.
• block motion compensated predictor 3212 shown here as (Bi-Tree Partitions) Char, and Motion Compensated Adaptive Precision (AP) Filtering Predictor
• Dominant Motion Compensated Prediction Pictures/Tiles Generator 3218 is deployed to generate the appropriate DMC motion compensation.
• the DMC mode may not be used since the encoder selects the best mode which may have been one of the other morphed prediction modes, one of the other synthesized prediction modes, or an intra mode.
• process 3400 may include one or more operations, functions or actions as illustrated by one or more of actions 3401 to 3413.
• process 3400 will be described herein with reference to example video coding system 3500 including encoder 100 of FIG. 1 and decoder 200 of FIG. 2, as is discussed further below with respect to FIG. 35.
• process 3400 may be undertaken by a system including both an encoder and decoder or by separate systems with one system employing an encoder (and optionally a decoder) and another system employing a decoder (and optionally an encoder). It is also noted, as discussed above, that an encoder may include a local decode loop employing a local decoder as a part of the encoder system.
• video coding system 3500 may include a processing unit such as a graphics processing unit 3520 with logic circuitry 3550, the like, and/or combinations thereof.
• logic circuitry 3550 may include encoder system 100 of
• video coding system 3500 as shown in FIG. 34 may include one particular set of blocks or actions associated with particular modules, these blocks or actions may be associated with different modules than the particular modules illustrated here.
• process 3500, as illustrated, is directed to encoding and decoding, the concepts and/or operations described may be applied to encoding and/or decoding separately, and, more generally, to video coding.
• Process 3400 may begin with "receive input video frames of a video sequence" 3401, where input video frames of a video sequence may be received via encoder 100 for example. This may include a current frame and a past or previous frame (that is to be used as a reference frame), and that is to be used for motion compensation to reconstruct a predicted frame as the current frame.
• the process 3400 also comprises "perform prediction and coding partitioning, quantization, and decoding loop to decode a reference frame" 3402.
• video frames are coded and then decoded in a decoder loop at an encoder in order to provide coded data that can be accurately obtained by a process that is repeated at the decoder.
• the process 3400 also comprises "perform frame-wide global motion compensation to form a warped GMC reference frame" 3403.
• global motion trajectories are applied at the corner of the frame to warp the frame for delta motion- vector based DMC.
• the process may be implemented as explained for the other implementations described herein.
• the process 3400 also comprises "determine delta motion vector for individual portions of the warped GMC reference" 3404. This may include first defining the portion as blocks, tiles, regions formed by tiles grouped together, or regions without defining tiles first. This may also create a tile merge map or region boundary map as needed. Motion estimation may then be applied to determine the delta motion vector (Amv) for each portion. This may be performed on all blocks or other portions in a frame, or merely selected blocks or selected portions of a frame.
• the process 3400 may then continue with "determine motion vector-based prediction for the individual portions" 3405.
• a motion predictor such as those described above (such as predictor 123 or 1814) may determine a prediction portion (prediction block, tile, sub-tile, region, sub-region, or other partition) that is provided to a prediction selector. The prediction, if selected, may be used for comparison to the original frame to determine whether a difference or residual exists that warrants coding.
• the process 3400 may comprise "perform local-global motion compensation on individual portions" 3406.
• the portion once again may be tiles, regions formed by grouping tiles, or regions without tile groupings, and may be object associated regions.
• the gmt or more accurately the dmt are applied at the boundary, such as the corners or vertices, of each portion where the compensation is desired. By one form, this is applied to all of the portions in a frame, but need not always be.
• a tile merge map or region boundary map may also be formed as needed.
• the process 3400 may also comprise "determine local GMC based prediction for the individual portions" 3407.
• the pixel values of the warped portions are used as predictions, and provided to a prediction selector on a portion (block, tile, sub-tile, region, sub- region, or other partition) basis as described previously.
• the process 3400 also comprises "select best prediction for coding of individual portions" 3408. Particularly, the prediction selector compares the different predictions to the original frame and selects the best fit, or may use other criteria. If a difference exists, the difference or residual is placed in the bitstream for transmission to the decoder.
• the process 3400 also comprises "transmit motion data and dominant motion compensation parameters to decoder" 3409.
• the motion data may include residuals and motion vectors
• the dominant motion compensation parameters may include the indicator or map of the selected prediction (dmsi), any tile merge map and/or region bndry map, the dmt and/or gmt trajectories, as well as the delta motion vectors Amvs, and these may be provided on a block, tile, region or other portion basis as needed. Where the same trajectories or motion vectors are applied to all or multiple portions in a region, the values may only need to be sent once with an explanatory map.
• the process 3400 also comprises "receive and decode bitstream" 3410. This may include parsing the bitstream into dominant motion compensation data, motion or other residual data, and image or frame data, and then entropy decoding the data.
• the process 3400 also comprises "perform inverse decoding operations to obtain dominant motion compensation parameters with dominant or global motion compensation trajectories, delta motion vectors, prediction selection map, merge map, and/or boundary map" 3411. This will finally reconstruct that values for each type of data.
• the process 3400 also comprises "perform dominant (local global) or motion vector based global motion compensation to obtain DMC reference frames or DMC portions" 3412.
• the DMC reference frames may be reconstructed, and the same frame or frame portion may be provided with alternative DMC reference frames or portions.
• This may include applying local global motion trajectories to a boundary to warp individual portions, and alternatively, applying delta motion vectors to portions on a warped reference frame, in order to obtain the pixel values at the resulting prediction portions.
• the process 3400 also comprises "provide the predictions to a predictions selector" 3413 where the best prediction for a frame portion is selected and used to form the final frame for display, storage, further encoding, and so forth.
• various components of the systems described herein may be implemented in software, firmware, and/or hardware and/or any combination thereof.
• various components of system 300 may be provided, at least in part, by hardware of a computing System-on-a-Chip (SoC) such as may be found in a computing system such as, for example, a smart phone.
• SoC System-on-a-Chip
• a computing system such as, for example, a smart phone.
• SoC System-on-a-Chip
• systems described herein may include additional components that have not been depicted in the corresponding figures.
• the systems discussed herein may include additional components such as bit stream multiplexer or de -multiplexer modules and the like that have not been depicted in the interest of clarity.
• implementation of the example processes herein may include the undertaking of all operations shown in the order illustrated, the present disclosure is not limited in this regard and, in various examples, implementation of the example processes herein may include the undertaking of only a subset of the operations shown and/or in a different order than illustrated.
• process 1300, 2600, and 3400, and other processes discussed herein may be illustrated in one or more examples of implementations discussed herein and, in particular, with respect to FIG. 35 below.
• FIG. 35 is an illustrative diagram of example video coding system 3500, arranged in accordance with at least some implementations of the present disclosure.
• video coding system 3500 may include imaging device(s) 3501, video encoder 100, video decoder 200 (and/or a video coder implemented via logic circuitry 3550 of processing unit(s) 3520), an antenna 3502, one or more processor(s) 3503, one or more memory store(s) 3504, and/or a display device 3505.
• imaging device(s) 3501, antenna 3502, processing unit(s) 3520, logic circuitry 3550, video encoder 100, video decoder 200, processor(s) 3503, memory store(s) 3504, and/or display device 3505 may be capable of communication with one another.
• video coding system 3500 may include only video encoder 100 or only video decoder 200 in various examples.
• video coding system 3500 may include antenna 3502. Antenna 3502 may be configured to transmit or receive an encoded bitstream of video data, for example. Further, in some examples, video coding system 3500 may include display device 3505. Display device 3505 may be configured to present video data. As shown, in some examples, logic circuitry 3550 may be implemented via processing unit(s) 3520. Processing unit(s) 3520 may include application-specific integrated circuit (ASIC) logic, graphics processor(s), general purpose processor(s), or the like. Video coding system 3500 also may include optional processor(s) 3503, which may similarly include application-specific integrated circuit (ASIC) logic, graphics processor(s), general purpose processor(s), or the like.
• ASIC application-specific integrated circuit
• logic circuitry 3550 may be implemented via hardware, video coding dedicated hardware, or the like, and processor(s) 3503 may implemented general purpose software, operating systems, or the like.
• memory store(s) 3504 may be any type of memory such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.), and so forth.
• memory store(s) 3504 may be implemented by cache memory.
• logic circuitry 3550 may access memory store(s) 3504 (for implementation of an image buffer for example).
• logic circuitry 3550 and/or processing unit(s) 3520 may include memory stores (e.g., cache or the like) for the implementation of an image buffer or the like.
• video encoder 100 implemented via logic circuitry may include an image buffer (e.g., via either processing unit(s) 3520 or memory store(s) 3504)) and a graphics processing unit (e.g., via processing unit(s) 3520).
• the graphics processing unit may be communicatively coupled to the image buffer.
• the graphics processing unit may include video encoder 100 as implemented via logic circuitry 3550 to embody the various modules as discussed with respect to FIG. 1 and/or any other encoder system or subsystem described herein.
• the graphics processing unit may include coding partitions generator logic circuitry, adaptive transform logic circuitry, content pre-analyzer, encode controller logic circuitry, adaptive entropy encoder logic circuitry, and so on.
• the logic circuitry may be configured to perform the various operations as discussed herein.
• Video decoder 200 may be implemented in a similar manner as implemented via logic circuitry 3550 to embody the various modules as discussed with respect to decoder 200 of
• FIG. 2 and/or any other decoder system or subsystem described herein.
• antenna 3502 of video coding system 3500 may be configured to receive an encoded bitstream of video data.
• the encoded bitstream may include data associated with the coding partition (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and/or data defining the coding partition (e.g., data associated with defining bi-tree partitions or k-d tree partitions using a symbol-run coding or codebook technique or the like)).
• Video coding system 3500 may also include video decoder 200 coupled to antenna 3502 and configured to decode the encoded bitstream.
• the decoder system may include a video decoder configured to decode an encoded bitstream.
• the video decoder may be further configured to receive the bitstream.
• features described herein may be undertaken in response to instructions provided by one or more computer program products.
• Such program products may include signal bearing media providing instructions that, when executed by, for example, a processor, may provide the functionality described herein.
• the computer program products may be provided in any form of one or more machine-readable media.
• a processor including one or more processor core(s) may undertake one or more features described herein in response to program code and/or instructions or instruction sets conveyed to the processor by one or more machine -readable media.
• a machine-readable medium may convey software in the form of program code and/or instructions or instruction sets that may cause any of the devices and/or systems described herein to implement at least portions of the features described herein.
• FIG. 36 is an illustrative diagram of an example system 3600, arranged in accordance with at least some implementations of the present disclosure.
• system 3600 may be a media system although system 3600 is not limited to this context.
• system 3600 may be incorporated into a personal computer (PC), laptop computer, ultra-laptop computer, tablet, touch pad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cellular telephone, combination cellular telephone/PDA, television, smart device (e.g., smart phone, smart tablet or smart television), mobile internet device (MID), messaging device, data communication device, cameras (e.g. point-and-shoot cameras, super-zoom cameras, digital single-lens reflex (DSLR) cameras), and so forth.
• PC personal computer
• laptop computer ultra-laptop computer
• tablet touch pad
• portable computer handheld computer
• palmtop computer personal digital assistant
• MID mobile internet device
• MID mobile internet device
• MID mobile internet device
• MID mobile internet device
• MID mobile internet device
• system 3600 includes a platform 3602 coupled to a display 3620.
• Platform 3602 may receive content from a content device such as content services device(s) 3630 or content delivery device(s) 3640 or other similar content sources.
• a navigation controller 3650 including one or more navigation features may be used to interact with, for example, platform 3602 and/or display 3620. Each of these components is described in greater detail below.
• platform 3602 may include any combination of a chipset 3605, processor 3610, memory 3612, antenna 3613, storage 3614, graphics subsystem 3615, applications 3616 and/or radio 3618.
• Chipset 3605 may provide intercommunication among processor 3610, memory 3612, storage 3614, graphics subsystem 3615, applications 3616 and/or radio 3618.
• chipset 3605 may include a storage adapter (not depicted) capable of providing intercommunication with storage 3614.
• Processor 3610 may be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors, x86 instruction set compatible processors, multi-core, or any other microprocessor or central processing unit (CPU). In various implementations, processor 3610 may be dual-core processor(s), dual-core mobile processor(s), and so forth.
• CISC Complex Instruction Set Computer
• RISC Reduced Instruction Set Computer
• processor 3610 may be dual-core processor(s), dual-core mobile processor(s), and so forth.
• Memory 3612 may be implemented as a volatile memory device such as, but not limited to, a Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), or Static RAM (SRAM).
• RAM Random Access Memory
• DRAM Dynamic Random Access Memory
• SRAM Static RAM
• Storage 3614 may be implemented as a non- volatile storage device such as, but not limited to, a magnetic disk drive, optical disk drive, tape drive, an internal storage device, an attached storage device, flash memory, battery backed-up SDRAM (synchronous DRAM), and/or a network accessible storage device.
• storage 3614 may include technology to increase the storage performance enhanced protection for valuable digital media when multiple hard drives are included, for example.
• Graphics subsystem 3615 may perform processing of images such as still or video for display.
• Graphics subsystem 3615 may be a graphics processing unit (GPU) or a visual processing unit (VPU), for example.
• An analog or digital interface may be used to communicatively couple graphics subsystem 3615 and display 3620.
• the interface may be any of a High-Definition Multimedia Interface, DisplayPort, wireless HDMI, and/or wireless HD compliant techniques.
• Graphics subsystem 3615 may be integrated into processor 3610 or chipset 3605. In some implementations, graphics subsystem 3615 may be a stand-alone device communicatively coupled to chipset 3605.
• graphics and/or video processing techniques described herein may be implemented in various hardware architectures.
• graphics and/or video functionality may be integrated within a chipset.
• a discrete graphics and/or video processor may be used.
• the graphics and/or video functions may be provided by a general purpose processor, including a multi-core processor.
• the functions may be implemented in a consumer electronics device.
• Radio 3618 may include one or more radios capable of transmitting and receiving signals using various suitable wireless communications techniques. Such techniques may involve communications across one or more wireless networks.
• Example wireless networks include (but are not limited to) wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area network (WMANs), cellular networks, and satellite networks. In communicating across such networks, radio 3618 may operate in accordance with one or more applicable standards in any version.
• display 3620 may include any television type monitor or display.
• Display 3620 may include, for example, a computer display screen, touch screen display, video monitor, television-like device, and/or a television.
• Display 3620 may be digital and/or analog.
• display 3620 may be a holographic display.
• display 3620 may be a transparent surface that may receive a visual projection.
• projections may convey various forms of information, images, and/or objects.
• such projections may be a visual overlay for a mobile augmented reality (MAR) application.
• MAR mobile augmented reality
• platform 3602 may display user interface 3622 on display 3620.
• content services device(s) 3630 may be hosted by any national, international and/or independent service and thus accessible to platform 3602 via the Internet, for example.
• Content services device(s) 3630 may be coupled to platform 3602 and/or to display 3620.
• Platform 3602 and/or content services device(s) 3630 may be coupled to a network 3660 to communicate (e.g., send and/or receive) media information to and from network 3660.
• Content delivery device(s) 3640 also may be coupled to platform 3602 and/or to display 3620.
• content services device(s) 3630 may include a cable television box, personal computer, network, telephone, Internet enabled devices or appliance capable of delivering digital information and/or content, and any other similar device capable of unidirectionally or bidirectionally communicating content between content providers and platform 3602 and/display 3620, via network 3660 or directly. It will be appreciated that the content may be communicated unidirectionally and/or bidirectionally to and from any one of the components in system 3600 and a content provider via network 3660. Examples of content may include any media information including, for example, video, music, medical and gaming information, and so forth.
• Content services device(s) 3630 may receive content such as cable television programming including media information, digital information, and/or other content.
• content providers may include any cable or satellite television or radio or Internet content providers. The provided examples are not meant to limit implementations in accordance with the present disclosure in any way.
• platform 3602 may receive control signals from navigation controller 3650 having one or more navigation features.
• the navigation features of controller 3650 may be used to interact with user interface 3622, for example.
• navigation controller 3650 may be a pointing device that may be a computer hardware component (specifically, a human interface device) that allows a user to input spatial (e.g., continuous and multi-dimensional) data into a computer.
• GUI graphical user interfaces
• televisions and monitors allow the user to control and provide data to the computer or television using physical gestures.
• Movements of the navigation features of controller 3650 may be replicated on a display (e.g., display 3620) by movements of a pointer, cursor, focus ring, or other visual indicators displayed on the display.
• a display e.g., display 3620
• the navigation features located on navigation controller 3650 may be mapped to virtual navigation features displayed on user interface 3622.
• controller 3650 may not be a separate component but may be integrated into platform 3602 and/or display 3620. The present disclosure, however, is not limited to the elements or in the context shown or described herein.
• drivers may include technology to enable users to instantly turn on and off platform 3602 like a television with the touch of a button after initial boot-up, when enabled, for example.
• Program logic may allow platform 3602 to stream content to media adaptors or other content services device(s) 3630 or content delivery device(s) 3640 even when the platform is turned "off.”
• chipset 3605 may include hardware and/or software support for 5.1 surround sound audio and/or high definition 7.1 surround sound audio, for example.
• Drivers may include a graphics driver for integrated graphics platforms.
• the graphics driver may comprise a peripheral component interconnect (PCI) Express graphics card.
• PCI peripheral component interconnect
• any one or more of the components shown in system 3600 may be integrated.
• platform 3602 and content services device(s) 3630 may be integrated, or platform 3602 and content delivery device(s) 3640 may be integrated, or platform 3602, content services device(s) 3630, and content delivery device(s) 3640 may be integrated, for example.
• platform 3602 and display 3620 may be an integrated unit. Display 3620 and content service device(s) 3630 may be integrated, or display 3620 and content delivery device(s) 3640 may be integrated, for example. These examples are not meant to limit the present disclosure.
• system 3600 may be implemented as a wireless system, a wired system, or a combination of both.
• system 3600 may include components and interfaces suitable for communicating over a wireless shared media, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, and so forth.
• a wireless shared media may include portions of a wireless spectrum, such as the RF spectrum and so forth.
• system 3600 may include components and interfaces suitable for communicating over wired communications media, such as input/output (I/O) adapters, physical connectors to connect the I/O adapter with a corresponding wired communications medium, a network interface card (NIC), disc controller, video controller, audio controller, and the like.
• wired communications media may include a wire, cable, metal leads, printed circuit board (PCB), backplane, switch fabric, semiconductor material, twisted-pair wire, co-axial cable, fiber optics, and so forth.
• Platform 3602 may establish one or more logical or physical channels to communicate information.
• the information may include media information and control information.
• Media information may refer to any data representing content meant for a user. Examples of content may include, for example, data from a voice conversation, videoconference, streaming video, electronic mail ("email") message, voice mail message, alphanumeric symbols, graphics, image, video, text and so forth. Data from a voice conversation may be, for example, speech information, silence periods, background noise, comfort noise, tones and so forth.
• Control information may refer to any data representing commands, instructions or control words meant for an automated system. For example, control information may be used to route media information through a system, or instruct a node to process the media information in a predetermined manner.
• FIG. 37 illustrates implementations of a small form factor device 3700 in which system 3700 may be embodied.
• device 3700 may be implemented as a mobile computing device a having wireless capabilities.
• a mobile computing device may refer to any device having a processing system and a mobile power source or supply, such as one or more batteries, for example.
• examples of a mobile computing device may include a personal computer (PC), laptop computer, ultra-laptop computer, tablet, touch pad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cellular telephone, combination cellular telephone/PDA, television, smart device (e.g., smart phone, smart tablet or smart television), mobile internet device (MID), messaging device, data communication device, cameras (e.g. point-and-shoot cameras, super-zoom cameras, digital single-lens reflex (DSLR) cameras), and so forth.
• PC personal computer
• laptop computer ultra-laptop computer
• tablet touch pad
• portable computer handheld computer
• palmtop computer personal digital assistant
• MID mobile internet device
• Examples of a mobile computing device also may include computers that are arranged to be worn by a person, such as a wrist computer, finger computer, ring computer, eyeglass computer, belt-clip computer, arm-band computer, shoe computers, clothing computers, and - I l l - other wearable computers.
• a mobile computing device may be implemented as a smart phone capable of executing computer applications, as well as voice communications and/or data communications.
• voice communications and/or data communications may be described with a mobile computing device implemented as a smart phone by way of example, it may be appreciated that other embodiments may be implemented using other wireless mobile computing devices as well. The embodiments are not limited in this context.
• device 3700 may include a housing 3702, a display 3704 which may include a user interface 3710, an input/output (I/O) device 3706, and an antenna 3708.
• Device 3700 also may include navigation features 3712.
• Display 3704 may include any suitable display unit for displaying information appropriate for a mobile computing device.
• I/O device 3706 may include any suitable I/O device for entering information into a mobile computing device. Examples for I/O device 3706 may include an alphanumeric keyboard, a numeric keypad, a touch pad, input keys, buttons, switches, rocker switches, microphones, speakers, voice recognition device and software, and so forth.
• Information also may be entered into device 3700 by way of microphone (not shown). Such information may be digitized by a voice recognition device (not shown). The embodiments are not limited in this context.
• implementation of the example processes herein may include the undertaking of all operations shown in the order illustrated, the present disclosure is not limited in this regard and, in various examples, implementation of the example processes herein may include the undertaking of only a subset of the operations shown and/or in a different order than illustrated.
• any one or more of the operations discussed herein may be undertaken in response to instructions provided by one or more computer program products.
• Such program products may include signal bearing media providing instructions that, when executed by, for example, a processor, may provide the functionality described herein.
• the computer program products may be provided in any form of one or more machine -readable media.
• a processor including one or more processor core(s) may undertake one or more of the operations of the example processes herein in response to program code and/or instructions or instruction sets conveyed to the processor by one or more machine-readable media.
• a machine-readable medium may convey software in the form of program code and/or instructions or instruction sets that may cause any of the devices and/or systems described herein to implement at least portions of the video systems as discussed herein.
• module refers to any combination of software logic, firmware logic and/or hardware logic configured to provide the functionality described herein.
• the software may be embodied as a software package, code and/or instruction set or instructions, and "hardware", as used in any implementation described herein, may include, for example, singly or in any combination, hardwired circuitry, programmable circuitry, state machine circuitry, and/or firmware that stores instructions executed by programmable circuitry.
• the modules may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), system on-chip (SoC), and so forth.
• a module may be embodied in logic circuitry for the implementation via software, firmware, or hardware of the coding systems discussed herein.
• Various embodiments may be implemented using hardware elements, software elements, or a combination of both.
• hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth.
• Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints.
• IP cores may be stored on a tangible, machine readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that actually make the logic or processor.
• a computer-implemented method for video coding comprising: obtaining frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; forming a warped global compensated reference frame by displacing at least one portion of the decoded reference frame by using global motion trajectories; determining a motion vector indicating the motion of the at least one portion and motion from a position based on the warped global compensated reference frame to a position at the current frame; and forming a prediction portion based, at least in part, on the motion vectors and corresponding to a portion on the current frame.
• the method also may be comprising wherein the at least one portion is at least one of: (1) a block of pixels used as a unit to divide the current frame and the reference frame into a plurality of the blocks; (2) at least one tile of pixels, each tile being at least 64 x 64 pixels, and used as a unit to divide the current frame and the reference frame into a plurality of the tiles; the method (2) comprising at least one of: (a) grouping tiles together based on common association with an object in the frame to form the at least one portion; and forming a single motion vector for each group of tiles, (b) grouping the tiles based on a merge map transmittable from an encoder to a decoder; or (3) a region of pixels shaped and sized depending on an object associated with the region, wherein a boundary of the region is at least one of: a shape that resembles the shape of the object associated with the region, and a rectangle placed around the object associated with the region; wherein the region is associated with at least one of: a background of the
• (iMVx, iMVy) is the transmitted motion vector in Sub-Pel Unit (fs) for a block at j, i) of size (W b x H b );
• i' i + (iMVy / fs)
• p j iMVx & (fs-l) (j', i') is integer motion adjusted current pixel location in Morphed Reference Image, p j ,
• Pi are the l/8 th pel phases in the Morphed Reference Image
• M ef[i']£j'] ((8-Px)*(8-p y )*Re o][x 0 ] + p x *(8-p y )*RefIy 0 ][xo+l] + P y *(8- Px )*Ref[y 0 +l][xO] + P y *p x *RefIy 0 +l][xo+l] + 31) » 6
• (j, i) is every (W s x H s ) sub-block location in current image, x and y are reference pixel coordinates in l/8th Pel accuracy;
• (xo, yo) is the integer pel location in the reference frame (Ref Image); p x , p y is the l/8th pel phase.
• n [0, W s - 1]
• k [-N,/2 - 1, +N,/2];
• MRef[i+m][j+n] SUM k (h[p y ][k]*tPred h [m+k][n])/T,
• n [0, W s - 1]
• k [-N,/2 - 1, +N,/2];
• (iMVx, iMVy) is the transmitted Motion Vector in Sub-Pel Units (fs) for a block at (j, i) of size (W b x H b ).
• i' (i + u*H s )*fs + iMVx
• (j, i) is the current block pixel location
• (u, v) is the index of every (W s x H s ) sub- block within given current block of (Wb x 3 ⁇ 4)
• (W s x H s ) sub-block is motion adjusted current pixel location in fs sub-pel accuracy
• tPredh[m][n] SUM k (h[p x ][k]*Ref[y 0 + m][x 0 + n + k])/T,
• n [0, W s -l]
• n [0, W s - l]
• a computer-implemented method for video coding comprising: obtaining frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; dividing the reference frame into a plurality of portions that are less than the area of the entire frame; performing dominant motion compensation comprising applying local global motion compensation on at least one of the portions by displacing the at least one portion of the decoded reference frame by using global motion trajectories at a boundary of the portion; and forming a prediction portion that corresponds to a portion on the current frame, and by using the pixel values of the displaced portion.
• the method may also be comprising performing local global motion compensation on a plurality of the portions by using a different set of global motion trajectories on each portion of the plurality of portions; wherein each portion is a tile, the method comprising dividing the frame into the tiles, and wherein each tile has a set of global motion trajectories; the method comprising providing the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles; wherein local global motion compensation trajectories are provided to half-tiles or quarter-tiles; the method comprising at least one of: (a) grouping a plurality of the tiles into a region, and applying the same global motion trajectories on the tiles within the same region, and different sets of global motion trajectories depending on the region, and (b) grouping a plurality of the portions into a region, and applying the same global motion trajectories on the portions within the same region, and different sets of global motion trajectories depending on the region; wherein
• a coder comprising: an image buffer; and a graphics processing unit configured to: obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; divide the reference frame into a plurality of portions that are less than the area of the entire frame; perform dominant motion compensation comprising applying local global motion compensation on at least one of the portions by displacing the at least one portion of the decoded reference frame by using global motion trajectories at a boundary of the portion; and form a prediction portion that corresponds to a portion on the current frame and by using the pixel values of the displaced portion.
• the coder may have the graphics processing unit configured to: perform local global motion compensation on a plurality of the portions by using a different set of global motion trajectories on each portion of the plurality of portions; wherein each portion is a tile, the graphics processing unit configured to divide the frame into the tiles, and wherein each tile has a set of global motion trajectories; the graphics processing unit configured to provide the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles; wherein local global motion compensation trajectories are provided to half-tiles or quarter-tiles; the graphics processing unit configured to at least one of: (a) group a plurality of the tiles into a region, and apply the same global motion trajectories on the tiles within the same region, and different sets of global motion trajectories depending on the region; and (b) group a plurality of the portions into a region, and apply the same global motion trajectories on the portions within the same region, and different sets of global motion trajectories
• a coder may comprise: an image buffer; and a graphics processing unit configured to: obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; form a warped global compensated reference frame by displacing at least one portion of the decoded reference frame by using global motion trajectories; determine a motion vector indicating the motion of the at least one portion and motion from a position based on the warped global compensated reference frame to a position at the current frame; and form a prediction portion based, at least in part, on the motion vectors and corresponding to a portion on the current frame.
• the coder may comprise wherein the at least one portion is at least one of: (1) a block of pixels used as a unit to divide the current frame and the reference frame into a plurality of the blocks; (2) at least one tile of pixels, each tile being at least 64 x 64 pixels, and used as a unit to divide the current frame and the reference frame into a plurality of the tiles; the graphics processing unit of (2) being configured to at least one of: (a) group tiles together based on common association with an object in the frame to form the at least one portion; and form a single motion vector for each group of tiles, (b) group the tiles based on a merge map transmittable from an encoder to a decoder; (3) a region of pixels shaped and sized depending on an object associated with the region, wherein a boundary of the region of (3) is at least one of: a shape that resembles the shape of the object associated with the region, and a rectangle placed around the object associated with the region; wherein the region is associated with at least one of: a
• the coder wherein the at least one portion comprises a frame divided into a background and a foreground, and wherein determining motion vectors comprises providing the background and foreground each with one motion vector;
• the graphics processing unit configured to perform dominant motion compensation comprising locally applied global motion compensation so that at least one other set of global motion trajectories are used at corners of at least one region on the frame that is less than the entire frame to form a displaced region; and use the pixel values of the displaced region to form a prediction region that corresponds to a region on the current frame;
• the graphics processing unit configured to at least one of: perform local global motion compensation on multiple regions of the frame by using a different set of global motion trajectories on each region; wherein each region is a tile, and dividing the frame into the tiles, and wherein each tile has a set of global motion trajectories; provide the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles; wherein each region is shaped and sized depending on an object associated
• (iMVx, iMVy) is the transmitted motion vector in Sub-Pel Unit (fs) for a block at j, i) of size (W b x H b );
• i' i + (iMVy / f s )
• (j', i') is integer motion adjusted current pixel location in Morphed Reference Image, p j ,
• Pi are the l/8 th pel phases in the Morphed Reference Image
• M ef[i']£j'] ((8-p x )*(8-p y )*ReiIy 0 ][xo] + Px*(8-p y )*RefIy 0 ][xo+l] + P y *(8- Px )*Ref[y 0 +l][xO] + P y *p x *RefTy 0 +l][xo+l] + 31) » 6
• (j, i) is every (W s x H s ) sub-block location in current image, x and y are reference pixel coordinates in l/8th Pel accuracy;
• (xo, yo) is the integer pel location in the reference frame (Ref Image); p x , p y is the l/8th pel phase.
• n [0, W s - 1]
• k [-N,/2 - 1 , +N,/2];
• MRef[i+m][j+n] SUM k (h[p y ][k]*tPred h [m+k][n])/T,
• n [0, W s - 1]
• k [-N,/2 - 1 , +N,/2];
• (iMVx, iMVy) is the transmitted Motion Vector in Sub-Pel Units (fs) for a block at (j, i) of size (W b x H b ).
• i' (i + u*H s )*fs + iMVx
• (j, i) is the current block pixel location
• (u, v) is the index of every (W s x H s ) sub- block within given current block of (Wb x 3 ⁇ 4)
• (W s x H s ) sub-block is motion adjusted current pixel location in fs sub-pel accuracy
• tPredh[m][n] SUM k (h[p x ][k]*Ref[yo + m][x 0 + n + k])/T,
• n [0, W s -l]
• n [0, W s - l]
• At least one computer readable memory comprising instructions, that when executed by a computing device, cause the computing device to: obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; divide the reference frame into a plurality of portions that are less than the area of the entire frame; perform dominant motion compensation comprising applying local global motion compensation on at least one of the portions by displacing the at least one portion of the decoded reference frame by using global motion trajectories at a boundary of the portion; and form a prediction portion that corresponds to a portion on the current frame and by using the pixel values of the displaced portion.
• the computer readable memory may also include wherein the instructions cause the computing device to: perform local global motion compensation on a plurality of the portions by using a different set of global motion trajectories on each portion of the plurality of portions; wherein each portion is a tile, the instructions cause the computing device to divide the frame into the tiles, and wherein each tile has a set of global motion trajectories; the instructions cause the computing device to provide the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles; wherein local global motion compensation trajectories are provided to half-tiles or quarter-tiles; the instructions cause the computing device to at least one of: (a) group a plurality of the tiles into a region, and apply the same global motion trajectories on the tiles within the same region, and different sets of global motion trajectories depending on the region; and (b) group a plurality of the portions into a region, and apply the same global motion trajectories on the portions within the same region, and different
• At least one computer readable memory comprising instructions, that when executed by a computing device, cause the computing device to: obtain frames of pixel data and having a current frame and a decoded reference frame to use as a motion compensation reference frame for the current frame; form a warped global compensated reference frame by displacing at least one portion of the decoded reference frame by using global motion trajectories; determine a motion vector indicating the motion of the at least one portion and motion from a position based on the warped global compensated reference frame to a position at the current frame; and form a prediction portion based, at least in part, on the motion vectors and corresponding to a portion on the current frame.
• the computer readable memory may also comprise wherein the at least one portion is at least one of: (1) a block of pixels used as a unit to divide the current frame and the reference frame into a plurality of the blocks; (2) at least one tile of pixels, each tile being at least 64 x 64 pixels, and used as a unit to divide the current frame and the reference frame into a plurality of the tiles; the instructions causing the computing device of
• a boundary of the region is at least one of: a shape that resembles the shape of the object associated with the region, and a rectangle placed around the object associated with the region; wherein the region is associated with at least one of: a background of the frame, a foreground of the frame, and a moving object in the frame; the instructions causing the computing device to define the region based on a boundary map transmittable from an encoder to a decoder; wherein form a warped global compensated reference frame comprises using the global motion trajectories at the outer corners of the frame; wherein form a warped global compensated reference frame comprises using an affine or perspective global motion compensation method.
• the memory wherein the at least one portion comprises a frame divided into a background and a foreground, and wherein determining motion vectors comprises providing the background and foreground each with one motion vector; the instructions causing the computing device to perform dominant motion compensation comprising locally applied global motion compensation so that at least one other set of global motion trajectories are used at corners of at least one region on the frame that is less than the entire frame to form a displaced region; and use the pixel values of the displaced region to form a prediction region that corresponds to a region on the current frame; the instructions causing the computing device to at least one of: (a) perform local global motion compensation on multiple regions of the frame by using a different set of global motion trajectories on each region; (b) wherein each region is a tile, and dividing the frame into the tiles, and wherein each tile has a set of global motion trajectories; (c) provide the option to perform local global motion compensation on a fraction of a tile in addition to entire tiles; wherein each region is shaped
• At least one machine readable medium may include a plurality of instructions that in response to being executed on a computing device, causes the computing device to perform the method according to any one of the above examples.
• an apparatus may include means for performing the methods according to any one of the above examples.
• the above examples may include specific combination of features. However, such the above examples are not limited in this regard and, in various implementations, the above examples may include the undertaking only a subset of such features, undertaking a different order of such features, undertaking a different combination of such features, and/or undertaking additional features than those features explicitly listed. For example, all features described with respect to the example methods may be implemented with respect to the example apparatus, the example systems, and/or the example articles, and vice versa.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• Discrete Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Television Systems (AREA)
• Image Analysis (AREA)

Applications Claiming Priority (2)

Publications (2)

Family

ID=54198733

Family Applications (2)

Family Applications After (1)

Country Status (4)

Cited By (1)

Families Citing this family (24)

Family Cites Families (15)

• 2014
  • 2014-02-24 KR KR1020167014066A patent/KR101789954B1/ko active IP Right Grant
  • 2014-03-12 EP EP14873838.8A patent/EP3087745A4/de not_active Withdrawn
  • 2014-03-12 CN CN201480070427.7A patent/CN105850133B/zh not_active Expired - Fee Related
  • 2014-04-14 EP EP14874767.8A patent/EP3087744B1/de active Active
  • 2014-11-27 TW TW103141224A patent/TWI583179B/zh not_active IP Right Cessation

Also Published As

Similar Documents

Legal Events