US8831086B2  Prediction techniques for interpolation in video coding  Google Patents
Prediction techniques for interpolation in video coding Download PDFInfo
 Publication number
 US8831086B2 US8831086B2 US12420333 US42033309A US8831086B2 US 8831086 B2 US8831086 B2 US 8831086B2 US 12420333 US12420333 US 12420333 US 42033309 A US42033309 A US 42033309A US 8831086 B2 US8831086 B2 US 8831086B2
 Authority
 US
 Grant status
 Grant
 Patent type
 Prior art keywords
 filter coefficients
 set
 residual values
 filter
 pixel
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Expired  Fee Related, expires
Links
Images
Classifications

 H04N19/00896—

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding

 H04N19/0009—

 H04N19/00139—

 H04N19/00175—

 H04N19/00357—

 H04N19/00381—

 H04N19/00551—

 H04N19/00569—

 H04N19/0063—

 H04N19/00739—

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
 H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
 H04N19/124—Quantisation

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
 H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
 H04N19/136—Incoming video signal characteristics or properties

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
 H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
 H04N19/146—Data rate or code amount at the encoder output
 H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
 H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
 H04N19/192—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding the adaptation method, adaptation tool or adaptation type being iterative or recursive

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
 H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
 H04N19/196—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters
 H04N19/198—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/46—Embedding additional information in the video signal during the compression process
 H04N19/463—Embedding additional information in the video signal during the compression process by compressing encoding parameters before transmission

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
 H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
 H04N19/51—Motion estimation or motion compensation
 H04N19/523—Motion estimation or motion compensation with subpixel accuracy

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/80—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation
 H04N19/82—Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation involving filtering within a prediction loop

 H04N7/26808—

 H04N7/364—

 H04N19/00066—

 H04N19/00303—

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
 H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
 H04N19/117—Filters, e.g. for preprocessing or postprocessing

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
 H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
 H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
 H04N19/182—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a pixel
Abstract
Description
This application claims the benefit of U.S. Provisional Application 61/044,020 filed on Apr. 10, 2008, U.S. Provisional Application 61/044,023 filed on Apr. 10, 2008, U.S. Provisional Application 61/044,240 filed on Apr. 11, 2008, and U.S. Provisional Application No. 61/057,373 filed on May 30, 2008, the entire contents of which are incorporated herein by reference.
This disclosure relates to digital video encoding and decoding and, more particularly, filtering techniques applied to generate predictive data used in the video encoding and decoding.
Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, video gaming devices, video game consoles, cellular or satellite radio telephones, and the like. Digital video devices implement video compression techniques, such as those described in standards defined by MPEG2, MPEG4, or ITUT H.264/MPEG4, Part 10, Advanced Video Coding (AVC), to transmit and receive digital video information more efficiently. Video compression techniques may perform spatial prediction and/or temporal prediction to reduce or remove redundancy inherent in video sequences.
Block based intercoding is a very useful coding technique that relies on temporal prediction to reduce or remove temporal redundancy between video blocks of successive coded units of a video sequence. The coded units may comprise video frames, slices of video frames, groups of pictures, or another defined unit of encoded video blocks. For intercoding, the video encoder performs motion estimation and motion compensation to track the movement of corresponding video blocks of two or more adjacent coded units. Motion estimation generates motion vectors, which indicate the displacement of video blocks relative to corresponding prediction video blocks in one or more reference frames or other coded units. Motion compensation uses the motion vectors to generate prediction video blocks from the one or more reference frames or other coded units. After motion compensation, residual video blocks are formed by subtracting prediction video blocks from the original video blocks being coded.
The video encoder may also apply transform, quantization and entropy coding processes to further reduce the bit rate associated with communication of residual blocks. Transform techniques may comprise discrete cosine transforms (DCTs) or conceptually similar processes. Alternatively, wavelet transforms, integer transforms, or other types of transforms may be used. In a DCT process, as an example, a set of pixel values are converted into transform coefficients, which may represent the energy of the pixel values in the frequency domain. Quantization is applied to the transform coefficients, and generally involves a process that reduces the number of bits associated with any given transform coefficient. Entropy coding comprises one or more processes that collectively compress a sequence of coding modes, motion information, coded block patterns, and quantized transform coefficients. Examples of entropy coding include but are not limited to content adaptive variable length coding (CAVLC) and context adaptive binary arithmetic coding (CABAC).
A coded video block may be represented by prediction information that can be used to create or identify a predictive block, and a residual block of data indicative of differences between the block being coded and the predictive block. The prediction information may comprise the one or more motion vectors that are used to identify the predictive block of data. Given the motion vectors, the decoder is able to reconstruct the predictive blocks that were used to code the residual. Thus, given a set of residual blocks and a set of motion vectors (and possibly some additional syntax), the decoder can reconstruct a video frame that was originally encoded. Intercoding based on motion estimation and motion compensation can achieve very good compression because successive video frames or other types of coded units are often very similar. An encoded video sequence may comprise blocks of residual data, motion vectors, and possibly other types of syntax.
Interpolation techniques have been developed in order to improve the level of compression that can be achieved in intercoding. In this case, the predictive data generated during motion compensation, which is used to code a video block, may be interpolated from the pixels of video blocks of the video frame or other coded unit used in motion estimation. Interpolation is often performed to generate predictive half pixel (halfpel) values and predictive quarter pixel (quarterpel) values. The half and quarterpel values are associated with subpixel locations. Fractional motion vectors may be used to identify video blocks at the subpixel resolution in order to capture fractional movement in a video sequence, and thereby provide predictive blocks that are more similar to the video blocks being coded than the integer video blocks.
In general, this disclosure describes filtering techniques applied by an encoder and a decoder during the prediction stage of a video encoding and/or decoding process. The described filtering techniques may enhance the accuracy of predictive data used during fractional interpolation, and in some cases, may improve predictive data of integer blocks of pixels. There are several aspects to this disclosure, including a useful twelvepixel filter support that may be used for interpolation, techniques that use coefficient symmetry and pixel symmetry to reduce the amount of data needed to be sent between an encoder and a decoder to configure the filter support for interpolation, and techniques for filtering data at integer pixel locations in a manner that is similar to subpixel interpolation. Other aspects of this disclosure concern techniques for encoding information in the bitstream to convey the type of filter used, and possibly the filter coefficients used. Predictive encoding techniques for filter coefficients are also described. These and other aspects of this disclosure will become apparent from the description below.
In one example, this disclosure describes a method comprising identifying a set of filter coefficients for interpolation of predictive data in video encoding, generating residual values associated with the set of filter coefficients based on predictive coding of the set of filter coefficients relative to filter coefficients associated with a fixed interpolation filter, applying quantization to the residual values, and entropy coding and outputting the quantized residual values as part of an encoded bitstream.
In another example, this disclosure describes a method comprising receiving residual values associated with a set of filter coefficients, generating the set of filter coefficients using predictive decoding based on the set of residual values and filter coefficients associated with a fixed interpolation filter, and applying the set of filter coefficients to interpolate predictive data used for predictive decoding of video blocks.
In another example, this disclosure describes an apparatus comprising a video encoder that identifies a set of filter coefficients for interpolation of predictive data in video encoding, generates residual values associated with the set of filter coefficients based on predictive coding of the set of filter coefficients relative to filter coefficients associated with a fixed interpolation filter, applies quantization to the residual values, and entropy codes and outputs the quantized residual values as part of an encoded bitstream.
In another example, this disclosure describes an apparatus comprising a video decoder that receives residual values associated with a set of filter coefficients, generates the set of filter coefficients using predictive decoding based on the set of residual values and filter coefficients associated with a fixed interpolation filter, and applies the set of filter coefficients to interpolate predictive data used for predictive decoding of video blocks.
In another example, this disclosure describes a device comprising means for identifying a set of filter coefficients for interpolation of predictive data in video encoding, means for generating residual values associated with the set of filter coefficients based on predictive coding of the set of filter coefficients relative to filter coefficients associated with a fixed interpolation filter, means for applying quantization to the residual values, and means for entropy coding and outputting the quantized residual values as part of an encoded bitstream.
In another example, this disclosure describes a device comprising means for receiving residual values associated with a set of filter coefficients, means for generating the set of filter coefficients using predictive decoding based on the set of residual values and filter coefficients associated with a fixed interpolation filter, and means for applying the set of filter coefficients to interpolate predictive data used for predictive decoding of video blocks.
The techniques described in this disclosure may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the software may be executed in one or more processors, such as a microprocessor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), or digital signal processor (DSP). The software that executes the techniques may be initially stored in a computerreadable medium and loaded and executed in the processor.
Accordingly, this disclosure also contemplates a computerreadable storage medium comprising instructions that when executed by a processor cause the processor to identify a set of filter coefficients for interpolation of predictive data in video encoding, generate residual values associated with the set of filter coefficients based on predictive coding of the set of filter coefficients relative to filter coefficients associated with a fixed interpolation filter, apply quantization to the residual values, and entropy code and output the quantized residual values as part of an encoded bitstream.
In another example, this disclosure describes a computerreadable storage medium comprising instructions that when executed by a processor cause the processor to upon receiving residual values associated with a set of filter coefficients, generate the set of filter coefficients using predictive decoding based on the set of residual values and filter coefficients associated with a fixed interpolation filter, and apply the set of filter coefficients to interpolate predictive data used for predictive decoding of video blocks.
The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.
This disclosure describes filtering techniques applied by an encoder and a decoder during the prediction stage of a video encoding and/or decoding process. The described filtering techniques may enhance the accuracy of predictive data used during fractional interpolation, and in some cases, may improve predictive data of integer blocks of pixels. There are several aspects to this disclosure, including a useful twelvepixel filter support that may be used for interpolation, techniques that use coefficient symmetry and pixel symmetry to reduce the amount of data needed to be sent between an encoder and a decoder to configure the filter support for interpolation, and techniques for filtering data at integer pixel locations in a manner that is similar to subpixel interpolation. These and other techniques are described in detail below.
In the example of
Again, the illustrated system 10 of
Video source 20 of source device 12 may include a video capture device, such as a video camera, a video archive containing previously captured video, or a video feed from a video content provider. As a further alternative, video source 20 may generate computer graphicsbased data as the source video, or a combination of live video, archived video, and computergenerated video. In some cases, if video source 20 is a video camera, source device 12 and destination device 16 may form socalled camera phones or video phones. In each case, the captured, precaptured or computergenerated video may be encoded by video encoder 22. The encoded video information may then be modulated by modem 23 according to a communication standard, e.g., such as code division multiple access (CDMA) or another communication standard, and transmitted to destination device 16 via transmitter 24 and communication channel 15. Modem 23 may include various mixers, filters, amplifiers or other components designed for signal modulation. Transmitter 24 may include circuits designed for transmitting data, including amplifiers, filters, and one or more antennas.
Receiver 26 of destination device 16 receives information over communication channel 15, and modem 27 demodulates the information. Like transmitter 24, receiver 26 may include circuits designed for receiving data, including amplifiers, filters, and one or more antennas. In some instances, transmitter 24 and/or receiver 26 may be incorporated within a single transceiver component that include both receive and transmit circuitry. Modem 27 may include various mixers, filters, amplifiers or other components designed for signal demodulation. In some instances, modems 23 and 27 may include components for performing both modulation and demodulation.
Again, the video encoding process performed by video encoder 22 may implement one or more of the techniques described herein during motion compensation. The video decoding process performed by video decoder 28 may also perform such techniques during its motion compensation stage of the decoding process. The term “coder” is used herein to refer to a specialized computer device or apparatus that performs video encoding or video decoding. The term “coder” generally refers to any video encoder, video decoder, or combined encoder/decoder (codec). The term “coding” refers to encoding or decoding. Display device 30 displays the decoded video data to a user, and may comprise any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.
In the example of
Video encoder 22 and video decoder 28 may operate according to a video compression standard, such as the ITUT H.264 standard, alternatively described as MPEG4, Part 10, Advanced Video Coding (AVC). The techniques of this disclosure, however, are not limited to any particular video coding standard. Although not shown in
Video encoder 22 and video decoder 28 each may be implemented as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any combinations thereof. Each of video encoder 22 and video decoder 28 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined codec that provides encoding and decoding capabilities in a respective mobile device, subscriber device, broadcast device, server, or the like.
A video sequence typically includes a series of video frames. Video encoder 22 operates on video blocks within individual video frames in order to encode the video data. The video blocks may have fixed or varying sizes, and may differ in size according to a specified coding standard. Each video frame includes a series of slices. Each slice may include a series of macroblocks, which may be arranged into subblocks. As an example, the ITUT H.264 standard supports intra prediction in various block sizes, such as 16 by 16, 8 by 8, or 4 by 4 for luma components, and 8×8 for chroma components, as well as inter prediction in various block sizes, such as 16 by 16, 16 by 8, 8 by 16, 8 by 8, 8 by 4, 4 by 8 and 4 by 4 for luma components and corresponding scaled sizes for chroma components. Video blocks may comprise blocks of pixel data, or blocks of transformation coefficients, e.g., following a transformation process such as discrete cosine transform (DCT) or a conceptually similar transformation process.
Smaller video blocks can provide better resolution, and may be used for locations of a video frame that include high levels of detail. In general, macroblocks and the various subblocks may be considered to be video blocks. In addition, a slice may be considered to be a series of video blocks, such as macroblocks and/or subblocks. Each slice may be an independently decodable unit of a video frame. Alternatively, frames themselves may be decodable units, or other portions of a frame may be defined as decodable units. The term “coded unit” refers to any independently decodable unit of a video frame such as an entire frame, a slice of a frame, or another independently decodable unit defined according to the coding techniques used.
To encode the video blocks, video encoder 22 performs intra or interprediction to generate a prediction block. Video encoder 22 subtracts the prediction blocks from the original video blocks to be encoded to generate residual blocks. Thus, the residual blocks are indicative of differences between the blocks being coded and the prediction blocks. Video encoder 22 may perform a transform on the residual blocks to generate blocks of transform coefficients. Following intra or interbased predictive coding and transformation techniques, video encoder 22 performs quantization. Quantization generally refers to a process in which coefficients are quantized to possibly reduce the amount of data used to represent the coefficients. Following quantization, entropy coding may be performed according to an entropy coding methodology, such as context adaptive variable length coding (CAVLC) or context adaptive binary arithmetic coding (CABAC). More details of each step of the encoding process performed by video encoder 22 will be described in more detail below in
In destination device 16, video decoder 28 receives the encoded video data. Video decoder 28 entropy decodes the received video data according to an entropy coding methodology, such as CAVLC or CABAC, to obtain the quantized coefficients. Video decoder 28 applies inverse quantization (dequantization) and inverse transform functions to reconstruct the residual block in the pixel domain. Video decoder 28 also generates a prediction block based on control information or syntax information (e.g., coding mode, motion vectors, syntax that defines filter coefficients and the like) included in the encoded video data. Video decoder 28 sums the prediction block with the reconstructed residual block to produce a reconstructed video block for display. More details of each step of the encoding process performed by video encoder 22 will be described in more detail below in
According to the techniques of this disclosure, video encoder 22 and video decoder 28 may use the one or more interpolation filtering techniques during motion compensation. In particular, in accordance with one aspect of this disclosure, video encoder 22 and/or video decoder 28 may obtain a block of pixels, wherein the block of pixels includes integer pixel values corresponding to integer pixel positions within the block of pixels, compute subpixel values for subpixel positions associated with the block of pixels based on the integer pixel values, wherein computing the subpixel values comprises applying an interpolation filter that defines a twodimensional array of filter support positions corresponding to a set of twelve or more integer pixel positions that surround the subpixel positions in a radial shape, and generate a prediction block based on at least some of the subpixel values. An example of the twodimensional array of filter support positions corresponding to a set of twelve or more integer pixel positions is explained in greater detail below.
In accordance with another aspect of this disclosure, video encoder 22 and/or video decoder 28 may utilize aspects of symmetry in order to reduce the amount of data that needs to be communicated between source device 12 and destination device 16 for communication of filter coefficients used in interpolation. Video encoder 22 may determine eight sets of filter coefficients for fifteen different subpixel locations, wherein the eight sets of filter coefficients are generated based on coefficient symmetry and pixel symmetry among fifteen subpixel locations, and output the eight sets of filter coefficients to another device as part of an encoded bitstream. In this way, the eight sets, along with aspects of pixel symmetry and coefficients symmetry may define all of the filter coefficients for all fifteen halfpel and quarterpel pixel positions. Moreover, pixel symmetry may exist between different ones of the fifteen subpixel locations in a vertical dimension and in a horizontal dimension, but pixel symmetry may not exist in a diagonal dimension for at least some of the fifteen sets subpixel locations. This lack of pixel symmetry in the diagonal dimension for at least some of the fifteen locations may improve interpolations and video quality in the video encoding and decoding.
Video decoder 28 of destination device 16 may receive the eight sets of filter coefficients as part of an encoded video bitstream, the generate fifteen sets of filter coefficients corresponding to fifteen different subpixel locations based on the eight sets of filter coefficients, generate interpolated predictive data for video decoding based on one of the fifteen sets of filter coefficients, wherein the interpolated predictive data corresponds to one of the fifteen different subpixel locations, and decode one or more video blocks base on the interpolated predictive data.
In accordance with another aspect of this disclosure, video encoder 22 and/or video decoder 28 may utilize interpolationlike filtering with respect to integer pixel positions in order to generate adjusted integer pixel values. Such interpolationlike filtering may improve compression specifically during illumination changes, scene fadeins or fadeouts, may remove noise and facilitate image frame sharpening, and may help improve encoding of fine object movement between successive video frames particularly when symmetry is not imposed on filter coefficients.
The interpolationlike filtering techniques of video encoder 22 and/or video decoder 28 may include obtaining blocks of pixels, wherein the blocks of pixels includes integer pixel values corresponding to integer pixel positions within the blocks of pixels, filtering the integer pixel values based on other integer pixel values within the block of pixels to generate adjusted integer pixel values, wherein the adjusted integer pixel values correspond to the integer pixel positions, and generating a prediction block based on the adjusted integer pixel values.
In accordance with another aspect of this disclosure video encoder 22 may generate first interpolated predictive data for encoding of video data based on a first interpolation filter, generate second interpolated predictive data for video encoding of the video data based on a second interpolation filter, select between the first interpolated predictive data and the second interpolated predictive data based on a ratedistortion analysis, encode the video data based on the selection, and encode syntax to indicate the selection. The first interpolation filter may comprise a fixed interpolation filter, and the second interpolation filter may comprise an adaptive interpolation filter, but this disclosure is not necessarily limited to these examples.
Furthermore, additional interpolation filters may also be applied to generate additional interpolated predictive data, which may also be considered in the rate distortion analysis. In other words, the techniques of this disclosure are not limited to generating only first and second interpolated predictive data based on two interpolation filters, but could be applied to generate any plurality of interpolated predictive data based on any number of interpolation filters. Importantly, a ratedistortion analysis interpolated predictive data is used to identify which filter to select.
In one example, a method may comprise generating a plurality of different versions of predictive data for encoding of video data based on a plurality of different interpolation filters, selecting among the plurality of different versions of predictive data based on a rate distortion analysis, encoding the video data based on the selection, and encoding syntax to indicate the selection.
This disclosure also contemplates technique for encoding filter coefficients. For example, video encoder 22 may identify a set of filter coefficients for interpolation of predictive data in video encoding, generate residual values associated with the set of filter coefficients based on predictive coding of the set of filter coefficients relative to filter coefficients associated with a fixed interpolation filter, apply quantization to the residual values, and output the quantized residual values as part of an encoded bitstream.
Video decoder 28 may receive residual values associated with a set of filter coefficients, generate the set of filter coefficients using predictive decoding based on the set of residual values and filter coefficients associated with a fixed interpolation filter, and apply the set of filter coefficients to interpolate predictive data used for predictive decoding of video blocks.
As shown in
Prediction unit 32 may include a motion estimation (ME) unit 35, and a motion compensation (MC) unit 37. Filter 37 may be included in prediction unit 32 and may be invoked by one or both of ME unit 35 and MC unit 37 to perform interpolation or interpolationlike filtering as part of motion estimation and/or motion compensation, according to this disclosure. Filter 37 may actually represent a plurality of different filters to facilitate numerous different types of interpolation and interpolationtype filtering as described herein. Thus, prediction unit 32 may include a plurality of interpolation or interpolationlike filters. During the encoding process, video encoder 50 receives a video block to be coded (labeled “VIDEO BLOCK” in
Motion estimation is typically considered the process of generating motion vectors, which estimate motion for video blocks. A motion vector, for example, may indicate the displacement of a prediction block within a prediction or reference frame (or other coded unit, e.g., slice) relative to the block to be coded within the current frame (or other coded unit). The reference frame (or portion of the frame) may be temporally located prior to or after the video frame (or portion of the video frame) to which the current video block belongs. Motion compensation is typically considered the process of fetching or generating the prediction block from memory 34, or possibly interpolating or otherwise generating filtered predictive data based on the motion vector determined by motion estimation.
ME unit 35 selects the appropriate motion vector for the video block to be coded by comparing the video block to video blocks of one or more reference frames (e.g., a previous and/or subsequent frame). ME unit 35 may perform motion estimation with fractional pixel precision, sometimes referred to as fractional pixel, fractional pel, or subpixel motion estimation. As such, the terms fractional pixel, fractional pel, and subpixel motion estimation may be used interchangeably. In fractional pixel motion estimation, ME unit 35 may select a motion vector that indicates displacement to a location other than an integer pixel location. In this manner, fractional pixel motion estimation allows prediction unit 32 to track motion with higher precision than integerpixel (or fullpixel) locations, thus generate a more accurate prediction block. Fractional pixel motion estimation may have halfpixel precision, quarterpixel precision, eighthpixel precision or any finer precision. ME unit 35 may invoke filter(s) 39 for any necessary interpolations during the motion estimation process.
To perform fractional pixel motion compensation, MC unit 37 may perform interpolation (sometimes referred to as interpolation filtering) in order to generate data at subpixel resolution (referred to herein as subpixel or fractional pixel values). MC unit 37 may invoke filter(s) 39 for this interpolation. Prediction unit 32 may perform the interpolation (or interpolationlike filtering of integer pixels) using the techniques described herein.
Once the motion vector for the video block to be coded is selected by ME unit 35, MC unit 37 generates the prediction video block associated with that motion vector. MC unit 37 may fetch the prediction block from memory 34 based on the motion vector determined by MC unit 35. In the case of a motion vector with fractional pixel precision, MC unit 37 filters data from memory 34 to interpolate such data to subpixel resolution, e.g., invoking filter(s) 39 for this process. In some cases, the interpolation filtering technique or mode that was used to generate the subpixel prediction data may be indicated as one or more interpolation syntax elements to entropy coding unit 46 for inclusion in the coded bitstream. Indeed, some aspects of this disclosure concern the use of pixel symmetry and coefficient symmetry to reduce the amount of syntax that needs to be conveyed.
Once prediction unit 32 has generated the prediction block, video encoder 50 forms a residual video block (labeled “RESID. BLOCK” in
Quantization unit 40 quantizes the residual transform coefficients to further reduce bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. Following quantization, entropy coding unit 46 entropy codes the quantized transform coefficients. For example, entropy coding unit 46 may perform CAVLC, CABAC, or another entropy coding methodology.
Entropy coding unit 46 may also code one or more prediction syntax elements obtained from prediction unit 32 or other component of video encoder 50. The one or more prediction syntax elements may include a coding mode, one or more motion vectors, an interpolation technique that was use to generate the subpixel data, a set or subset of filter coefficients, or other information associated with the generation of the prediction block. Coefficient prediction and quantization unit 41 may predictively encode and quantize the prediction syntax, such as filter coefficients, according to some aspects of this disclosure. Following the entropy coding by entropy coding unit 46, the encoded video and syntax elements may be transmitted to another device or archived for later transmission or retrieval.
Inverse quantization unit 42 and inverse transform unit 44 apply inverse quantization and inverse transformation, respectively, to reconstruct the residual block in the pixel domain, e.g., for later use as a reference block. The reconstructed residual block (labeled “RECON. RESID. BLOCK” in
As described above, prediction unit 32 may perform motion estimation with fractional pixel (or subpixel) precision. When prediction unit 32 uses fractional pixel motion estimation, prediction unit 32 may generate data at subpixel resolution (e.g., subpixel or fractional pixel values) using interpolation operations described in this disclosure. In other words, the interpolation operations are used to compute values at positions between the integer pixel positions. Subpixel positions located half the distance between integerpixel positions may be referred to as halfpixel (halfpel) positions, subpixel positions located half the distance between an integerpixel position and a halfpixel position may be referred to as quarterpixel (quarterpel) positions, subpixel positions located half the distance between an integerpixel position (or halfpixel position) and a quarterpixel position are referred to as eighthpixel (eighthpel) positions, and the like.
Integerpixel locations may be associated with a physical sensor element, such as a photodiode when the video data was originally generated. The photodiode may measure an intensity of a light source at the location of the sensor and associate a pixel intensity value with the integerpixel location. Again, each integerpixel location may have an associated set of fifteen subpixel locations (or possibly more). The number of subpixel locations associated with integerpixel locations may be dependent upon the desired precision. In the example illustrated in
Y may represent luminance, and Cb and Cr may represent two different values of chrominance of a threedimensional YCbCr color space. Each pixel location may actually define three pixel values for a threedimensional color space. The techniques of this disclosure, however, may refer to prediction with respect to one dimension for purposes of simplicity. To the extent that techniques are described with respect to pixel values in one dimension, similar techniques may be extended to the other dimensions.
In the example of
Prediction unit 32 of video encoder 40 may determine pixel values for subpixel locations “a” through “o” using interpolation filtering by MC unit 37. Consistent with the ITUT H.264 standard, for example, prediction unit 32 may determine pixel values for halfpixel locations using a 6tap interpolation filter, such as a Wiener filter. In the case of the H.264 standard, the filter coefficients for the 6tap interpolation filter are typically [1, −5, 20, 20, −5, 1], although other coefficients may be used. Prediction unit 32 may apply the interpolation filter first in the horizontal direction and then in the vertical direction, or vice versa. For halfpixel positions “b” and “h,” each tap may correspond to an integer pixel position in the horizontal and vertical direction, respectively. In particular, for halfpixel position “b,” the taps of the 6tap filter correspond to C1, C2, C3, C4, C5 and C6. Likewise, for halfpixel position “h,” the taps of the 6tap filter correspond to A3, B3, C3, D3, E3 and F3. For example, pixel values for subpixel positions “b” and “h” may be computed using equations (1) and (2):
b=((C1−5*C2+20*C3+20*C4−5*C5+C6)+16)/32 (1)
h=((A3−5*B3+20*C3+20*D3−5*E3+F3)+16)/32 (2)
For halfpixel position “j,” the taps of the 6tap filter correspond themselves to interpolated horizontally between positions C1C6 and D1D6, or vertically between positions A3F3 and A4F4. Halfpixel location “j” may be computed with a 6tap filter that uses previously interpolated pixel values of the halfpixel positions, e.g., in accordance with one of equations (3) or (4):
j=((aa−5*bb+20*b+20*hh−5*ii+jj)+16)/32 (3)
j=((cc−5*dd+20*h+20*ee−5*ff+gg)+16)/32 (4)
where (as illustrated in
Consistent with the H.264 standard, prediction unit 32 may determine pixel values at quarterpixel locations “a,” “c,” “d,” “e,” “f,” “g,” “i,” “k,” “l,” “m,” “n” and “o” using a bilinear interpolation filter and the pixel values of the surrounding integer and halfpixel locations. For example, prediction unit 32 may determine a pixel value associated with subpixel position “a” using pixel values of C3 and “b,” determine a pixel value associated with subpixel position “c” using pixel values of “b” and C4, and the like.
The actual filter that is applied by MC unit 37 to generate interpolated data at the subpixel locations may be subject to a wide variety of implementations. As one example, prediction unit 32 may use adaptive interpolation filtering (AIF), as described below, to define the interpolated values. The ITUT SG16/Q.6/VCEG (Video Coding Expert Group) committee has been exploring coding technologies that offer higher coding efficiency than H.264 and, in particular, AIF. AIF offers large coding gain over the interpolation filtering used in the H.264 standard, especially on video sequences with high resolution (e.g., 720i/p or 1080i/p). In AIF, the interpolation filter for each subpixel position is analytically calculated for each video frame by minimizing the prediction error energy. This helps to address aliasing, quantization and motion estimation errors, camera noise or other artifact contained in the original and reference video frames. The analytically derived adaptive filter coefficients for each frame are then predicted, quantized, coded and sent in the video bitstream. Some of the techniques of this disclosure could work within an AIF scheme, as well as many other interpolation schemes.
There are many different types of AIF schemes consistent with aspects of this disclosure. For example a first scheme is a twodimensional nonseparable AIF (NSAIF), a second is a separable AIF (SAIF), and a third is an AIF with directional filters (DAIF). Although each of these AIF schemes use different interpolation techniques and support, all three AIF schemes may use similar analytical processes to derive the filter coefficients, which is explained below using nonseparable AIF as an example.
Assume a 6by6 twodimensional nonseparable filter has coefficients h_{i,j} ^{SP }where i,j=0 . . . 5 and SP represents one of the 15 subpixel positions (“a” through “o”) shown in
Let S_{x,y }be the pixel value in the current video frame at position (x, y).
{tilde over (x)}=x+└mvx┘−FO, {tilde over (y)}=y+└mvy┘−FO,
where (mvx,mvy) is the motion vector, (└mvx┘,└mvy┘) is the integer component of the motion vector, and FO is the filter offset. The value ({tilde over (x)}, {tilde over (y)}) is the corresponding pixel position in the reference frames. For example, in the case of 6tap filter, FO=6/2−1=2. For each subpixel position SP, the prediction error energy (e^{SP})^{2 }between the actual pixel value in the current frame and the interpolated value can be accumulated by prediction unit 32 for all pixels that have motion vector precision corresponding to subpixel position SP. The prediction error energy (e^{SP})^{2 }may be calculated by prediction unit 32 using the following equation:
For each of the subpixel positions a through o, MC unit 37 may set up an individual set of equations by computing the derivative of (e^{SP})^{2 }with respect to the filter coefficients h_{i,j} ^{SP}. The number of equations, in this case, is equal to the number of filter coefficients used for the current subpixel position SP. For each twodimensional (2D) subpixel position “e,” “f,” “g,” “i,” “j,” “k,” “m,” “n,” and “o,” prediction unit 32 may use a 6by6 tap 2D interpolation filter. Subpixel positions “e,” “f,” “g,” “i,” “j,” “k,” “m,” “n,” and “o,” are 2D in the sense that they are not located in a vertical line or horizontal line between two integerpixel positions. In this case, a system of thirtysix equations with thirtysix unknowns can be solved by MC unit 37. The remaining 1D subpixel positions “a,” “b,” “c,” “d,” “h,” and “l” may only require a 1D interpolation filter (e.g., 1D 6tap filter). In the case of a 1D 6tap filter, a system of six equations can be solved by MC unit 37.
Filter(s) 39 may represent one filter or a set of many different filters that may be used by MC unit 37 to generate the predictive data.
Thus, one example process of deriving and applying the AIF filters may have the following steps, which can be performed by prediction unit 32:

 1. Estimate motion vectors (mvx,mvy) for every video block to be coded. During motion estimation, a fixed interpolation filter (e.g., the interpolation process of H.264/AVC) can be applied.
 2. Using these motion vectors, accumulate prediction error energy for each subpixel position SP over the current video frame. Then, calculate adaptive filter coefficients h_{i,j} ^{SP }for each subpixel position SP independently by minimizing the prediction error energy as in the two prediction energy equations above.
 3. Estimate new motion vectors. During this motion estimation process, the adaptive interpolation filters computed in step 2 may be applied. Using the adaptive interpolation filters, motion estimation errors, caused by aliasing, camera noise, etc., are reduced and better motion prediction is achieved.
Different AIF schemes may use the same analytical process as given above. The differences between the different schemes mostly lie in the number of unique filter coefficients used, whether the interpolation filters are separable or nonseparable, and the filter support used (i.e., integer pixel positions used to interpolate at least a portion of the subpixel positions). In each of these schemes, certain symmetry constraints on the AIF filters may be imposed to reduce the number of filter coefficients that need to be encoded and sent in the video bitstream.
For NSAIF, for example, MC unit 37 of prediction unit 32 may interpolate 1D subpixel positions “a,” “b,” “c,” “d,” “h,” and “l” using a 1D 6tap interpolation filter (also referred to as a 6position filter as each tap corresponds with a integerpixel position), which requires six coefficients. The six coefficients of the 6position filters used to interpolate the 1D subpixel each correspond with one of the integerpixel positions illustrated in
Prediction unit 32 may interpolate 2D subpixel positions “e,” “f,” “g,” “i,” “j,” “k,” “m,” “n,” and “o” using a 2D 6by6 interpolation filter, which requires thirtysix filter coefficients. The thirtysix coefficients of the 2D 6×6 interpolation filter used to interpolate the 2D subpixel each correspond with integerpixel positions A1A6, B1B6, C1C6, D1D6, E1E6 and F1F6. These integerpixel positions represent the “filter support” of the interpolation filter. If no additional restriction is imposed, e.g., no coefficient or pixel symmetry, video encoder 50 may encode and transmit nine sets of thirtysix coefficients for the 2D subpixel positions and six sets of six coefficients for the 1D subpixel positions, for a total of 360 coefficients. Encoding and sending that number of coefficients in the video bitstream may result in costly bit overhead. The high bit overhead may, in turn, increase the bit rate for a given level of distortion, which is undesirable.
To reduce the bit overhead associated with sending the filter coefficients, certain symmetry restrictions may be imposed on the interpolation filters to reduce the number of unique filter coefficients that need to be sent to decoder 28. Two types of symmetry, i.e., pixel symmetry and coefficient symmetry may be imposed, alone or in combination. Pixel symmetry enforces the same set of filter coefficients (and the mirrored, flipped and/or rotated versions of the set) for different subpixel positions. Pixel symmetry may also be referred to as subpixel symmetry insofar as such pixel symmetry applies with respect to the filter coefficients associated with two or more subpixel locations. Coefficient symmetry, on the other hand, enforces a given interpolation filter to be symmetric in a certain direction (e.g., horizontal direction, vertical direction or both) for the various filter support positions relative to other filter support positions for a given subpixel values to be interpolated.
Pixel symmetry may be used in NSAIF. Referring again to
p ^{a} =h _{0} ^{a} ·C _{1} +h _{1} ^{a} ·C _{2} +h _{2} ^{a} ·C _{3} +h _{3} ^{a} ·C _{4} +h _{4} ^{a} ·C _{5} +h _{5} ^{a} ·C _{6} (8)
p ^{c} =h _{5} ^{a} ·C _{1} +h _{4} ^{a} ·C _{2} +h _{3} ^{a} ·C _{3} +h _{2} ^{a} ·C _{4} +h _{1} ^{a} ·C _{5} +h _{0} ^{a} ·C _{6} (9)
Subpixel positions “d” and “l” may be interpolated using the same set of interpolation filter coefficients as subpixel positions “a” and “c,” respectively. As such, subpixel positions “a” and “c” may have pixel symmetry relative to subpixel positions “d” and “l.”
One aspect of this disclose is to actually remove pixel symmetry between subpixel positions “a” and “c” relative to subpixel positions “d” and “l.” In addition, pixel symmetry may be avoided for subpixel position “f” relative to subpixel position “i.” In addition, pixel symmetry may be avoided for subpixel position “k” relative to subpixel position “n.” In such cases, diagonal correlation may be low, making it inefficient or ineffective to impose diagonal symmetry in these cases.
As another example, let h_{0,0} ^{2}, h_{0,1} ^{e}, . . . , h_{0,5} ^{e}, . . . , h_{5,0} ^{e}, h_{5,1} ^{e}, . . . , h_{5,5} ^{e }be the set of 6by6 2D filter coefficients for subpixel position “e.” Then the set of filter coefficients for position “g” is h_{0,5} ^{e}, h_{0,4} ^{e}, . . . , h_{0,4} ^{e}, . . . , h_{5,5} ^{e}, h_{5,4} ^{e}, . . . , h_{5,0} ^{e }(horizontally flipped version). Similarly, the set of filter coefficients for subpixel position “m” is h_{5,0} ^{e}, h_{5,1} ^{e}, . . . , h_{5,5} ^{e}, . . . , h_{0,0} ^{e}, h_{0,1} ^{e}, . . . , h_{0,5} ^{e }(vertically flipped version), and the set of filter coefficients for subpixel position “o” is h_{5,5} ^{e}, h_{5,4} ^{e}, . . . , h_{5,0} ^{e}, . . . , h_{0,5} ^{e}, h_{0,4} ^{e}, . . . , h_{0,0} ^{e }(first horizontally flipped and then vertically flipped). As such, subpixel positions “e,” “g,” “m” and “o” have pixel symmetry. Symmetry between subpixel positions “b” and “h,” and between positions “f,” “i,” “k,” and “n” are imposed in a similar fashion as the examples given above. Under such a pixel symmetry constraint, there remain only five groups of unique filter sets, a first set of filter coefficients for subpixel group “a,” “c,” “d” and “l,” a second set of filter coefficients for subpixel group “b” and “h,” a third set of filter coefficients for subpixel group “e,” “g,” “m” and “o,” a fourth set of filter coefficients for group “f,” “i,” “k,” and “n,” and a fifth set of filter coefficients for subpixel group “j.”
Furthermore, NSAIF may impose coefficient symmetry restriction on some of these filter sets, alone or in combination with the subpixel symmetry described above. In some cases, diagonal coefficient symmetry may be purposely avoided, e.g., eliminating any diagonal coefficient symmetry constraints. In one instance, no coefficient symmetry is imposed on filter for the first group of subpixel positions, which includes subpixel locations “a,” “c,” “d” and “l.” However, the second group of subpixel positions, which includes coefficient “b” and “h,” may have coefficient symmetry in accordance with equation (10).
h _{0} ^{b} =h _{5} ^{b} , h _{1} ^{b} =h _{4} ^{b} , h _{2} ^{b} =h _{3} ^{b} (10)
Likewise, the coefficients for interpolation filters for the third, fourth and fifth groups of subpixel positions, which include subpixel positions “e,” “f,” and “j,” respectively, may have coefficient symmetry as given in equations (11), (12), and (13), respectively.
h _{i,j} ^{e} =h _{j,i} ^{e}, for i,j=0 . . . 5, i≠j (11)
h _{i,j} ^{f} =h _{i,5j} ^{f}, for i,j=0 . . . 5 (12)
h _{i,j} ^{j} =h _{i,5j} ^{j} =h _{5i,j} ^{j} =h _{5i,5j} ^{j}, for i,j=0 . . . 2
h _{i,j} ^{j} =h _{j,i} ^{j } i≠j (13)
Under such pixel symmetry and coefficient symmetry constraints described above, the number of unique filter coefficients can be reduced from 360 (no symmetry constraint) to 6(a)+3(b)+21(e)+18(f)+6(i)=54 coefficients, i.e., 6 coefficients for the group including subpixel position “a,” 3 coefficients for the group including subpixel position “b,” 21 coefficients for the group including subpixel position “e,” 18 coefficients for the group including subpixel position “f,” and 6 coefficients for the group including subpixel position “j.” Thus, in NSAIF, video encoder 22 may encode and transmits fiftyfour coefficients instead of 360, as in the case of no symmetry constraints. As described above, the fiftyfour coefficients may be analytically solved based on equation (7) above. Video encoder 50 may then predict, quantize, code (e.g, using signed ExpGolomb code), and send the coefficients in the bitstream. Additional details on the predictive coding of filter coefficients is discussed in greater detail below. Coefficient prediction and quantization unit 41 may be used for the predictive coding and quantization of filter coefficients consistent with this disclosure.
In another AIF scheme, i.e., SAIF, prediction unit 32 may use separable interpolation filters in the horizontal direction and in the vertical direction, instead of nonseparable interpolation filters as used in NSAIF. For the 1D subpixel positions, prediction unit 32 (e.g., MC unit 37 of prediction unit 32) applies only horizontal directional filters or only vertical directional filters depending on the subpixel location. In one example, the horizontal and vertical directional filters comprise 6position (or 6tap) filters. Prediction unit 32 applies horizontal directional filters for subpixel positions “a,” “b,” and “c” with integerpixel positions C1, C2, C3, C4, C5, and C6 (see
SAIF may further impose one or more symmetry restrictions on at least some of the interpolation filters. Unlike the subpixel symmetry of NSAIF, with SAIF, the filter sets used for subpixel positions “a” and “c” are not symmetric, i.e., horizontally flipped versions of each other. Instead, the coefficients of the filter sets used for subpixel positions “a” and “c” are derived independently of one another as h_{0} ^{a}, h_{1} ^{a}, . . . , h_{5} ^{a }and h_{0} ^{c}, h_{1} ^{c}, . . . , h_{5} ^{c}, respectively. For each subpixel position that has vertical movement, the subpixel symmetry is similar to NSAIF. That is, coefficients for subpixel positions “d” and “l,” coefficients for subpixel positions “e” and “m,” coefficients for subpixel positions “f” and “n,” and coefficients for subpixel positions “g” and “o” are vertically flipped versions of each other, respectively. For example, if the coefficients of the vertical filter for subpixel position “d” is h_{0} ^{d}, h_{1} ^{d}, . . . , h_{5} ^{d}, then the coefficients of the vertical filter for subpixel position “l” is h_{5} ^{d}, h_{4} ^{d}, . . . , h_{0} ^{d}. Thus, the subpixel symmetry used in SAIF only has vertical subpixel symmetry in one dimension. SAIF does not, however, have horizontal subpixel symmetry. The subpixel symmetry in SAIF may reduce the number of coefficients that need to be encoded and sent from 144 coefficients to 102 coefficients.
SAIF, like NSAIF, may also use coefficient symmetry to further reduce the number of coefficients that need to be encoded and sent by video encoder 50. The coefficient symmetry used in SAIF may be the same as in NSAIF, except that in SAIF there are only 1D (horizontal or vertical) 6tap filters. In other words, there are no 6by6 2D filters. In SAIF, for each subpixel position SP, SPε{b, h, i, j, k}, the filter coefficients may have symmetry as indicated in equation (14).
h _{0} ^{SP} =h _{5} ^{SP} , h _{1} ^{SP} =h _{4} ^{SP} , h _{2} ^{SP} =h _{3} ^{SP} (14)
Therefore, with both subpixel symmetry and coefficient symmetry, SAIF filters may have eleven sets of unique 1D 6tap filters, five of which are symmetric filters that each have three unique coefficients. The remaining six filters may each have six unique coefficients. In this case, video encoder 22 may quantize, predict, code, and send a total of fiftyone unique filter coefficients in the video bitstream.
In another AIF scheme, DAIF, prediction unit 32 uses directional filters for the 2D subpixel positions “e,” “f,” “g,” “i,” “j,” “k,” “m,” “n,” and “o.” The directional filters may comprise six tap filters, and coefficients may have symmetry so that only a subset of the coefficients needs to be transmitted. As described herein, filter support refers to the pixel positions from the reference frame that is used in interpolating the subpixel positions. MC unit 37 may compute subpixel positions “e” and “o” with a 6position (or 6tap) diagonal interpolation filter that has integer pixel positions A1, B2, C3, D4, E5 and F6 as filter support. Prediction unit 32 may compute subpixel positions “g” and “m” with a 6position diagonal interpolation filter that has integer pixel positions A6, B5, C4, D3, E2 and F1 as filter support. Prediction unit 32 may compute subpixel positions “f,” “i,” “j,” “k” and “n” with a 12position diagonal interpolation filter that has integer pixel positions A1, B2, C3, D4, E5, F6, A6, B5, C4, D3, E2 and F1 as filter support.
DAIF may use the same subpixel and coefficient symmetry as described above with respect to NSAIF. In terms of subpixel symmetry, there are five unique sets of filter coefficients for interpolating the fifteen subpixel positions with subpixel positions “a,” “c,” “d” and “l” sharing the same filter coefficients (e.g., mirrored, flipped and/or rotated versions), subpixel positions “e,” “g,” “m” and “o” sharing the same filter coefficients (e.g., mirrored, flipped and/or rotated versions), subpixel positions “b” and “h” sharing the same filter coefficients (“b” being a horizontal filter and “h” being a vertical filter), subpixel positions “f,” “i,” “k,” and “n” sharing the same filter coefficients (e.g., mirrored, flipped and/or rotated versions), and subpixel position “j” has its own set of filter coefficients.
In terms of coefficient symmetry, filter coefficients for the first subpixel group including “a,” “c,” “d” and “l,” uses a 1D 6position filter having six unique coefficients (i.e., the first group being nonsymmetric), filter coefficients for the second subpixel group “b” and “h” uses a 1D 6position filter having three unique coefficients, (i.e., b is symmetric), and a third set of filter coefficients for subpixel group “e,” “g,” “m” and “o” uses a directional 6position filter having six unique coefficients. A fourth set of filter coefficients for group “f,” “i,” “k,” and “n” use a 12tap filter having 6 unique coefficients and a fifth set of filter coefficients for group “j” having three unique coefficients. The total number of unique coefficients in the DAIF scheme is 6(a)+3(b)+6(e)+6(f)+3(j)=24 coefficients. These filter coefficients may be predicted, quantized, coded, and sent in the video bitstream.
MC unit 37 of prediction unit 32 may use one or more of the interpolation techniques described herein to enhance the accuracy of interpolation of subpixel positions and/or reduce the number of interpolation filter coefficients that need to be encoded and transmitted. Described in greater detail below is a useful twelvepixel filter support with a radial shape that may be used for interpolation, techniques that use coefficient symmetry and pixel symmetry to reduce the amount of data needed to be sent between an encoder and a decoder to configure the filter support for interpolation, techniques for removing some aspects of symmetry relative to conventional techniques, and techniques for filtering data at integer pixel locations in a manner that is similar to subpixel interpolation. Offset techniques are also described. In addition, techniques for selecting between interpolation filters, and techniques for predictively encoding filter coefficients are also described.
As can be appreciated from
The set of twelve or more integer pixel positions that surround the subpixel positions are shown with shading in
Entropy coding unit 46 may encode the sets of coefficient values and output the coefficient values of the interpolation filter as part of an encoded bitstream, which may then be transmitted to another device. Each of the plurality of sets of coefficient values defined with respect to each different subpixel location shown in
Thus, video encoder 22 and/or video decoder 28 may utilize interpolationlike filtering with respect to integer pixel positions in order to generate adjusted integer pixel values. Video encoder 22 or video decoder 28 may, for example, obtain blocks of pixels, wherein the blocks of pixels includes integer pixel values corresponding to integer pixel positions within the block of pixels. Video encoder 22 or video decoder 28 may filter the integer pixel values based on other integer pixel values within the blocks of pixels to generate adjusted integer pixel values, wherein the adjusted integer pixel values correspond to the integer pixel positions, and generating a prediction block based on the adjusted integer pixel values. The prediction block can then be used to encode or decode a video block, depending upon whether the process is performed during encoding or decoding.
If the technique is performed during an encoding process, video encoder 22 may generate a residual video block as a difference between a current video block being encoded and the prediction block. Video encoder 22 may also encode one set of filter coefficients per coded unit for integer filtering, or possibly encode two or more sets of filter coefficients per coded unit for integer filtering at different locations within the coded unit. In the example of
If the integerpel filtering is performed during a decoding process, the technique may comprise generating a reconstructed video block as a sum of a residual video block associated with a current video block being decoded and the prediction block. In this case, the decoding device (e.g., destination device 16) may receive one set of filter coefficients per coded unit for integer filtering, or may receive two or more sets of filter coefficients per coded unit for integer filtering at different locations within the coded unit. Each coded unit may have one or more sets of integerpel filtering coefficients, as well as sets of subpel filtering coefficients, e.g., eight sets that may be used to generate all fifteen sets for halfpel and quarterpel positions.
Again, interpolationlike filtering of integer pixel values may improve compression specifically during illumination changes, scene fadeins or fadeouts, may remove noise and facilitate image frame sharpening, and may help improve encoding of fine object movement between successive video frames particularly when symmetry is not imposed on filter coefficients. Furthermore, integerpel filtering is also found to be extremely useful, e.g., to improve compression, in video sequences that have focus changes.
Conventionally, interpolationlike filtering is not performed for integerpixel positions in AIF. Instead, the corresponding integerpixel position in the reference frame is simply used “as is.” However, in accordance with this disclosure, a filter may be used for integerpixel locations in a manner that is similar to interpolations. Interpolation filtering on integerpixel positions is technically a misnomer, as the pixel values of integerpixel positions already exist. Thus, the filtering of integerpixel positions is refereed to herein as interpolationlike filtering, and may be viewed as simply filtering the integerpixel values to generate new adjusted pixel values.
Unlike interpolation filtering for subpixel positions, where the goal is to obtain the nonexistent values in the reference frame, pixels at integer positions in the reference frame already exists. Yet, applying filtering on the integerpixel positions in the reference frame offers a number of benefits. For example, when integerpixel positions contain an offset, filtering of the integerpixel positions may allow better capture of illumination changes between frames. Additionally, filtering on the integerpixel positions may allow the integerpixel positions to capture special effects such as fadein and fadeout. Filtering on the integerpixel positions may also remove certain signal noises (such as camera noise) and/or performs sharpening, if needed. Furthermore, if no symmetry on the integerpixel filter is imposed, the integerpixel filter may help capture fine object movement, such as object movement that is not a multiple of quarterpixel. Finally, integer pixel filtering may also be useful in improving compression of video sequences when focus changes occur.
In some instances, interpolation filtering of integerpixel positions may be dependent upon motion vectors. In other words, MC unit 37 of prediction unit 32 may adaptively apply the interpolation filtering based on the motion vector identified by ME unit 35. For instance, interpolation filtering of integerpixel positions may be performed when the motion vector points to an integerpixel position. The interpolation filtering may result in an adjusted pixel value for the integerpixel position. When the motion vector points to a subpixel position, no filtering of integerpixel positions is performed with respect to the integer pixel values, but filtering may be used for interpolation of the subpixel values. In this manner, integerpixel filtering may be considered adaptive based on motion vectors.
For integerpixel positions, a 5by5 interpolation filter in the form of equation (15) may be derived using the same analytical process used to derive the subpixel position filters.
where p^{FP }is the filtered integer pixel value, P_{i,j }is the integer pixel value at position (i, j), h^{FP} _{i,j }is the coefficient for position (i, j), and o^{FP }is a filter offset. In some instances, the interpolation filter for integerpixel positions may not have a filter offset (i.e., o^{FP}=0). In other instances, the filter offset o^{FP }may use any of a number of techniques. Different offsets may be defined for each subpixel position, e.g., fifteen different offsets may be defined for each subpixel position shown in
In order to reduce the number of filter coefficients to be transmitted, coefficient symmetry as in equation (16) may be imposed on the integerpixel filter.
h _{i,j} ^{FP} =h _{i,4j} ^{FP} =h _{4i,j} ^{FP} =h _{4i,4j} ^{FP}, for i,j=0 . . . 4 (16)
Other forms of coefficient symmetry may also be used. Whether coefficient symmetry is imposed may be decided by the encoder and signaled to the decoder. If the symmetry as in (16) is used on the integerpixel filter, then 9 filter coefficients and 1 offset may need to be sent to the decoder.
Coding of the offsets may be done by first coding an integer offset and then coding each fractional offset. In this case, the integer offset and the fractional offsets are coded differently. The integer offset may be coded using signed exponentialGolomb code. Depending on the magnitude of the integer offset, the fractional offsets may be quantized differently than the integer offset. For example, coarser levels of quantization may be applied on the fractional offsets when an integer offset has a large magnitude. After quantization, the fractional offsets may be coded using fixed length codes. As an example, the following pseudocode may illustrate how a given fractional offset may be coded using different fixed length codes defined based on the integer offset value.
Let offsetI be the integer offset  
Let offsetF be the fractional offset  
Let offsetFbits be the number of bits used to quantize offsetF  
Let abs(a) be the absolute value of a  
if(abs(offsetI) = 0)  
offsetFbits = 5  
else if(abs(offsetI) < 3)  
offsetFbits = 4  
else if(abs(offsetI) < 7)  
offsetFbits = 2  
else  
offsetFbits = 0  
In this case, video encoder 50 may use coefficient prediction and quantization unit 41 that for each of the offset values assigns a first number of bits to an integer portion of a given offset value and assigns a second number of bits to a fractional portion of the given offset value, wherein the first and second numbers of bits are determined based on a magnitude of the integer portion. In this case, the quantized versions of the offset values may be forwarded from prediction unit 32 as prediction syntax elements after quantization of the offset values is performed consistent with the pseudocode above. In any case, video encoder 50 may encode the first number of bits differently than the second number of bits.
In this scenario, pixel symmetry may be imposed such that the filter coefficients for pixel location “a” are symmetry with respect to the filter coefficients for pixel location “c.” Similarly, pixel symmetry may be imposed such that the filter coefficients for pixel location “d” are symmetric with respect to the filter coefficients for pixel location “l.” Pixel symmetry may be imposed such that the filter coefficients for pixel location “e” are symmetric with respect to the filter coefficients for pixel locations “g,” “m” and “o,” and pixel symmetry may be imposed such that the filter coefficients for pixel location “i” are symmetric with respect to the filter coefficients for pixel locations “k,” and pixel symmetry may be imposed such that the filter coefficients for pixel location “f” are symmetric with respect to the filter coefficients for pixel locations “n.” Therefore, only eight sets of coefficient values may need to be communicated as part of a video bitstream in order to enable a decoder to generate complete sets of coefficient values for all fifteen pixel locations. Furthermore, for any given set of coefficients, coefficient symmetry may allow only a subset of the coefficient values to be sent, and the decoder can generate the complete set of coefficient values for a given pixel location based on the subset and coefficient symmetry.
One aspect of this disclose is to actually remove any pixel symmetry between subpixel positions “a” and “c” relative to subpixel positions “d” and “l.” In addition, pixel symmetry may be avoided for subpixel position “f” relative to subpixel position “i.” In addition, pixel symmetry may be avoided for subpixel position “k” relative to subpixel position “n.” In such cases, diagonal correlation may be low, making it inefficient or ineffective to impose diagonal symmetry. In this manner, the eight sets of coefficients used to generate all fifteen sets of coefficients may be a larger set than some conventional techniques insofar as symmetry is avoided in the diagonal dimension for some of the pixel locations. In some cases, diagonal pixel symmetry and diagonal coefficient symmetry may be eliminated or avoided to allow for variance in the vertical dimensions.
Referring again to
The fifteen sets correspond to every halfpel and quarterpel subpixel location shown in
At least some of the eight sets of filter coefficients may have coefficient symmetry such that a given set with coefficient symmetry comprises a subset of the filter coefficients associated with a filter support for a given subpixel location associated with the given set. Thus, video decoder 28 may generate the fifteen sets of filter coefficients based at least in part on pixel symmetry between two or more different subpixel locations. Moreover, video decoder 28 may generate the fifteen sets of filter coefficients based at least in part on pixel symmetry between two or more different subpixel locations, and based at least in part on coefficient symmetry defined for a given set such that the given set with coefficient symmetry comprises a subset of the filter coefficients associated with a filter support for a given subpixel location associated with the given set.
Again, the fifteen sets may correspond to every halfpel and quarterpel subpixel location. Pixel symmetry may exist between different ones of the fifteen sets of filter coefficients in a vertical dimension and in a horizontal dimension, but pixel symmetry does not exist in a diagonal dimension for at least some of the fifteen sets of filter coefficients. As explained in greater detail below, two of the eight sets may define three of the fifteen sets for a horizontal filter support, two of the eight sets may define three of the fifteen sets for a vertical filter support, and four of the eight sets may define nine of the fifteen sets for a twodimensional filter support. The twodimensional support may comprise a useful 12 position filter support surrounding nine subpixel positions in a radial shape.
Specifically, the interpolated predictive data may comprise fifteen halfpel and quarterpel values, e.g., shown in
Horizontal symmetry and lack of diagonal symmetry can be seen with respect to subpixel positions “a” and “c” or with respect to subpixel positions “i” and “k.” In this case, subinteger pixel positions “a” and “c” are symmetric along a horizontal X axis, but symmetry does not exist along a diagonal axis for these pixels such that subpixel locations “d” and “l” are not symmetric with respect to subpixel positions “a” and “c” respectively. Similarly, subpixel positions “i” and “k” are symmetric along a horizontal X axis, but symmetry does not exist along diagonal axis for these pixels such that subpixel locations “f” and “n” are not symmetric with respect to subpixel positions “i” and “k” respectively.
The pixel symmetry means that a first set of coefficients with respect to the first subpixel position are symmetric relative to a second set of coefficients with respect to the second subpixel position. For example, pixel location “a” has pixel symmetry with respect to pixel location “c,” and pixel location “d” has pixel symmetry with respect to pixel location “l.” Pixel location “f” has pixel symmetry with respect to pixel location “n,” and pixel location “i” has pixel symmetry with respect to pixel location “k.” Pixel location “e” has pixel symmetry with respect to pixel locations “g,” “m,” and “o.” The shading shown in
Video encoder 22 can generate encoded video data based on interpolated predictive data, wherein the interpolated predictive data comprises fifteen halfpel and quarterpel values, wherein the interpolation filter defines a twodimensional array of twelve filter support positions that are applied to define nine of the fifteen halfpel and quarterpel values. Generating the encoded video data based on interpolated predictive data may further comprise applying linear interpolation filters that define onedimensional arrays of filter support positions for six of the fifteen halfpel and quarterpel values.
Thus, all six coefficients are needed in order to define the set of coefficients for filter support with respect to subpixel position “d.” As noted above, however, pixel symmetry means that these same coefficients with respect to subpixel position “d” can also be used to derive the filter support for subpixel position “l” (see
The prediction syntax elements, e.g., the coefficients, are forwarded to prediction unit 55. If prediction is used to code the coefficients relative to coefficients of a fixed filter, or relative to one another, coefficient prediction and inverse quantization unit 53 can decode the syntax elements to define the actual coefficients. Also, if quantization is applied to any of the prediction syntax, coefficient prediction and inverse quantization unit 53 can also remove such quantization. Filer coefficients, for example, may be predictively coded and quantized according to this disclosure, and in this case, coefficient prediction and inverse quantization unit 53 can be used by video decoder 60 to predictively decode and dequantize such coefficients.
Prediction unit 55 may generate prediction data based on the prediction syntax elements and one or more previously decoded blocks that are stored in memory 62, in much the same way as described in detail above with respect to prediction unit 32 of video encoder 50. In particular, prediction unit 55 may perform one or more of the interpolation filtering techniques of this disclosure during motion compensation to generate a prediction block with a particular precision, such as quarterpixel precision. As such, one or more of the techniques of this disclosure may be used by video decoder 60 in generating a prediction block. Prediction unit 55 may include a motion compensation unit that comprises filters used for interpolation and interpolationlike filtering techniques of this disclosure. The motion compensation component is not shown in
Inverse quantization unit 56 inverse quantizes, i.e., dequantizes, the quantized coefficients. The inverse quantization process may be a process defined for H.264 decoding. Inverse transform unit 58 applies an inverse transform, e.g., an inverse DCT or conceptually similar inverse transform process, to the transform coefficients in order to produce residual blocks in the pixel domain. Summer 64 sums the residual block with the corresponding prediction block generated by prediction unit 55 to form a reconstructed version of the original block encoded by video encoder 50. If desired, a deblocking filter may also be applied to filter the decoded blocks in order to remove blockiness artifacts. The decoded video blocks are then stored in reference frame store 62, which provides reference blocks for subsequent motion compensation and also produces decoded video to drive display device (such as device 28 of
NSAIF techniques may compute pixel values for the 2D subpixel positions using an interpolation filter having 36postion filter support (i.e., a 2D 6×6 filter support). SAIF uses separable integer pixel positions as filter support for interpolation filtering first in a horizontal direction than in a vertical dimension. DAIF uses a less complex interpolation filter for computing the 2D subpixel positions than NSAIF. In DAIF the 2D subpixel positions are computed using an interpolation filter having either a 6postion filter support or a 12position diagonal filter support.
One drawback of using the diagonal filter support as in DAIF is that the integerpixel positions used in filtering are far away from the current position to be interpolated. As distance between the current position to be interpolated and the positions of the pixels used as filter support increases, spatial pixel correlation decreases. As such, the diagonal filter support used in DAIF is less suitable for forming an accurate prediction.
In order to provide better prediction using pixels with higher correlation with the position to be interpolated (that is, pixel positions that are closer by or a shorter distance from the position to be interpolated), while maintaining the same low complexity offered by DAIF, the 12position filter support shown in
Filter support in accordance with the techniques of this disclosure may, however, include additional integerpixel locations. For example, the filter support may include integerpixel locations that are located within three integerpixel positions of the subpixel position to be interpolated that form a radial shape.
Even quarterpixel positions or finer resolution may be interpolated based on integer pixel positions located within at least two integerpixel positions. In this manner, the filter support techniques of this disclosure provide better interpolation prediction using integerpixels with higher correlation with the position to be interpolated, while maintaining the same low complexity offered by DAIF.
As described above, subpixel symmetry may enforce the same set of filter coefficients (and the mirrored, inverted and/or rotated versions of the set) for a group of one or more subpixel positions. Using an analytical process, such as that in equation (5) may be used to derive interpolation filter coefficients for all subpixel positions, e.g., fifteen subpixel positions in the case of quarterpixel precision. Analysis of correlation among the fifteen sets of interpolation filter coefficients reveals a relatively strong correlation between filter coefficients of subpixel positions (both 1D and 2D subpixel positions) in the same dimension (e.g., vertical or horizontal dimension). To the contrary, correlation between filter coefficients of the subpixel positions in different diagonal directions may be noticeably weaker.
For example, there may be a high correlation between filter coefficients for subpixel positions “a” and “c,” which are both in the horizontal dimension. As another example, there may be a high correlation between filter coefficients for subpixel positions “d” and “l,” which are both in the vertical dimension. However, there may be little, if any, correlation between filter coefficients for subpixel positions “a,” which is in the horizontal dimension, and subpixel position “d,” which is in the vertical dimension. Based on these observations of the correlations, subpixel symmetry may be designed such that subpixel symmetry for filter coefficients of subpixel positions in diagonal dimensions is not imposed. This leads to eight sets of coefficients, as described herein, which may be more sets than needed for some conventional processes. By eliminating diagonal pixel symmetry for some pixel locations, as described herein, video encoding and compression may be improved.
Again,
In NSAIF and DAIF subpixel symmetry exists between subpixel positions in the same dimension, e.g., two subpixel positions in the vertical dimension or two subpixel positions in the horizontal dimension. In some conventional NSAIF and DAIF, symmetry also exists diagonally between subpixel positions in different dimensions. For example, subpixel symmetry exists between subpixel position “a” in the vertical dimension and subpixel position “d” in the horizontal direction in some conventional NSAIF and DAIF. With subpixel symmetry in horizontal, vertical and diagonal dimensions, as few as five unique sets of interpolation filter coefficients may be used for quarterpixel precision.
For some types of SAIF, on the other hand, subpixel symmetry exists between subpixel positions in the vertical direction (or dimension), but not subpixel positions in the horizontal dimension. In other words, subpixel symmetry does not always exist for subpixel positions in the same dimension. Subpixel symmetry also does not exist in diagonal dimensions. The symmetry of some SAIF schemes, thus, require more sets of interpolation coefficients. In particular, for quarterpixel precision, some SAIF schemes require eleven unique sets of interpolation coefficients.
The subpixel symmetry scheme described in this disclosure, and illustrated in
By not imposing subpixel symmetry for subpixel positions diagonally, the weak correlation between the interpolation filter coefficients and the subpixel positions in different dimensions is not integrated into the interpolation. Although more sets of interpolation coefficients may be required (e.g., eight instead of five), the resulting interpolated prediction data may be more accurate. Diagonal coefficient symmetry may also be avoided or eliminated, in much the same way.
Prediction unit 32 may also impose coefficient symmetry as described above. In particular, coefficient symmetry is imposed for filter coefficients in one dimension (e.g., the horizontal or vertical direction), but not for both dimensions in a diagonal manner. For example, the filter coefficients for subpixel position “e” are not forced to be diagonally symmetric as in the NSAIF scheme described above and represented in equation (11). The coefficient symmetry is summarized below in equations (17)(21).
h _{0} ^{b} =h _{5} ^{b} , h _{1} ^{b} =h _{4} ^{b} , h _{2} ^{b} =h _{3} ^{b} (17)
h _{0} ^{h} =h _{5} ^{h} , h _{1} ^{h} =h _{4} ^{h} , h _{2} ^{h} =h _{3} ^{h} (18)
h _{i,j} ^{f} =h _{i,3j} ^{f}, for i,j=0 . . . 3 (19)
h _{i,j} ^{i} =h _{3i,j} ^{i}, for i,j=0 . . . 3 (20)
h _{i,j} ^{i}=h_{i,3j} ^{j}=h_{3i,j} ^{j} =h _{3i,3j} ^{j}, for i,j=0 . . . 3 (21)
Note that for subpixel positions “f,” “i,” and “j,” some filter coefficients, i.e., h_{0,0} ^{SP}=h_{0,3} ^{SP}=h_{3,0} ^{SP}=h_{3,3} ^{SP }are equal to zero in instances in which the 12position filter described in detail with respect to
MC unit 37 then generates a prediction block based on the computed subpixel values (173). In particular, MC unit 37 may generate and output an interpolated prediction block comprising interpolated subpixel values. Adder 48 can then encode a current video block based on the interpolated prediction block (174), e.g., by subtracting the interpolated prediction block from the video block being encoded to generate a residual block. The residual block may then be transformed and quantized by transform unit 38 and quantization unit 40, respectively. Following entropy coding by entropy coding unit 46, video encoder 50 can output an encoded video bitstream and filter information (175). The filter information, as described herein, may comprise eight sets of coefficients used for generating all fifteen sets of coefficients for fifteen subpel positions. The filter information may be output once per coded unit, or possibly several times per coded unit if different areas of a coded unit use different types of subpel interpolation.
As shown in
Next, MC unit 37 of prediction unit 32 generates second interpolated predictive data for video encoding of the video data based on a second interpolation filter (232). In one example, the second interpolation filter may comprise an adaptive interpolation filter. In this case, consistent with adaptive interpolation, MC unit 37 may define the filter coefficients to be used. In particular, MC unit 37 may execute an adaptive interpolation process in which MC unit 37 starts with preset filter coefficients, generates preliminary predictive data, and then adjusts the filter coefficients in an adaptive process so that such filter coefficients define more desirable predictive data.
Once the MC unit 37 of prediction unit 32 has generated both the first and the second interpolated predictive data, MC unit 37 can select between the first interpolated predictive data and the second interpolated predictive data based on a ratedistortion analysis (233). In this way, MC unit 37 not only considers which set of predictive data will yield the best results (i.e., the least distortion), but MC unit 37 also factors in the different amounts of data (i.e., the different bit rates) that will be required for the first interpolation filter relative to the second interpolation filter. Notably, if the fixed interpolation filter is used (e.g., as the first interpolation filter), video encoder 50 will not need to encode filter coefficients in the bitstream, whereas if the adaptive interpolation filter is used (e.g., as the second interpolation filter) video encoder 50 will need to encode filter coefficients. Accordingly, the ratedistortion analysis can determine which set of predictive data will yield the best results (i.e., the least rate distortion cost), by accounting for the fact that the use of the fixed interpolation filter does not require additional bits in order to convey filter coefficients to the decoder.
More specifically, selecting between the first interpolated predictive data and the second interpolated predictive data based on a ratedistortion analysis (233) may comprise calculating a first ratedistortion cost associated with the video data if the video data is encoded via the first interpolated predictive data, calculating a second ratedistortion cost associated with the video data if the video data is encoded via the second interpolated predictive data, and selecting between the first interpolated predictive data and the second interpolated predictive data based on the first and second ratedistortion costs.
The rate distortion cost associated with the video data if the video data is encoded via the first interpolated predictive data may comprise a first difference metric, e.g., mean squared error (MSE) of pixel values or sum of absolute difference (SAD) of pixel values or sum of squared difference (SSD) of pixel values indicative of differences between the video data and the first interpolated predictive data, plus a first value that quantifies cost associated with encoding of filter coefficients. In this case, if the first interpolation filter is a fixed interpolation filter, the first value that quantifies cost may be defined as zero. Similarly, the second ratedistortion cost may comprise a second difference metric indicative of differences between the video data and the second interpolated predictive data (MSE, SAD or SSD), plus a second value that quantifies cost associated with encoding of filter coefficients. In the case where the second interpolation filter is an adaptive interpolation filter, the second value that quantifies cost associated with encoding of filter coefficients may comprise the number of bits (r) needed to encode adaptive interpolation filter coefficients, or possibly this number (r) multiplied by a Lagrangian multiplier (λ).
After selecting between the first interpolated predictive data and the second interpolated predictive data based on a ratedistortion analysis (233), MC unit 37 of prediction unit 32 encodes the video data based on the selection (234), and encodes syntax to indicate the selection (235). The syntax may comprise a onebit flag or a multibit flag that defines whether the first interpolation filter or the second interpolation filter should be used by a decoder. The process of
Once video encoder 50 (
The techniques of this disclosure may be applied in many scenarios, including scenarios when there are more than two sets of filters being selected by the encoder. In other words, additional interpolation filters may also be applied to generate additional interpolated predictive data, which may also be considered in the rate distortion analysis. In other words, the method is not limited to generating only first and second interpolated predictive data based on two interpolation filters, but could be applied to generate any plurality of interpolated predictive data based on any number of interpolation filters. Importantly, a ratedistortion analysis interpolated predictive data is used to identify which filter to select. In one example, a method executed by video encoder 50 may comprise generating a plurality of different versions of predictive data for encoding of video data based on a plurality of different interpolation filters, selecting among the plurality of different versions of predictive data based on a rate distortion analysis, encoding the video data based on the selection, and encoding syntax to indicate the selection.
In addition to selecting the interpolation filter based on a ratedistortion analysis, and generating syntax to indicate the selection, MC unit 37 of prediction unit 32 may also conduct a similar ratedistortion analysis with respect to integer pixel filtering, which is explained in more detail above. In particular, MC unit 37 of prediction unit 32 may generate two or more different sets of predictive data for integer pixel locations, e.g., one with integer pixel filtering and one without integer pixel filtering, and may conduct a ratedistortion analysis with respect to these two different sets in order to determine whether integer pixel filtering is desired. Accordingly, MC unit 37 of prediction unit 32 may generate another syntax element based on a ratedistortion analysis associated with two different sets of integer predictive data to indicate whether integer pixel filtering should be applied for the coded unit, wherein a first set of the integer predictive data is nonfiltered and a second set of the integer predictive data is filtered. In this way, the decision whether to conduct integer pixel filtering may be based on not only the quality of video coding, but also the possible bit overhead associated with integer pixel filtering, particularly when integer pixel filtering would involve the encoding and sending of integer pixel filter coefficients used to perform such integer pixel filtering. Integer filtering may also consider N integer filters (e.g., where N is any positive and plural integer. Consistent with the use of N integer filters, the example above would correspond to the case where N is two and one of the filters does not apply any filtering.
Whenever interpolation filter coefficients are actually encoded and sent from source device 12 to destination device 16, this disclosure also contemplates techniques for coding such interpolation filter coefficients. The described encoding for filter coefficients can improve data compression. In particular, this disclosure contemplates prediction techniques for filter coefficients, e.g., relative to fixed filter coefficients. In addition, this disclosure contemplates prediction techniques for a second set of filter coefficients relative to first set of filter coefficients. In these ways, imperfect symmetry between different filter coefficients may be exploited to allow for data compression. In addition to the use of such prediction techniques for interpolation filter coefficients, this disclosure also provides for useful quantization techniques and entropy coding of interpolation filter coefficients based on prefix and suffix coding. The following discussion provides more details on these aspects of this disclosure.
As shown in
In order to encode the filter coefficients so that such filter coefficients may be transmitted as part of a bitstream, coefficient prediction and quantization unit 41 may generate residual values associated with the set of filter coefficients based on predictive coding of the set of filter coefficients relative to filter coefficients associated with a fixed interpolation filter (242). In particular, coefficient prediction and quantization unit 41 may subtract the filter coefficients determined in the adaptive interpolation filtering process from corresponding filter coefficients associated with a fixed filter, such as an ITUT H.264 interpolation filter, or an ITUT H.264 interpolation filter without intermediate rounding of halfpixel values. By encoding and transmitting residual values, rather than transmitting the actual filter coefficients, the amount of data communicated in the bitstream can be reduced. In this case, the decoder may be programmed to know the manner in which the filter coefficients are encoded.
Video encoder 50 may invoke coefficient prediction and quantization unit 41 to both predict and quantize the residual coefficients, and entropy coding unit 46 may entropy code the quantized residuals (243). Video encoder 50 can then output the residual values as part of the encoded bitstream (244). The quantization of residual values associated with the filter coefficients may involve quantizing the residual values, wherein at least some of the residual values associated with different filter coefficients are assigned different levels of quantization. In this way, coefficient prediction and quantization unit 41 may assign more quantization to larger residual coefficients, and may assign less quantization to finer residual coefficients in order to achieve a desirable balance of quantization and accuracy. When more quantization is used, more data is eliminated but more compression may be achieved. Entropy coding unit 46 may entropy code the residual values by assigning a prefix code, assigning a suffix code and assigning a sign value for the residual values following quantization of the residual values. Video encoder 50 can then output the residual values as part of the encoded bitstream (244).
In some cases, the set of filter coefficients comprises a first set of filter coefficients that define only a portion of an entire set of filter coefficients associated with an adaptive interpolation filter, wherein a second set of filter coefficients can be determined by a decoder based on the first set of filter coefficients and coefficient symmetry. For example, filter coefficients X_{1 }and X_{2 }may be predictively encoded based on O_{1 }and O_{2 }respectively. In this case, however, X_{3 }and X_{4 }may be symmetric with X_{1 }and X_{2 }and the decoder may be programmed to know that such symmetry is imposed. Accordingly, through the use of symmetry, coefficients X_{3 }and X_{4 }in this simple example, may be eliminated from the bitstream, and may be calculated at the decoder based on known coefficient symmetry once coefficients X_{1 }and X_{2 }are predictively decoded.
In order to encode the filter coefficients so that such filter coefficients may be transmitted as part of a bitstream, coefficient prediction and quantization unit 41 may generate a first set of residual values associated with a first set of the filter coefficients based on predictive coding relative to filter coefficients associated with a fixed interpolation filter (252). In particular, coefficient prediction and quantization unit 41 may subtract the first set of filter coefficients from corresponding filter coefficients associated with a fixed filter, such as an ITUT H.264 interpolation filter, or an ITUT H.264 interpolation filter without intermediate rounding of halfpixel values. Next, coefficient prediction and quantization unit 41 may generate a second set of residual values associated with a second set of the filter coefficients based on predictive coding relative to the first set of filter coefficients (253). In particular, coefficient prediction and quantization unit 41 may subtract the second set of filter coefficients from mirrored or rotated values of the first set of coefficients. Thus, the first set of coefficients are predictively coded based on the coefficients of a fixed filter, and the second set of coefficients are predictively coded based on the first set of coefficients. By generating residual values rather than using the actual filter coefficients, the amount of data communicated in the bitstream can be reduced. Furthermore, by using the fixed filter to predict the first set of coefficients and then using the first set of coefficients to predict the second set of coefficients, further data compression may be achieved relative to prediction that relies only on the fixed filter.
As with the process of
Next, a second set of filter coefficients Z_{1 }and Z_{2 }may be predictively coded based on the first set of coefficients X_{1 }and X_{2}, e.g., specifically from coefficients {circumflex over (X)}_{1 }and {circumflex over (X)}_{2 }which are defined based on coefficients X_{1 }and X_{2}. In particular, a third residual may be formed as the difference between Z_{1 }and {circumflex over (X)}_{1}, and a fourth residual may be formed as the difference between Z_{2 }and {circumflex over (X)}_{2}. {circumflex over (X)}_{1 }and {circumflex over (X)}_{2 }may be more similar to Z_{1 }and Z_{2 }than O_{1 }and O_{2 }and therefore, by using {circumflex over (X)}_{1 }and {circumflex over (X)}_{2 }to predictively encode Z_{1 }and Z_{2 }further data compression may be promoted.
In particular, prediction unit 56 may generate the entire set of filter coefficients based on the residual values and filter coefficients associated with a fixed interpolation filter, e.g., as conceptually illustrated in
Again, the set of predictively decoded filter coefficients may comprise a first set of filter coefficients that define only a portion of an entire set of filter coefficients associated with an adaptive interpolation filter. In this case, coefficient prediction and inverse quantization unit 53 of video decoder 60 may generate a second set of filter coefficients based on the first set of filter coefficients and coefficient symmetry, and apply the first and second sets of filter coefficients to interpolate the predictive data.
In another case, the set of predictively decoded filter coefficients may comprises a first set of filter coefficients that define only a portion of an entire set of filter coefficients associated with an adaptive interpolation filter. In this case, video decoder 60 may receive additional residual values associated with the entire set of filter coefficients. Coefficient prediction and inverse quantization unit 53 may generate a second set of filter coefficients using predictive decoding based on additional residual values and the first set of filter coefficients, and prediction unit 55 may apply the first and second sets of filter coefficients to interpolate the predictive data.
In some cases, fixed interpolation filters based on H.264/AVC filters may be used to predict the 1D filters (which may include filters for the subpixel positions a, b, d, h shown in

 1. Set prediction to zero (no prediction), p(h_{i,j} ^{SP})=0, SPε{e,f,i,j}
 2. Use a fixed filter prediction, such as the average filter gathered over a training set, i.e., p(h_{i,j} ^{SP})=
h _{i,j} ^{SP}, SPε{e,f,i,j}, whereh _{i,j} ^{SP }is the (i,j)th filter coefficient in the average filter for subpixel position SP.  3. Exploit the possible symmetry in the coefficients and use the already coded coefficients to predict the remaining coefficients.
For the 1D filters, any of these three prediction methods may also be applied.
For prediction,
In any case, after prediction of the coefficients (e.g., by coefficient prediction and quantization unit 41 of prediction unit 32), prediction errors are quantized (e.g., by coefficient prediction and quantization unit 41). As outlined above, socalled “uneven quantization” may be used. In this case, the quantization precision applied by coefficient prediction and quantization unit 41 may depend on the coefficient location. It has been found that for coefficients with smaller magnitude (which are typically coefficients farther away from the center of the filter), better precision may be desirable. In contrast, coefficients with larger magnitude (which are typically coefficients closer to the center of the filter), less precision is more desirable.
The following matrices, Q^{1D}, Q^{2D}, Q^{FP}, may be used by coefficient prediction and quantization unit 41 to specify the quantization precision for coefficients in the 1D filters, the 2D filters, and the integerpixel filter, respectively. Note that the numbers of bits given in the matrices may include 1 bit to encode the sign of the respective coefficients.
Coefficient prediction and quantization unit 41 may code the quantized coefficient prediction errors i.e., the coefficient residuals, based on a simple prefix coding scheme. First, the dynamic range of the magnitude of the prediction error is divided into N bins, for example N=6. If the dynamic range of the magnitude is [0, . . . ,2^{q1}−1], where q is the quantization precision for the given coefficient position, (such as specified in the matrices above) then each bin n, n=0, . . . N−1, may span the following ranges:
In this case, the bin b, b=0, . . . , N−1, to which the input magnitude m belongs, may be coded using a unary prefix code (which takes b+1 bits). Then, the remainder of the magnitude, m−b_{start}, may be coded using fixed length suffix code of (q−N+b−1) bits. Finally, the sign of the prediction error is coded using 1 bit.
For example, for the center coefficient in the 1D filter, 9 bits precision may be used by coefficient prediction and quantization unit 41 to quantize the prediction error, i.e., q=9, of which 8 bits may be used to quantize the error magnitude. If the error magnitude is 97, then it falls within the 4th bin, i.e., b=4. Therefore, the unary prefix code “11110” may be encoded to indicate the 4th bin, and the remainder of m−b_{start}=97−64=33 may be encoded using a fixed length suffix code of (q−N+b−1)=(9−6+4−1)=6 bits. Finally, a bit “0” can be encoded to indicate positive sign.
The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, and integrated circuit (IC) or a set of ICs (i.e., a chip set). Any components, modules or units have been described provided to emphasize functional aspects and does not necessarily require realization by different hardware units.
Accordingly, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, any features described as modules, units or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computerreadable medium comprising instructions that, when executed in a processor, performs one or more of the methods described above. The computerreadable medium may comprise a computerreadable storage medium and may form part of a computer program product, which may include packaging materials. The computerreadable storage medium may comprise random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), readonly memory (ROM), nonvolatile random access memory (NVRAM), electrically erasable programmable readonly memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computerreadable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer.
The code may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured for encoding and decoding, or incorporated in a combined video codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.
Various aspects of the disclosure have been described. These and other aspects are within the scope of the following claims.
Claims (34)
Priority Applications (5)
Application Number  Priority Date  Filing Date  Title 

US4402308 true  20080410  20080410  
US4402008 true  20080410  20080410  
US4424008 true  20080411  20080411  
US5737308 true  20080530  20080530  
US12420333 US8831086B2 (en)  20080410  20090408  Prediction techniques for interpolation in video coding 
Applications Claiming Priority (10)
Application Number  Priority Date  Filing Date  Title 

US12420333 US8831086B2 (en)  20080410  20090408  Prediction techniques for interpolation in video coding 
KR20107025196A KR101202632B1 (en)  20080410  20090410  Prediction techniques for interpolation in video coding 
BRPI0910635A2 BRPI0910635A2 (en)  20080410  20090410  interpolation prediction techniques in video encoding 
RU2010145536A RU2479941C2 (en)  20080410  20090410  Prediction techniques for interpolation in video coding 
EP20090731116 EP2277316A1 (en)  20080410  20090410  Prediction techniques for interpolation in video coding 
CA 2719215 CA2719215A1 (en)  20080410  20090410  Prediction techniques for interpolation in video coding 
JP2011504209A JP2011517237A (en)  20080410  20090410  Prediction techniques for interpolation in video coding 
PCT/US2009/040250 WO2009126911A1 (en)  20080410  20090410  Prediction techniques for interpolation in video coding 
TW98112116A TW201004362A (en)  20080410  20090410  Prediction techniques for interpolation in video coding 
CN 200980112682 CN101990759B (en)  20080410  20090410  Prediction techniques for interpolation in video coding 
Publications (2)
Publication Number  Publication Date 

US20090257668A1 true US20090257668A1 (en)  20091015 
US8831086B2 true US8831086B2 (en)  20140909 
Family
ID=40793002
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

US12420333 Expired  Fee Related US8831086B2 (en)  20080410  20090408  Prediction techniques for interpolation in video coding 
Country Status (7)
Country  Link 

US (1)  US8831086B2 (en) 
EP (1)  EP2277316A1 (en) 
JP (1)  JP2011517237A (en) 
KR (1)  KR101202632B1 (en) 
CN (1)  CN101990759B (en) 
CA (1)  CA2719215A1 (en) 
WO (1)  WO2009126911A1 (en) 
Cited By (3)
Publication number  Priority date  Publication date  Assignee  Title 

US20120051438A1 (en) *  20100901  20120301  Qualcomm Incorporated  Filter description signaling for multifilter adaptive filtering 
US20130216150A1 (en) *  20101208  20130822  Sony Corporation  Image processing device, image processing method, and program 
US20150071345A1 (en) *  20130909  20150312  Apple Inc.  Chroma quantization in video coding 
Families Citing this family (38)
Publication number  Priority date  Publication date  Assignee  Title 

US8926435B2 (en)  20081215  20150106  Sony Computer Entertainment America Llc  Dualmode program execution 
US8147339B1 (en)  20071215  20120403  Gaikai Inc.  Systems and methods of serving game video 
US8613673B2 (en)  20081215  20131224  Sony Computer Entertainment America Llc  Intelligent game loading 
US9967590B2 (en)  20080410  20180508  Qualcomm Incorporated  Ratedistortion defined interpolation for video coding based on fixed filter or adaptive filter 
US8218633B2 (en) *  20080618  20120710  Kiu Sha Management Limited Liability Company  Bidirectionally decodable WynerZiv video coding 
US10123050B2 (en) *  20080711  20181106  Qualcomm Incorporated  Filtering video data using a plurality of filters 
JP5200788B2 (en) *  20080909  20130605  富士通株式会社  A video signal processing apparatus, a video signal processing method and a video signal processing program 
JP4735696B2 (en) *  20080926  20110727  ソニー株式会社  Image processing apparatus, image processing method, and program 
US20110243437A1 (en) *  20081024  20111006  Solomon Systech Limited.  System and method for image compression 
US8774541B2 (en) *  20081105  20140708  Sony Corporation  Intra prediction with adaptive interpolation filtering for image compression 
KR20170129296A (en)  20100913  20171124  소니 인터랙티브 엔터테인먼트 아메리카 엘엘씨  A method and system of providing a computer game at a computer game system including a video server and a game server 
US9143803B2 (en) *  20090115  20150922  Qualcomm Incorporated  Filter prediction based on activity metrics in video coding 
US8888592B1 (en)  20090601  20141118  Sony Computer Entertainment America Llc  Voice overlay 
US8968087B1 (en)  20090601  20150303  Sony Computer Entertainment America Llc  Video game overlay 
US9723319B1 (en)  20090601  20170801  Sony Interactive Entertainment America Llc  Differentiation for achieving buffered decoding and bufferless decoding 
JP5597968B2 (en) *  20090701  20141001  ソニー株式会社  An image processing apparatus and method, program, and recording medium 
WO2011086672A1 (en) *  20100113  20110721  株式会社 東芝  Moving image coding device and decoding device 
US9237355B2 (en) *  20100219  20160112  Qualcomm Incorporated  Adaptive motion resolution for video coding 
WO2011111341A1 (en)  20100309  20110915  パナソニック株式会社  Dynamic image decoding device, dynamic image encoding device, dynamic image decoding circuit, and dynamic image decoding method 
KR101847072B1 (en) *  20100405  20180409  삼성전자주식회사  Method and apparatus for video encoding, and method and apparatus for video decoding 
KR101682147B1 (en) *  20100405  20161205  삼성전자주식회사  Method and apparatus for interpolation based on transform and inverse transform 
US9094658B2 (en)  20100510  20150728  Mediatek Inc.  Method and apparatus of adaptive loop filtering 
KR20180118254A (en)  20100525  20181030  엘지전자 주식회사  New planar prediction mode 
US9154807B2 (en) *  20100625  20151006  Qualcomm Incorporated  Inclusion of switched interpolation filter coefficients in a compressed bitstream 
US9317896B2 (en) *  20100709  20160419  Samsung Electronics Co., Ltd.  Image interpolation method and apparatus 
US8676591B1 (en)  20100802  20140318  Sony Computer Entertainment America Llc  Audio deceleration 
JP2012060208A (en) *  20100906  20120322  Nippon Telegr & Teleph Corp <Ntt>  Moving image encoding method, moving image encoding apparatus, moving image decoding method, moving image decoding apparatus, and program thereof 
US9878240B2 (en)  20100913  20180130  Sony Interactive Entertainment America Llc  Addon management methods 
JP2012085211A (en) *  20101014  20120426  Sony Corp  Image processing device and method, and program 
US20120134425A1 (en) *  20101129  20120531  Faouzi Kossentini  Method and System for Adaptive Interpolation in Digital Video Coding 
US10045046B2 (en) *  20101210  20180807  Qualcomm Incorporated  Adaptive support for interpolating values of subpixels for video coding 
US9172972B2 (en)  20110105  20151027  Qualcomm Incorporated  Low complexity interpolation filtering with adaptive tap size 
US8964852B2 (en)  20110223  20150224  Qualcomm Incorporated  Multimetric filtering 
JP5552092B2 (en) *  20110613  20140716  日本電信電話株式会社  Moving picture coding apparatus, moving picture coding method and a moving picture encoding program 
KR20130003706A (en) *  20110630  20130109  엘지전자 주식회사  Interpolation method and prediction method thereof 
US8982948B2 (en) *  20111221  20150317  Sony Corporation  Video system with quantization matrix coding mechanism and method of operation thereof 
US20160080771A1 (en) *  20140917  20160317  Mediatek Inc.  Motion compensation apparatus having at least one processing circuit shared by normal mode and resized reference frame mode and related motion compensation method 
US20180098072A1 (en) *  20160930  20180405  Qualcomm Incorporated  Memory and bandwidth reduction of stored data in image/video coding 
Citations (41)
Publication number  Priority date  Publication date  Assignee  Title 

JPH034685A (en)  19890601  19910110  Canon Inc  Encoding method for picture data 
JPH04230584A (en)  19910107  19920819  Toshiba Corp  Encoder and decoder 
EP0833520A2 (en)  19960930  19980401  Hyundai Electronics Industries Co., Ltd.  Video predictive coding apparatus and method 
JPH11145839A (en)  19971107  19990528  Nec Corp  Method for determining quantizing coefficient of digital filter and computerreadable record medium recorded with determination program therefor 
JP2000353961A (en)  19990611  20001219  Fuji Xerox Co Ltd  Picture processor 
US20010003553A1 (en) *  19950825  20010614  Yoichiro Sako  Signal recording/reproducing method and apparatus, signal record medium and signal 
WO2001054415A1 (en)  20000120  20010726  Nokia Mobile Phones Ltd  A method and associated device for filtering digital video images 
US20020090961A1 (en) *  20010105  20020711  Conexant Systems, Inc.  System and method for cordless telephonic communication 
WO2003026296A1 (en)  20010917  20030327  Nokia Corporation  Method for subpixel value interpolation 
WO2003058945A2 (en)  20020114  20030717  Nokia Corporation  Coding dynamic filters 
EP1353514A2 (en)  20020410  20031015  Microsoft Corporation  Subpixel interpolation in motion estimation and compensation 
US20040062307A1 (en)  20020709  20040401  Nokia Corporation  Method and system for selecting interpolation filter type in video coding 
US20040076333A1 (en)  20021022  20040422  Huipin Zhang  Adaptive interpolation filter system for motion compensated predictive video coding 
EP1432249A1 (en)  20010918  20040623  Matsushita Electric Industrial CO., LTD  Image encoding method and image decoding method 
JP2005501442A (en)  20010425  20050113  ローベルト ボツシユ ゲゼルシヤフト ミツト ベシユレンクテル ハフツングＲｏｂｅｒｔ Ｂｏｓｃｈ Ｇｍｂｈ  Apparatus for interpolating sampled values, the image encoder and the image decoder 
US20050013369A1 (en)  20030623  20050120  TsuChang Lee  Method and apparatus for adaptive multipledimensional signal sequences encoding/decoding 
KR20050018948A (en)  20020709  20050228  노키아 코포레이션  Method and system for selecting interpolation filter type in video coding 
US20050117652A1 (en) *  20020502  20050602  FraunhoferGesellschaft Zur Forderung Der Angewandten Forschung E.V.  Method and arrangement for coding transform coefficients in picture and/or video coders and decoders and a corresponding computer program and a corresponding computerreadable storage medium 
EP1603338A1 (en)  20030310  20051207  Mitsubishi Denki Kabushiki Kaisha  Video signal encoding device and video signal encoding method 
WO2006004331A1 (en)  20040707  20060112  Samsung Electronics Co., Ltd.  Video encoding and decoding methods and video encoder and decoder 
EP1617672A1 (en)  20040713  20060118  Matsushita Electric Industrial Co., Ltd.  Motion estimator/compensator including a 16bit 1/8 pel interpolation filter 
EP1641274A1 (en)  20040923  20060329  Matsushita Electric Industrial Co., Ltd.  VIDEO CODING EMPLOYING An image DEGRAINING LOOPFILTER 
US20060093039A1 (en)  20041102  20060504  Kabushiki Kaisha Toshiba  Video image encoding method and video image encoding apparatus 
US20060133506A1 (en)  20041221  20060622  Stmicroelectronics, Inc.  Method and system for fast implementation of subpixel interpolation 
WO2006076602A1 (en)  20050113  20060720  Ntt Docomo, Inc.  A nonlinear, intheloop, denoising filter for quantization noise removal for hybrid video compression 
JP2006211152A (en)  20050126  20060810  Hokkaido Univ  Device and method for coding image and decoding image, and programs for coding and decoding image 
WO2006108654A2 (en)  20050413  20061019  Universität Hannover  Method and apparatus for enhanced video coding 
US20060291563A1 (en)  20020831  20061228  Samsung Electronics Co., Ltd.  Interpolation apparatus and method for motion vector compensation 
JP2007060199A (en)  20050824  20070308  Matsushita Electric Ind Co Ltd  Variable length decoder and integrated circuit including same 
WO2007047271A2 (en)  20051012  20070426  Thomson Licensing  Methods and apparatus for weighted prediction in scalable video encoding and decoding 
KR20070045662A (en)  20051028  20070502  경희대학교 산학협력단  Scalable h.264 encoding/decoding method for performing interlayer prediction using selected interpolator filter based on the rate distortion cost 
US20070120867A1 (en)  20051013  20070531  Shastry Pavan V  FIRbased interpolation in advanced video codecs on VLIW processor 
KR20080001116A (en)  20060629  20080103  삼성전자주식회사  Method and apparatus for controlling fusing temperature, image forming apparatus 
US20080013847A1 (en)  20060110  20080117  Texas Instruments Incorporated  Method and Apparatus for Processing AnalyticalForm Compression Noise in Images with Known Statistics 
WO2008016605A2 (en)  20060802  20080207  Thomson Licensing  Adaptive geometric partitioning for video decoding 
US20080037656A1 (en)  20060808  20080214  Miska Hannuksela  Method, device, and system for multiplexing of video streams 
US20080247467A1 (en)  20070109  20081009  Nokia Corporation  Adaptive interpolation filters for video coding 
JP2009047917A (en)  20070820  20090305  Fujitsu Ltd  Optical wavelength selective switch and control method 
US20090097545A1 (en)  20071014  20090416  Nokia Corporation  FixedPoint Implementation of an Adaptive Image Filter with High Coding Efficiency 
WO2009047917A1 (en)  20071011  20090416  Panasonic Corporation  Video coding method and video decoding method 
US20090257502A1 (en)  20080410  20091015  Qualcomm Incorporated  Ratedistortion defined interpolation for video coding based on fixed filter or adaptive filter 
Patent Citations (53)
Publication number  Priority date  Publication date  Assignee  Title 

JPH034685A (en)  19890601  19910110  Canon Inc  Encoding method for picture data 
JPH04230584A (en)  19910107  19920819  Toshiba Corp  Encoder and decoder 
US20010003553A1 (en) *  19950825  20010614  Yoichiro Sako  Signal recording/reproducing method and apparatus, signal record medium and signal 
EP0833520A2 (en)  19960930  19980401  Hyundai Electronics Industries Co., Ltd.  Video predictive coding apparatus and method 
JPH11145839A (en)  19971107  19990528  Nec Corp  Method for determining quantizing coefficient of digital filter and computerreadable record medium recorded with determination program therefor 
JP2000353961A (en)  19990611  20001219  Fuji Xerox Co Ltd  Picture processor 
WO2001054415A1 (en)  20000120  20010726  Nokia Mobile Phones Ltd  A method and associated device for filtering digital video images 
RU2295203C2 (en)  20000120  20070310  Нокиа Мобайл Фоунс Лтд  Method and appropriate device for filtration of digital video images 
US20020090961A1 (en) *  20010105  20020711  Conexant Systems, Inc.  System and method for cordless telephonic communication 
JP2005501442A (en)  20010425  20050113  ローベルト ボツシユ ゲゼルシヤフト ミツト ベシユレンクテル ハフツングＲｏｂｅｒｔ Ｂｏｓｃｈ Ｇｍｂｈ  Apparatus for interpolating sampled values, the image encoder and the image decoder 
WO2003026296A1 (en)  20010917  20030327  Nokia Corporation  Method for subpixel value interpolation 
RU2317654C2 (en)  20010917  20080220  Нокиа Корпорейшн  Method for interpolation of subpixel values 
US20030112864A1 (en)  20010917  20030619  Marta Karczewicz  Method for subpixel value interpolation 
EP1432249A1 (en)  20010918  20040623  Matsushita Electric Industrial CO., LTD  Image encoding method and image decoding method 
CN1615645A (en)  20020114  20050511  诺基亚有限公司  Coding dynamic filters 
JP2005514872A (en)  20020114  20050519  ノキア コーポレイション  Coding of the dynamic filter 
US20030169931A1 (en) *  20020114  20030911  Nokia Corporation  Coding dynamic filters 
WO2003058945A2 (en)  20020114  20030717  Nokia Corporation  Coding dynamic filters 
RU2302707C2 (en)  20020114  20070710  Нокиа Корпорейшн  Dynamic encoding filters 
EP1353514A2 (en)  20020410  20031015  Microsoft Corporation  Subpixel interpolation in motion estimation and compensation 
JP2003333599A (en)  20020410  20031121  Microsoft Corp  Subpixel interpolation in motion estimation and compensation 
US20050117652A1 (en) *  20020502  20050602  FraunhoferGesellschaft Zur Forderung Der Angewandten Forschung E.V.  Method and arrangement for coding transform coefficients in picture and/or video coders and decoders and a corresponding computer program and a corresponding computerreadable storage medium 
KR20050018948A (en)  20020709  20050228  노키아 코포레이션  Method and system for selecting interpolation filter type in video coding 
US20040062307A1 (en)  20020709  20040401  Nokia Corporation  Method and system for selecting interpolation filter type in video coding 
US20060291563A1 (en)  20020831  20061228  Samsung Electronics Co., Ltd.  Interpolation apparatus and method for motion vector compensation 
JP2004147328A (en)  20021022  20040520  Docomo Communications Laboratories Usa Inc  Adaptive interpolation filter system for motion compensated predictive video coding 
US20040076333A1 (en)  20021022  20040422  Huipin Zhang  Adaptive interpolation filter system for motion compensated predictive video coding 
EP1603338A1 (en)  20030310  20051207  Mitsubishi Denki Kabushiki Kaisha  Video signal encoding device and video signal encoding method 
US7471724B2 (en)  20030623  20081230  Vichip Corp. Limited  Method and apparatus for adaptive multipledimensional signal sequences encoding/decoding 
US20050013369A1 (en)  20030623  20050120  TsuChang Lee  Method and apparatus for adaptive multipledimensional signal sequences encoding/decoding 
WO2006004331A1 (en)  20040707  20060112  Samsung Electronics Co., Ltd.  Video encoding and decoding methods and video encoder and decoder 
EP1617672A1 (en)  20040713  20060118  Matsushita Electric Industrial Co., Ltd.  Motion estimator/compensator including a 16bit 1/8 pel interpolation filter 
EP1641274A1 (en)  20040923  20060329  Matsushita Electric Industrial Co., Ltd.  VIDEO CODING EMPLOYING An image DEGRAINING LOOPFILTER 
JP2006135376A (en)  20041102  20060525  Toshiba Corp  Moving picture coder, moving picture coding method, moving picture decoder, and moving picture decoding method 
US20060093039A1 (en)  20041102  20060504  Kabushiki Kaisha Toshiba  Video image encoding method and video image encoding apparatus 
US20060133506A1 (en)  20041221  20060622  Stmicroelectronics, Inc.  Method and system for fast implementation of subpixel interpolation 
WO2006076602A1 (en)  20050113  20060720  Ntt Docomo, Inc.  A nonlinear, intheloop, denoising filter for quantization noise removal for hybrid video compression 
JP2006211152A (en)  20050126  20060810  Hokkaido Univ  Device and method for coding image and decoding image, and programs for coding and decoding image 
WO2006108654A2 (en)  20050413  20061019  Universität Hannover  Method and apparatus for enhanced video coding 
JP2008536414A (en)  20050413  20080904  ゴットフリート・ヴィルヘルム・ライプニッツ・ウニヴェルジテート・ハノーヴァー  Extended coding method and apparatus for video 
JP2007060199A (en)  20050824  20070308  Matsushita Electric Ind Co Ltd  Variable length decoder and integrated circuit including same 
WO2007047271A2 (en)  20051012  20070426  Thomson Licensing  Methods and apparatus for weighted prediction in scalable video encoding and decoding 
US20070120867A1 (en)  20051013  20070531  Shastry Pavan V  FIRbased interpolation in advanced video codecs on VLIW processor 
KR20070045662A (en)  20051028  20070502  경희대학교 산학협력단  Scalable h.264 encoding/decoding method for performing interlayer prediction using selected interpolator filter based on the rate distortion cost 
US20080013847A1 (en)  20060110  20080117  Texas Instruments Incorporated  Method and Apparatus for Processing AnalyticalForm Compression Noise in Images with Known Statistics 
KR20080001116A (en)  20060629  20080103  삼성전자주식회사  Method and apparatus for controlling fusing temperature, image forming apparatus 
WO2008016605A2 (en)  20060802  20080207  Thomson Licensing  Adaptive geometric partitioning for video decoding 
US20080037656A1 (en)  20060808  20080214  Miska Hannuksela  Method, device, and system for multiplexing of video streams 
US20080247467A1 (en)  20070109  20081009  Nokia Corporation  Adaptive interpolation filters for video coding 
JP2009047917A (en)  20070820  20090305  Fujitsu Ltd  Optical wavelength selective switch and control method 
WO2009047917A1 (en)  20071011  20090416  Panasonic Corporation  Video coding method and video decoding method 
US20090097545A1 (en)  20071014  20090416  Nokia Corporation  FixedPoint Implementation of an Adaptive Image Filter with High Coding Efficiency 
US20090257502A1 (en)  20080410  20091015  Qualcomm Incorporated  Ratedistortion defined interpolation for video coding based on fixed filter or adaptive filter 
NonPatent Citations (37)
Title 

"Information technologyGeneric coding of moving pictures and associated audio information: Video; H.262 (02/00)" ITUT Standard in Force (I), International Telecommunication Union, Geneva, CH, No. H.262 (02/00), Feb. 1, 2000, XP017401436. 
"Information technology—Generic coding of moving pictures and associated audio information: Video; H.262 (02/00)" ITUT Standard in Force (I), International Telecommunication Union, Geneva, CH, No. H.262 (02/00), Feb. 1, 2000, XP017401436. 
"Text of ISO/IEC 1449610:200X / FDIS Advanced Video Coding (4th edition)" 81. MPEG Meeting;Jun. 26, 2007; Lausanne; (Motion Picture Expert Group or ISO/IEC JTC1/SC29/WG11) No. N9198, Oct. 28, 2007, XP030015692. 
"Text of ISO/IEC 1449610:200X/FDIS Advanced Video Coding (4th edition)" 81. MPEG Meeting; Jun. 26, 2007; Lausanne; (Motion Picture Expert Froup or ISO/IEC JTC1/SC29/WG11), No. N9198, Oct. 29, 2007, XP030015692. 
Chen P., et al.,"Switched Interpolation Filter with Offset," ITUT SGI 16/Q.6 Doc. VCEGAI35, Berlin Germany, Jul. 1618, 2008. 
D. Rusanovskyy, K. Ugur, J. Lainema, "Adaptive Interpolation with Directional Filters," ITUT SGI 6/Q.6 Doc. VCEGAG21, Shenzhen, China. 
International Search ReportPCT/US09/040250, International Search AuthorEuropean Patent OfficeJul. 28, 2009. 
International Search Report—PCT/US09/040250, International Search Author—European Patent Office—Jul. 28, 2009. 
International Search ReportPCT/US09/040254, International Search AuthorityEuropean Patent OfficeJul. 14, 2009. 
International Search Report—PCT/US09/040254, International Search Authority—European Patent Office—Jul. 14, 2009. 
J. Jung and G. Laroche, "Performance Evaluation of the KTA 1.2 Software," ITUT SG16/Q.6 VCEG, VCEGAE09, Marrakech, Morocco, Jan. 2007. 
Miyamoto: "Adaptive Motion Interpolation on MBbasis" Joint Video Team (JVT) of ISO/IEC MPEG & ITUT VCEG(ISO/IEC JTC1/SC29/ WG11 and ITUT SG16 06), XX, XX, No. JVT0040, May 10, 2002, XP030005148. 
Miyamoto: "Adaptive Motion interpolation on MBbasis" Joint Video Team (JVT) of ISO/IEC MPEG & ITUT VCEG(ISO/IEC JTC1/SC29/WG11 and ITUT SG16 Q6), XX, XX, No. JVTC040, May 10, 2002, XP030005148. 
QUALCOMM: "Enhanced Adaptive Interpolation Filter", Apr. 14, 2008, ITUT SG16IQ.6 DOC. T05SG16C 464E, International Telecommunication Union, Geneva, CH, pp. 14, XP007908846. 
QUALCOMM: "Switched Interpolation Filter with Offset" ITUT SG16/Q.6 DOC. T05SGC 463 E, International Telecommunication Union, Geneva, CH, Apr. 14, 2008, pp. 14, XP007908845. 
Rapporteur 06/16: "Report of the former Q6/16 Raporteurs meeting (Berlin, Germany, Jul. 1618, 2008 and Hannover, Germany, Jul. 2025, 2008)", ITUT SG16 meeting; Jan. 27, 2009Feb. 6, 2009; Geneva, T09SG16090127TDWP30007, Jan. 8, 2009. 
S. Wittmann, T. Wedi, "Separable Adaptive Interpolation Filter," ITUT SG 16/Q.6 Doc. T05SG16C0219, Geneva, Switzerland, Jun. 2007. 
Spinivasan S et al: "An overview of VC1" Visual Commuications and Image Procesing; Jul. 1215, 2005; Beijing, XP030080912. 
Srinivasan S et al: "An overview of VC1" Visual Communications and Image Processing; Jul. 1215, 2005; Beijing,, XP030080912. 
Taiwan Search ReportTW098112116TIPOApr. 8, 2013. 
Taiwan Search Report—TW098112116—TIPO—Apr. 8, 2013. 
Ugur K. et al, Adaptive Interpolation Filter with Flexible Symmetry for Coding High Resolution High Quality Video, Acoustics, Speech and Signal Processing, 2007. ICASSP, 2007 IEEE International Conference on vol. 1, Apr. 1520, 2007, pp. I1013. 
Vatis Y et al: "MotionAnd AliasingCompensated Prediction Using a TwoDimensional NonSeparable Adaptive Wiener Interpolation Filter" Image Processing, 2005. ICIP 2005. IEEE International Conference on Genova, Italy Sep. 1114, 2005, Piscataway, NJ, USA, IEEE, vol. 2, Sep. 11, 2005, pp. 894897, XP010851198 ISBN: 9780780391 345. 
Vatis, Y., et al., "Prediction of P and BFrames Using TwoDimensional NonSeparable Adaptive Wiener Interpolation Filter for H.264/AVC" Joint Video Team (JVT) of ISO/IEC MPEG & ITUT VCEG(ISO/IEC JTC1/SC29WG11 and ITUT SG16 Q6), XX, XX, No. VCEGAD08, Oct. 29, 2006, XP030003499. 
Vats Y et al: "Coding of coefficients of twodimensional nonseparable adaptive Wiener interpoiation filter" Proceedings of the SPIEThe International Society for Optical Engineering, SPIE, PO BOX 10; Bellingham WA 982270010 USA, vol. 5960, Jul. 12, 2005, pp. 623631, XP002419200. 
Vats Y et al: "Coding of coefficients of twodimensional nonseparable adaptive Wiener interpoiation filter" Proceedings of the SPIE—The International Society for Optical Engineering, SPIE, PO BOX 10; Bellingham WA 982270010 USA, vol. 5960, Jul. 12, 2005, pp. 623631, XP002419200. 
Wedi T., "Adaptive Interpolation Filter for Motion Compensated Prediction", Proc. IEEE International Conference on Image Processing (ICIP), New York, Sep. 2002. 
Wedi T: "Adaptive interpolation filters and highresolution displacement for video coding" IEEE Transactions on Circuits and Systems for Video Technology, IEEE Service Center, Piscataway, NJ, US vol. 16, No. 4, Apr. 1, 2006, pp. 484491, XP002464225. 
Wedi T: "Adaptive Interpolation Filters and HighResolution Displacements for Video Coding" IEEE Transactions on Circuits and Systems for Video Technology, IEEE Service Center, Piscataway, NJ, US, vol. 16, No. 4, Apr. 1, 2006, pp. 484491, XP002464225, ISSN: 10518215; & EP1641274A (Matsushita Electric Ind Co Ltd [JP]) Mar. 29, 2006. 
Wittmann S et al: "Separable adaptive interpolation filter" TUT SG16/06, Question 6/16, Study Group 16Contribution 219, Document Com 16C 219E, Geneve, Switzerland, Jun. 1, 2007, pp. 19, XP007907321. 
Wittmann S et al: "Separable adaptive interpolation filter" TUT SG16/06, Question 6/16, Study Group 16—Contribution 219, Document Com 16—C 219—E, Geneve, Switzerland, Jun. 1, 2007, pp. 19, XP007907321. 
Written OpinionPCT/US09/040250, International Search AuthorityEuropean Patent OfficeJul. 28, 2009. 
Written Opinion—PCT/US09/040250, International Search Authority—European Patent Office—Jul. 28, 2009. 
Written OpinionPCT/US09/040254, International Search AuthorityEuropean Patent OfficeJul. 14, 2009. 
Written Opinion—PCT/US09/040254, International Search Authority—European Patent Office—Jul. 14, 2009. 
Y. Vatis and J. Ostermann, "Comparison of Complexity Between TwoDimensional NonSeparable Adaptive Interpolation Filter and Standard Wiener Filter," ITUT SGI 6/Q.6 Doc. VCEGAA11, Nice, France, Oct. 2005. 
Yuri Vatis et al: "Syntax of adaptive filter coefficients in the KTA reference model" Joint Video Team (JVT) of ISO/IEC MPEG & ITUT VCEG(ISO/IEC JTC1/SC29/WG11 and ITUT SG16 Q6), XX, XX, No. VCEGAF09, May 2, 2007, XP030003530. 
Cited By (5)
Publication number  Priority date  Publication date  Assignee  Title 

US20120051438A1 (en) *  20100901  20120301  Qualcomm Incorporated  Filter description signaling for multifilter adaptive filtering 
US9819966B2 (en) *  20100901  20171114  Qualcomm Incorporated  Filter description signaling for multifilter adaptive filtering 
US20130216150A1 (en) *  20101208  20130822  Sony Corporation  Image processing device, image processing method, and program 
US20150071345A1 (en) *  20130909  20150312  Apple Inc.  Chroma quantization in video coding 
US9294766B2 (en) *  20130909  20160322  Apple Inc.  Chroma quantization in video coding 
Also Published As
Publication number  Publication date  Type 

CN101990759B (en)  20130717  grant 
US20090257668A1 (en)  20091015  application 
CA2719215A1 (en)  20091015  application 
JP2011517237A (en)  20110526  application 
KR20100133004A (en)  20101220  application 
EP2277316A1 (en)  20110126  application 
CN101990759A (en)  20110323  application 
WO2009126911A1 (en)  20091015  application 
KR101202632B1 (en)  20121119  grant 
Similar Documents
Publication  Publication Date  Title 

US7379501B2 (en)  Differential coding of interpolation filters  
US20040062307A1 (en)  Method and system for selecting interpolation filter type in video coding  
US20050013497A1 (en)  Intraframe and interframe interlace coding and decoding  
US20100002770A1 (en)  Video encoding by filter selection  
US20120082224A1 (en)  Intra smoothing filter for video coding  
US20030156646A1 (en)  Multiresolution motion estimation and compensation  
US20120163455A1 (en)  Mode dependent scanning of coefficients of a block of video data  
US20120170649A1 (en)  Video coding using mapped transforms and scanning modes  
US20120320969A1 (en)  Unified merge mode and adaptive motion vector prediction mode candidates selection  
US20100284458A1 (en)  Adaptive filtering  
US20110103480A1 (en)  Global motion parameter estimation using blockbased motion vectors  
US20120093226A1 (en)  Adaptive motion vector resolution signaling for video coding  
US20060268166A1 (en)  Method and apparatus for coding motion and prediction weighting parameters  
US20140146875A1 (en)  Loop filtering across constrained intra block boundaries in video coding  
US20110206123A1 (en)  Block type signalling in video coding  
US20050013372A1 (en)  Extended range motion vectors  
EP2262267A1 (en)  Filter coefficient coding scheme for video coding  
US20100246692A1 (en)  Flexible interpolation filter structures for video coding  
US20090022220A1 (en)  Method and apparatus for enhanced video coding  
US20070065026A1 (en)  Method of and apparatus for lossless video encoding and decoding  
US20090110066A1 (en)  Method and Apparatus for Selecting a Coding Mode  
US20120033736A1 (en)  Image processing device and method  
US20100220788A1 (en)  Video coding method and video decoding method  
US20120033728A1 (en)  Method and apparatus for encoding and decoding images by adaptively using an interpolation filter  
US20120243609A1 (en)  Bipredictive merge mode based on unipredictive neighbors in video coding 
Legal Events
Date  Code  Title  Description 

AS  Assignment 
Owner name: QUALCOMM INCORPORATED, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YE, YAN;KARCZEWICZ, MARTA;REEL/FRAME:022521/0384;SIGNING DATES FROM 20090406 TO 20090407 Owner name: QUALCOMM INCORPORATED, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YE, YAN;KARCZEWICZ, MARTA;SIGNING DATES FROM 20090406 TO 20090407;REEL/FRAME:022521/0384 

FEPP 
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.) 

FP  Expired due to failure to pay maintenance fee 
Effective date: 20180909 