RU2505938C2 - Distortion-based interpolation depending on transmission rate for video coding based on fixed filter or adaptive filter - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: disclosed is twelve-pixel filter support, which may be used for interpolation, techniques which use coefficient symmetry and pixel symmetry to reduce the amount of data needed to be transmitted between an encoder and a decoder to configure the filter support for interpolation, and techniques for filtering data at integer pixel locations in a manner that is similar to sub-pixel interpolation. Other aspects relate to techniques for coding information in a bit stream in order to transmit the type of the used filter and used filter coefficients. Also described is predictive coding of filter coefficients.

EFFECT: high accuracy of predictive data used during fractional interpolation and in some cases improved predictive data of integer blocks of pixels.

47 cl, 29 dwg

Description

This application claims the priority of provisional application US 61/044020, filed April 10, 2008, provisional US application 61/044023, filed April 10, 2008, provisional application US 61/044240, filed April 11, 2008, and provisional application US 61/057373 filed May 30, 2008, the entire contents of which are incorporated herein by reference.

FIELD OF THE INVENTION

This disclosure relates to the encoding and decoding of digital video, and more specifically, to filtering technologies used to generate predictive data used in encoding and decoding video.

BACKGROUND

Digital video capabilities can be embedded in a wide range of devices, including digital TVs, digital direct broadcast systems, wireless broadcast systems, personal digital devices (PDAs), travel or desktop computers, digital cameras, digital recording devices, video game devices, video game consoles, cellular or satellite radiotelephones, etc. Digital video devices implement video compression technologies, such as those described in the standards defined by MPEG-2, MPEG-4, or ITU-T H.264 / MPEG-4, Part 10, Advanced Video Encoding (AVC) to more efficiently transmit and receive digital video information. Video compression technologies may perform spatial prediction and / or temporal prediction to reduce or remove redundancy internally present in video sequences.

Block-based cross-coding is a useful coding technology that is based on temporal prediction to reduce or remove temporal redundancy between video blocks of consecutive coded units of a video sequence. Coded units may comprise video frames, series of consecutive macroblocks of video frames, image groups, or another specified unit of encoded video blocks. For inter-coding, a video encoder performs motion estimation to track the movement of matching video blocks between two or more adjacent coded units. The motion estimation generates motion vectors that indicate the offset of the video blocks relative to the corresponding predicted video blocks in one or more reference frames or other coded units. Motion compensation uses motion vectors to form predictive video blocks from one or more reference frames or other coded units. After motion compensation, residual video blocks are formed by subtracting the predicted video blocks from the encoded source video blocks.

The video encoder can also apply transform, quantization, and entropy coding processes to further reduce the bit rate associated with the transmission of residual blocks. Transformation technologies may include discrete cosine transforms (DCT) or conceptually similar processes. Alternatively, wavelet transforms, integer transforms, or other types of transforms may be used. In the DCT process, as an example, a set of pixel values is converted to transform coefficients that can represent the energy of the pixel values in the frequency domain. Quantization is applied to transform coefficients and, in general, comprises a process that reduces the number of bits associated with any given transform coefficient. Entropy coding comprises one or more processes that together compress a sequence of coding modes, motion information, encoded block patterns, and quantized transform coefficients. Examples of entropy coding include, but are not limited to, context-adaptive variable-length coding (CAVLC) and context-adaptive binary arithmetic coding (CABAC).

An encoded video block may be represented by prediction information that can be used to create or identify a prediction block and a residual block of data indicating differences between the encoded block and the prediction block. The prediction information may comprise one or more motion vectors that are used to identify the prediction data block. Given the motion vectors, the decoder can reconstruct the prediction blocks that are used to encode the remainder. Thus, given the set of residual blocks and the set of motion vectors (and possibly some additional syntax), the decoder can reconstruct the video frame that was originally encoded. Mutual coding based on motion estimation and motion compensation can achieve very good compression, since sequential video frames or other types of encoded units are often pretty much the same. An encoded video sequence may contain blocks of residual data, motion vectors, and possibly other types of syntax.

Interpolation technologies have been developed to increase the level of compression that can be achieved with inter-frame coding. In this case, the predictive data generated during the motion compensation, which is used to encode the video block, can be interpolated from the pixels of the video blocks of the video frame or other coded unit used in the motion estimation. Interpolation is often performed to generate predictive half-pixel values (at half a pixel) and predictive quarter-pixel values (at a quarter of a pixel). Half-pixel and quarter-pixel values are associated with sub-pixel locations. Fractional motion vectors can be used to identify video blocks at subpixel resolution, to capture fractional movement in the video sequence, and thereby provide predictive blocks that are more similar to encoded video blocks than integer video blocks.

SUMMARY OF THE INVENTION

In general, the present disclosure describes filtering techniques employed by an encoder and a decoder during the prediction step of a video encoding and / or decoding process. The described filtering technologies make it possible to increase the accuracy of the predictive data used during fractional interpolation, and in some cases allow to improve the predictive data of entire blocks of pixels. Several aspects are envisioned for this disclosure, including applicable twelve pixel filter support that can be used for interpolation, technologies that use coefficient symmetry and pixel symmetry to reduce the amount of data needed to be sent between the encoder and decoder in order to configure filter support for interpolation; and technology for filtering data at integer-pixel locations in a manner that is similar to sub-pixel interpolation. Other aspects of this disclosure relate to techniques for encoding information in a bitstream to convey the type of filter used and possibly the filter coefficients used. Predictive coding techniques for filter coefficients are also described. These and other aspects of the present disclosure will be apparent from the description below.

In one example, the present disclosure describes a method comprising generating first interpolated predictive data for encoding video data based on a first interpolation filter, generating second interpolated predictive data for encoding video data based on a second interpolation filter, choosing between first interpolated predictive data and second interpolated predictive data on Based on the analysis of distortion depending on the transmission rate, video coding data based on the selection and coding syntax to indicate the choice. The first interpolation filter may comprise a non-tunable interpolation filter, and the second interpolation filter may comprise an adaptive interpolation filter, but the present disclosure is not limited to these examples.

Additional interpolation filters can also be used to generate additional interpolated predictive data, which can also be considered in the analysis of distortion depending on the transmission speed. In other words, the method is not limited to generating only the first and second interpolated predictive data based on two interpolation filters, but can be used to generate any set of interpolated predictive data based on any number of interpolation filters. It is important that the interpolated predictive distortion analysis data as a function of the transmission rate is used to identify which filter should be selected.

According to another example, the present disclosure describes a method comprising generating a plurality of different versions of predictive data for encoding video data based on a plurality of different interpolation filters, selecting from a plurality of different versions of predictive data based on an analysis of distortion depending on a transmission rate, selectively encoding the video data, and encoding syntax to indicate a choice.

In another example, the present disclosure describes a device comprising a video encoder that generates a plurality of different versions of predictive data for encoding video data based on a plurality of different interpolation filters, selects from many different versions of predictive data based on a distortion analysis depending on a transmission rate, encodes video data based on selection and encodes syntax to indicate selection.

In another example, the present disclosure describes a device comprising a means for generating a plurality of different versions of predictive data for encoding video data based on a plurality of different interpolation filters, means for selecting from a plurality of different versions of predictive data based on a distortion analysis depending on a transmission rate, means for encoding selection-based video data; and syntax encoding means to indicate selection.

The techniques described in this disclosure may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the software may run on one or more processors, such as a microprocessor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), or digital signal processor (DSP). Software that implements technology can be initially stored on a machine-readable medium and downloaded and executed on a processor.

Accordingly, the present disclosure also contemplates a computer-readable storage medium containing instructions that, when executed by a processor, instruct the processor to generate a plurality of different versions of predictive data for encoding video data based on a plurality of different interpolation filters, select from a plurality of different versions of predictive data based on a distortion analysis depending on the transmission speed, encode the video data based on the choice and encode the syntax that to indicate the selection.

Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and in the description below. Other features, objectives and advantages of the technologies described in this disclosure will be apparent from the description and drawings, as well as from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 is a block diagram illustrating one exemplary video encoding and decoding system that may implement the technologies of the present invention.

FIG. 2 is a block diagram illustrating an example of a video encoder that can perform filtering techniques in accordance with the present invention.

3 is a conceptual diagram illustrating integer-pixel positions associated with predictive data and sub-pixel positions associated with interpolated predictive data.

4 is a conceptual diagram illustrating 12 pixel filter support with respect to nine sub-pixel locations.

5 is a conceptual diagram illustrating 6 pixel horizontal filter support with respect to three horizontal subpixel locations and 6 pixel vertical filter support with respect to three vertical subpixel locations.

6 is a conceptual diagram illustrating a "five pixel by five pixel" filter support for filtering an integer pixel location.

7 is a conceptual diagram illustrating four integer-pixel positions and fifteen sub-pixel positions with a hatch to group pixel positions that can use pixel symmetry for filter coefficients in accordance with the present invention.

8 is a conceptual diagram illustrating six positions of supporting horizontal linear pixels relative to a subpixel, with a hatching that shows symmetry in coefficients.

9 is a conceptual diagram illustrating six positions of supporting horizontal linear pixels with respect to a subpixel, with a hatching that shows the lack of symmetry in the coefficients.

10 is a conceptual diagram illustrating six support positions of vertical linear pixels with respect to a subpixel, with a hatching that shows symmetry in coefficients.

11 is a conceptual diagram illustrating six positions of supporting vertical linear pixels with respect to a subpixel, with a hatching that shows the lack of symmetry in the coefficients.

12 is a conceptual diagram illustrating twelve positions of supporting two-dimensional pixels with respect to a subpixel, with hatching, which shows a lack of symmetry in the coefficients.

13 is a conceptual diagram illustrating twelve positions of supporting two-dimensional pixels with respect to a subpixel, with a hatching that shows symmetry in coefficients.

14 is a conceptual diagram illustrating twelve positions of supporting two-dimensional pixels with respect to a subpixel, with a hatching that shows symmetry in coefficients.

15 is a conceptual diagram illustrating twelve positions of supporting two-dimensional pixels with respect to a subpixel, with a hatching that shows symmetry in coefficients.

16 is a block diagram illustrating an example of a video decoder that can decode a video sequence encoded by the method described herein.

17 is a flowchart illustrating an example operation of a video encoder that uses twelve pixel filter support in accordance with the present invention.

18 is a flowchart illustrating an example operation of a video decoder that uses twelve pixel filter support in accordance with the present invention.

19 is a flowchart illustrating an exemplary operation of a video encoder that uses coefficient symmetry and pixel symmetry in accordance with the present invention.

FIG. 20 is a flowchart illustrating an example operation of a video decoder that uses coefficient symmetry and pixel symmetry in accordance with the present invention.

FIG. 21 is a flowchart illustrating an example operation of a video encoder that uses integer-pixel filtering to generate adjustable integer-pixel values in accordance with the present invention.

FIG. 22 is a flowchart illustrating an example operation of a video decoder that uses integer-pixel filtering to generate adjustable integer-pixel values in accordance with the present invention.

23 is a flowchart illustrating a technology for distortion-based interpolation depending on a transmission rate for encoding a video based on a non-tunable filter or an adaptive filter.

24 is a flowchart illustrating a technique for encoding filter coefficients using predictive coding.

25 is another flowchart illustrating a technique for encoding filter coefficients using predictive coding.

26 is a flowchart illustrating a technique for decoding filter coefficients using predictive coding.

27 and 28 are conceptual graphs illustrating filtering coefficients that can be predictively encoded.

29 is an illustrative example of a matrix of integer-pixel filtering coefficients for which prediction techniques can be used for coding.

DETAILED DESCRIPTION OF THE INVENTION

The present disclosure describes filtering techniques applied by an encoder and a decoder during the prediction step of a video encoding and / or decoding process. The described filtering technologies make it possible to increase the accuracy of the predictive data used during fractional interpolation, and in some cases allow to improve the predictive data of entire blocks of pixels. Several aspects of the present disclosure are envisioned, including applicable twelve pixel filter support that can be used for interpolation, technologies that use coefficient symmetry and pixel symmetry to reduce the amount of data needed to be sent between the encoder and decoder to configure support a filter for interpolation; and technology for filtering data at integer-pixel locations in a manner that is similar to sub-pixel interpolation. These and other technologies are described in detail below.

1 is a block diagram illustrating an example video encoding and decoding system 10 that can be used to implement one or more technologies of the present invention. As shown in FIG. 1, system 10 includes a source device 12 that transmits encoded video to a target device 16 through a communication channel 15. The source device (source) 12 and the target device (destination) 16 may comprise any of a wide range of devices. In some cases, the source device 12 and the target device 16 comprise wireless communication devices such as cordless handsets, so-called cellular or satellite radiotelephones, or any wireless devices that can transmit video information over communication channel 15 when communication channel 15 is wireless. However, the technologies of the present disclosure that relate to filtering and generating predictive data during predictive coding are not limited to wireless applications or settings. Technologies can also be applied in a wide range of other settings and devices, including devices that exchange data through physical wires, optical fiber, or other physical or wireless media. In addition, encoding or decoding technologies can also be applied in a stand-alone device, which does not necessarily communicate with any other device.

In the example of FIG. 1, the source device 12 may include a video source 20, a video encoder 22, a modulator / demodulator (modem) 23, and a transmitting device 24. The target device 16 may include a receiving device 26, a modem 27, a video decoder 28, and a display device 30. In accordance with the present disclosure, video encoder 22 of source device 12 may be configured to apply one or more technologies of the present disclosure as part of a video encoding process. Similarly, the video decoder 28 of the target device 16 may be configured to apply one or more of the technologies of the present disclosure as part of the video decoding process.

The system 10 illustrated in FIG. 1 is for example purposes only. Various techniques of the present disclosure may be performed by any encoding device that supports block-based predictive coding, or any decoding device that supports block-based predictive decoding. The source device 12 and the target device 16 are simply examples of such encoding devices in which the source device 12 generates encoded video data for transmission to the target device 16. In some cases, the devices 12, 16 can operate almost symmetrically, so that each of the devices 12, 16 includes Video encoding and decoding components. Therefore, the system 10 can support one-way and two-way video transmission between video devices 12, 16, for example, for video streaming, video playback, video broadcasting, or video telephony.

The video source 20 of the source device 12 may include a video capture device, such as a video camera, a video archive containing previously captured video, or a video stream from a video content provider. As an additional alternative, video source 20 may generate computer-based graphics data as a source video or a combination of directly transmitted video, archived video, and computer-generated video. In some cases, if the video source 20 is a video camera, the source device 12 and the target device 16 may form so-called camera phones or video phones. In each case, the captured, pre-captured, or machine-generated video may be encoded by video encoder 22. The encoded video information may then be modulated by modem 23 according to a communication standard, such as, for example, code division multiple access (CDMA) or another communication standard, and transmitted to the target device 16 through the transmitting device 24 and the communication channel 15. The modem 23 may include various mixers, filters, amplifiers, or other components designed to modulate the signals. Transmitter 24 may include circuits designed to transmit data, including amplifiers, filters, and one or more antennas.

The receiver 26 of the target device 16 receives information on channel 15, and the modem 27 demodulates the information. Similar to the transmitter 24, the receiver 26 may include circuits for receiving data, including amplifiers, filters, and one or more antennas. In some cases, the transmitting device 24 and / or the receiving device 26 may be included in one component of the receiving-transmitting device, which includes both the receiving and transmitting circuits. Modem 27 may include various mixers, filters, amplifiers, or other components designed to demodulate signals. In some cases, modems 23 and 27 may include components for performing both modulation and demodulation.

The video encoding process performed by the video encoder 22 may implement one or more of the technologies described herein during motion compensation. The video decoding process performed by the video decoder 28 can also perform such technologies during its stage of motion compensation of the decoding process. The term "encoder" is used here to refer to a specialized computer device that performs video encoding or video decoding. The term “encoder” generally means any of a video encoder, video decoder, or combined encoder / decoder (codec). The term "coding" means encoding or decoding. The display device 30 displays the decoded video data to the user and may comprise any of a variety of display devices, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light emitting diode (OLED) display, or other type of display device.

In the example of FIG. 1, communication channel 15 may comprise any wireless and wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines, or any combination of wireless and wired media. Communication channel 15 may form part of a packet-switched network, such as a local area network, wide area network, or wide area network such as the Internet. Communication channel 15 generally represents any suitable communication medium or a set of different communication media for transmitting video data from source device 12 to target device 16. Communication channel 15 may include routers, switches, base stations, or any other equipment that may be used to simplify the transfer of data from the source device 12 to the target device 16.

Video encoder 22 and video decoder 28 may operate according to a video compression standard, such as ITU-T H.264, alternatively described as MPEG-4, part 10, advanced video encoding (AVC). However, the technology of the present disclosure is not limited to any particular video coding standard. Although not shown in FIG. 1, in some aspects, video encoder 22 and video decoder 28 may be integrated with an audio encoder and decoder, respectively, and may include appropriate multiplexer-demultiplexer modules or other hardware and software to process the encoding as audio video and in the general data stream or in separate data streams. The multiplexer-demultiplexer modules may conform to the ITU H.223 multiplexer protocol or other protocols such as user datagram protocol (UDP).

Video encoder 22 and video decoder 28 can be implemented as one or more microprocessors, digital signal processors (DSPs), specialized integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any one thereof combinations. Each of video encoder 22 and video decoder 28 can be included in one or more encoders or decoders, any of which can be integrated as part of a combined codec that provides encoding and decoding capabilities in a corresponding mobile device, subscriber device, broadcast device, server, etc. P.

A video sequence typically includes a series of video frames. Video encoder 22 operates with video blocks within separate video frames to encode video data. Video blocks may have a fixed or variable size and may vary in size according to a given coding standard. Each video frame includes a series of consecutive macroblocks. Each series of consecutive macroblocks may include a series of macroblocks that can be arranged in subunits. As an example, the ITU-T H.264 standard supports intra prediction for various block sizes, such as 16 × 16, 8 × 8 or 4 × 4 for luminance components and 8 × 8 for chrominance components, as well as inter prediction for different block sizes, such as 16 × 16, 16 × 8, 8 × 16, 8 × 8, 8 × 4, 4 × 8 and 4 × 4 for luminance components and corresponding scaled sizes for chroma components. Video blocks may comprise blocks of pixel data or blocks of transform coefficients, for example, after a transform process such as discrete cosine transform (DCT) or a conceptually similar transform process.

Smaller video blocks can provide better resolution and can be used for video frame locations that include high levels of detail. In general, macroblocks and various subunits can be considered video blocks. In addition, a series of consecutive macroblocks may be considered a series of video blocks, such as macroblocks and / or subblocks. Each series of consecutive macroblocks can be an independently decoded video frame unit. Alternatively, the frames themselves may be decoded units, or other parts of the frame may be specified as decoded units. The term “coded unit” means any independently decoded unit of a video frame, such as an entire frame, a series of consecutive macroblocks of a frame, or another independently decoded unit specified in accordance with the used coding techniques.

To encode video blocks, video encoder 22 performs internal (“intra”) or mutual (“inter”) prediction to form a prediction block. Video encoder 22 subtracts the prediction blocks from the source video blocks, which must be encoded to form residual blocks. Thus, residual blocks are indicative of differences between the coded blocks and the prediction blocks. Video encoder 22 may perform the conversion for the residual blocks to form blocks of transform coefficients. After technologies of internal or mutual predictive coding and conversion, video encoder 22 performs quantization. Quantization generally means a process in which coefficients are quantized in order to possibly reduce the amount of data used to represent the coefficients. After quantization, entropy coding can be performed according to entropy coding technology, such as context adaptive variable length coding (CAVLC) or context adaptive binary arithmetic coding (CABAC). Additional information for each step of the encoding process performed by video encoder 22 is described in more detail below in FIG.

At target device 16, video decoder 28 receives encoded video data. Video decoder 28 entropy decodes the received video data according to entropy coding technology such as CAVLC or CABAC to obtain quantized coefficients. Video decoder 28 applies the functions of inverse quantization (dequantization) and inverse transformation to restore the residual block in the pixel region. Video decoder 28 also generates a prediction block based on control information or syntactic information (for example, coding mode, motion vectors, syntax that defines filtering coefficients, etc.) included in the encoded video data. Video decoder 28 summarizes the prediction block with the reconstructed residual block to form the reconstructed video block for display. Additional information for each step of the encoding process performed by video encoder 22 is described in more detail below in FIG. 16.

According to the technologies of the present disclosure, video encoder 22 and video decoder 28 can use one or more interpolation filtering technologies during motion compensation. In particular, in accordance with one aspect of this disclosure, video encoder 22 and / or video decoder 28 may receive a pixel block, wherein the pixel block includes integer-pixel values corresponding to integer-pixel positions within the pixel block, to calculate sub-pixel values for sub-pixel positions associated with a block of pixels, based on integer-pixel values, while the calculation of sub-pixel values contains the use of an interpolation filter, which defines a two-dimensional array of filter support positions, with corresponding to a set of twelve or more integer-pixel positions that surround the sub-pixel positions in radial form and form a prediction block based on at least some of the sub-pixel values. An example of a two-dimensional array of filter support positions corresponding to a set of twelve or more integer-pixel positions is explained in more detail below.

In accordance with another aspect of the present disclosure, video encoder 22 and / or video decoder 28 may use symmetry aspects to reduce the amount of data that must be transmitted between source device 12 and target device 16 to exchange filter coefficients used in the interpolation. Video encoder 22 can determine eight sets of filter coefficients for fifteen different subpixel locations, with eight sets of filter coefficients being generated based on symmetry in coefficients and pixel symmetry for fifteen subpixel locations, and output eight sets of filter coefficients to another device as part of the encoded bitstream. Thus, eight sets, along with the aspects of pixel symmetry and coefficient symmetry, can specify all filtering coefficients for all fifteen quarter-pixel and half-pixel pixel positions. In addition, pixel symmetry may exist between different of these fifteen subpixel locations in the vertical dimension and in the horizontal dimension, but pixel symmetry may not exist in the diagonal dimension for at least some of the subpixel locations of the fifteen sets. This lack of pixel symmetry in the diagonal dimension, for at least some of these fifteen locations, allows for better interpolations and higher video quality for video encoding and decoding.

Video decoder 28 of the target device 16 can receive eight sets of filter coefficients as part of the encoded video bit stream, generate fifteen sets of filter coefficients corresponding to fifteen different sub-pixel locations, based on eight sets of filter coefficients, generate interpolated predictive data for video decoding based on one of the fifteen sets of coefficients filtering, while the interpolated predictive data correspond to one of the spots twenty different sub-pixel locations, and decode one or more video blocks based on interpolated predictive data.

In accordance with another aspect, video encoder 22 and / or video decoder 28 may use similar interpolation filtering with respect to integer-pixel positions to form adjustable integer-pixel values. Such similar interpolation filtering allows you to improve compression specifically during lighting changes, exits from darkening or darkening the scene, allows you to remove noise and simplify sharpening frames with images, and helps to improve the encoding of the exact movement of an object between consecutive video frames, in particular, when the symmetry does not overlap filter coefficients.

Techniques for similar interpolation filtering of video encoder 22 and / or video decoder 28 may include obtaining pixel blocks, wherein pixel blocks include integer-pixel values corresponding to integer-pixel positions within pixel blocks, filtering integer-pixel values based on other integer-pixel values within the pixel block, to form adjustable integer-pixel values, while the adjusted integer-pixel values correspond to integer-pixel positions, and the formation of predictive eye based on the adjusted values tselopikselnyh.

In accordance with another aspect, video encoder 22 may generate first interpolated prediction data for encoding video data based on a first interpolation filter, generate second interpolated prediction data for encoding video for video data based on a second interpolation filter, choose between first interpolated prediction data and second interpolated prediction data based on the analysis of distortion depending on the transmission speed, encode the video data ie on the basis of selection and encode syntax to indicate the selection. The first interpolation filter may contain a non-tunable interpolation filter, and the second interpolation filter may contain an adaptive interpolation filter, but this disclosure is not necessarily limited to these examples.

In addition, additional interpolation filters can also be used to generate additional interpolated predictive data, which can also be considered in the analysis of distortion depending on the transmission speed. In other words, the technologies of the present disclosure are not limited to generating only the first and second interpolated predictive data based on two interpolation filters, but can be used to generate any plurality of interpolated predictive data based on any number of interpolation filters. It is important that the interpolated predictive analysis data of distortion versus transmission rate is used to identify which filter should be selected.

In one example, the method may comprise generating a plurality of different versions of predictive data for encoding video data based on a plurality of different interpolation filters, selecting from a plurality of different versions of predictive data based on an analysis of distortion depending on a transmission rate, encoding the video data based on the selection, and syntax encoding so that indicate the choice.

The present disclosure also contemplates a technique for encoding filter coefficients. For example, video encoder 22 can identify a set of filter coefficients for interpolating predictive data when encoding video, generate residual values associated with a set of filter coefficients, based on predictive coding of a set of filter coefficients relative to filter coefficients associated with the non-tunable interpolation filter, apply quantization to the residual values and output quantized residual values as part of the encoded bitstream.

Video decoder 28 may take residual values associated with a set of filter coefficients, generate a set of filter coefficients using predictive decoding based on a set of residual values and filter coefficients associated with a non-tunable interpolation filter, and apply a set of filter coefficients to interpolate the predictive data used for the predictive decoding video blocks.

FIG. 2 is a block diagram illustrating an example of a video encoder 50 that can perform filtering techniques in accordance with the present disclosure. Video encoder 50 is one example of a dedicated computer video device, referred to herein as a “coder”. Video encoder 50 may correspond to video encoder 22 of device 20 or a video encoder of another device. Video encoder 50 may perform intra and inter-coding of blocks within the frames of video frames, although intra-coding components are not shown in FIG. 2 for ease of illustration. Intra-coding is based on spatial prediction to reduce or remove spatial redundancy of video in a given video frame. Mutual coding is based on temporal prediction to reduce or remove temporal video redundancy in adjacent frames of a video sequence. The internal mode (I-mode) may be referred to as spatial compression mode, and mutual modes, such as the prediction mode (P-mode) or bidirectional mode (mode B), may be referred to as temporary compression modes. The technologies of this disclosure are used during inter-coding, and therefore, internal coding modules, for example, a spatial prediction module, are not illustrated in FIG. 2 for simplicity and convenience of illustration.

As shown in FIG. 2, video encoder 50 receives the current video block in a video frame to be encoded. In the example of FIG. 2, video encoder 50 includes a prediction module 32, a memory 34, an adder 48, a transform module 38, a quantization module 40, and an entropy encoding module 46. To recover the video block, video encoder 50 also includes an inverse quantization module 42, an inverse transform module 44, and an adder 51. A deblocking filter (not shown) may also be included to filter block boundaries to remove blocking artifacts from the reconstructed video. If desired, the deblocking filter typically should filter the output of the adder 51.

Prediction module 32 may include motion estimation (ME) module 35 and motion compensation (MC) module 37. The filter 37 may be included in the prediction module 32 and may be activated by one or both of the ME module 35 and the MC module 37 to perform interpolation or similar interpolation filtering as part of the motion estimation and / or motion compensation, according to the present disclosure. The filter 37 may actually represent many different filters to simplify the many different types of interpolation and similar interpolation filtering. Thus, the prediction unit 32 may include a plurality of interpolation or similar interpolation filters. During the encoding process, video encoder 50 receives a video block to be encoded (labeled “video block” in FIG. 2) and the prediction unit 32 performs mutual prediction encoding to form a prediction block (labeled “forecast block” in FIG. 2). In particular, the ME module 35 may perform motion estimation to identify the predictive block in the memory 34, and the MC module 37 may perform motion compensation to form the predictive block.

Motion estimation is typically considered the process of generating motion vectors that evaluate motion for video blocks. The motion vector, for example, can indicate the offset of the predictive block within the predictive or reference frame (or other coded unit, for example, a series of consecutive macroblocks) relative to the block to be encoded within the current frame (or other coded unit). The reference frame (or part of the frame) may temporarily be located before or after the video frame (or part of the video frame) to which the current video block belongs. Motion compensation is typically considered the process of sampling or generating a prediction block from memory 34, or possibly interpolating or otherwise generating filtered predictive data based on a motion vector determined by motion estimation.

The ME module 35 selects the corresponding motion vector for the video block to be encoded by comparing the video block with the video blocks of one or more reference frames (for example, a previous and / or subsequent frame). The ME module 35 may perform motion estimation with fractional-pixel accuracy, sometimes referred to as fractional-pixel, partial-pixel, or sub-pixel motion estimation. In fact, the terms fractional-pixel, partial-pixel, or sub-pixel motion estimation can be used interchangeably. In evaluating the fractional-pixel motion, the ME module 35 may select a motion vector that indicates an offset to a location other than the integer-pixel location. Thus, the evaluation of the fractional-pixel motion enables the prediction unit 32 to track the motion with higher accuracy than the integer-pixel locations (or full-pixel) to thereby form a more accurate prediction unit. The fractional-pixel motion estimation may have half-pixel accuracy, quarter-pixel accuracy, one-eighth pixel accuracy, or any greater accuracy. The ME module 35 can activate filter (s) 39 for all required interpolations during the motion estimation process.

To compensate for fractional-pixel motion, the MC module 37 may perform interpolation (sometimes called interpolation filtering) to generate data at sub-pixel resolution (referred to as sub-pixel or fractional-pixel values in this document). MC module 37 may activate filter (s) 39 for this interpolation. Prediction unit 32 may perform interpolation (or similar interpolation filtering of whole pixels) using the techniques described herein.

After the motion vector for the video block to be encoded is selected by the ME module 35, the MC module 37 generates a predictive video block associated with this motion vector. The MC module 37 may select a prediction block from the memory 34 based on the motion vector determined by the MC module 35. In the case of a fractional pixel precision motion vector, the MC module 37 filters the data from the memory 34 to interpolate such data to subpixel resolution, for example, when calling filter (s) 39 for this process. In some cases, the interpolation filtering technology or mode that is used to generate predictive subpixel data may be indicated as one or more interpolation syntax elements for entropy encoding unit 46 to be included in the encoded bitstream. Indeed, some aspects of the present disclosure relate to the use of pixel symmetry and coefficient symmetry to reduce the amount of syntax to be transmitted.

After the prediction unit 32 generates a prediction block, the video encoder 50 forms a residual video block (labeled “residual block” in FIG. 2) by subtracting the prediction block from the encoded source video block. Adder 48 represents the component or components that perform this subtraction operation. The transform unit 38 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, forming a video block containing residual block transform coefficients. Conversion module 38, for example, may perform other transformations defined by the H.264 standard, which are conceptually similar to DCTs. Wavelet transforms, integer transforms, subband transforms, or other types of transforms can also be used. In any case, the transform unit 38 applies the transform to the residual block, forming a block of residual transform coefficients. The conversion may convert residual information from the pixel region to the frequency domain.

Quantization module 40 quantizes the residual transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. After quantization, entropy encoding unit 46 entropy encodes the quantized transform coefficients. For example, entropy encoding unit 46 may perform CAVLC, CABAC, or other entropy encoding technology.

Entropy encoding unit 46 may also encode one or more prediction syntax elements obtained from prediction unit 32 or another component of video encoder 50. One or more prediction syntax elements may include an encoding mode, one or more motion vectors, interpolation technology that is used to in order to generate subpixel data, a set or subset of filter coefficients, or other information associated with the formation of the prediction block. The coefficient prediction and quantization module 41 can predictively encode and quantize the prediction syntax, for example, filter coefficients, according to some aspects of this disclosure. After entropy encoding by entropy encoding unit 46, the encoded video and syntax elements may be transferred to another device or archived for subsequent transmission or retrieval.

The inverse quantization unit 42 and the inverse transform unit 44 apply inverse quantization and inverse transform, respectively, to restore the residual block in the pixel region, for example, for subsequent use as a reference block. The reconstructed residual block (labeled “reconstructed residual block” in FIG. 2) may represent a reconstructed version of the residual block provided in the conversion unit 38. The reconstructed residual block may differ from the residual block formed by the adder 48 due to the loss of detail caused by the quantization and inverse quantization operations. The adder 51 adds the reconstructed residual block to the motion compensation prediction block generated by the prediction unit 32 to form the reconstructed video block for storage in the memory 34. The reconstructed video block can be used by the prediction unit 32 as a reference block, which can be used to then encode the block in a subsequent video frame or subsequent encoded unit.

As described above, the prediction unit 32 may perform motion estimation with fractional-pixel accuracy (or subpixel). When the prediction unit 32 uses a fractional-pixel motion estimate, the prediction unit 32 can generate data at sub-pixel resolution (eg, sub-pixel or fractional-pixel values) using the interpolation operations described in this disclosure. In other words, interpolation operations are used to calculate values at positions between integer-pixel positions. Sub-pixel positions located halfway between integer-pixel positions may be referred to as half-pixel positions (half a pixel), sub-pixel positions located halfway between the integer-pixel position and half-pixel positions may be referred to as quarter-pixel positions (one fourth of a pixel), sub-pixel positions located half the distance between the integer-pixel position (or half-pixel position) and the quarter-pixel position are referred to as one-eighth pi positions sat (one-eighth), etc.

3 is a conceptual diagram illustrating integer-pixel positions (or a full pixel) associated with predictive data and sub-pixel positions (or a fractional pixel) associated with interpolated predictive data. In the conceptual illustration of FIG. 3, various rectangles represent pixel and subpixel position locations in a pixel and subpixel within a frame or block of a frame. Uppercase letters (in rectangles with solid lines) represent integer-pixel locations, while lowercase letters (in rectangles with dashed lines) represent subpixel locations. In particular, pixel locations A1-A6, B1-B6, C1-C6, D1-D6, E1-E6 and F1-F6 represent a 6 × 6 matrix of integer-pixel locations within a frame, a series of consecutive macroblocks, or another coded unit. The sub-pixel locations “a” to “o” represent the fifteen sub-pixel locations associated with the whole pixel C3, for example, between the integer pixel locations C3, C4, D3 and D4. Similar subpixel locations may exist for each integer pixel location. The sub-pixel locations "a" - "o" represent each half-pixel and quarter-pixel pixel location associated with the whole pixel C3.

Integer-pixel locations may be associated with a physical sensor element, such as a photodiode, when video data is initially generated. A photodiode can measure the luminous intensity at the light source at the location of the sensor and associate the luminous intensity value of the pixel with an integer-pixel location. On the other hand, each integer-pixel location may have an associated set of fifteen sub-pixel locations (or possibly more). The number of sub-pixel locations associated with integer-pixel locations may depend on the accuracy required. In the example illustrated in FIG. 3, the required accuracy is quarter-pixel accuracy when each of the integer-pixel locations corresponds to fifteen different sub-pixel positions. A larger or smaller number of sub-pixel positions may be associated with each integer-pixel location based on the required accuracy. For half-pixel accuracy, for example, each integer-pixel location may correspond to three sub-pixel positions. As another example, each of the integer-pixel locations may correspond to sixty-three sub-pixel positions for an accuracy of one-eighth of a pixel. Each pixel location may specify one or more pixel values, for example, one or more values of a luminance signal and a color signal.

Y can represent a luminance signal, and Cb and Cr can represent two different values of the color signal of the three-dimensional color space YCbCr. Each pixel location can actually specify three pixel values for a three-dimensional color space. However, the techniques of the present disclosure may be referred to as single dimensional prediction for simplicity. To the extent that technologies are described with respect to pixel values in one dimension, similar technologies can be extended to other dimensions.

In the example of FIG. 3, sub-pixel locations associated with the whole pixel “C3” are illustrated for quarter-pixel accuracy. Fifteen subpixel positions associated with pixel C3 are labeled as "a", "b", "c", "d", "e", "f", "g", "h", "i", "j" , "k", "l", "m", "n" and "o". Most other fractional locations associated with other integer-pixel locations are not shown for simplicity (other than those used to form one or more of 15 different fractional locations associated with the location of pixel C3, as described in more detail below). The sub-pixel locations "b", "h" and "j" may be referred to as half-pixel locations, and the sub-pixel locations "a", "c", "d", "e", "f", "g", "i", "k", "l", "m" and "o" may be referred to as quarter-pixel locations.

Prediction unit 32 of video encoder 40 may determine pixel values for sub-pixel locations “a” to “o” using interpolation filtering by MC module 37. According to ITU-T H.264, for example, prediction unit 32 may determine pixel values for half-pixel locations using a 6-tap interpolation filter, such as a Wiener filter. In the case of the H.264 standard, the filter coefficients for a 6-tap interpolation filter are typically [1, -5, 20, 20, -5, 1], although other coefficients may be used. Prediction unit 32 may apply an interpolation filter first in the horizontal direction and then in the vertical direction, or vice versa. For half-pixel positions "b" and "h", each tap can correspond to an integer-pixel position in the horizontal and vertical directions, respectively. In particular, for the half-pixel position “b”, the taps of the 6-tap filter correspond to C1, C2, C3, C4, C5 and C6. Similarly, for the half-pixel position “h”, the taps of the 6-tap filter correspond to A3, B3, C3, D3, E3 and F3. For example, pixel values for sub-pixel positions “b” and “h” can be calculated using equations (1) and (2):

b = ((C1-5 * C2 + 20 * C3 + 20 * C4-5 * C5 + C6) +16) / 32 (one) h = ((A3-5 * B3 + 20 * C3 + 20 * D3-5 * E3 + F3) +16) / 32 (2)

For the half-pixel position "j", the taps of the 6-tap filter correspond to themselves for interpolation horizontally between positions C1-C6 and D1-D6 or vertically between positions A3-F3 and A4-F4. The half-pixel location “j” can be calculated with a 6-tap filter that uses previously interpolated pixel values of half-pixel positions, for example, in accordance with one of equations (3) or (4):

j = ((aa-5 * bb + 20 * b + 20 * hh-5 * ii + jj) +16) / 32 (3) j = ((cc-5 * dd + 20 * h + 20 * ee-5 * ff + gg) +16) / 32 (four)

where (as illustrated in FIG. 3) aa corresponds to interpolation between A3, and A4, bb corresponds to interpolation between B3 and B4, b corresponds to interpolation between C3 and C4, hh corresponds to interpolation between D3 and D4, ii corresponds to interpolation between E3 and E4, and jj corresponds to the interpolation between F3 and F4. In equation 4, cc corresponds to interpolation between C1 and D1, dd corresponds to interpolation between C2 and D2, h corresponds to interpolation between C3 and D3, ee corresponds to interpolation between C4 and D4, ff corresponds to interpolation between C5 and D5, and gg corresponds to interpolation between C6 and D6.

According to the H.264 standard, the prediction unit 32 can determine pixel values at quarter-pixel locations "a", "c", "d", "e", "f", "g", "i", "k", "l", "m", "n" and "o" using a bilinear interpolation filter and the pixel values of the surrounding integer-pixel and half-pixel locations. For example, prediction unit 32 may determine the pixel value associated with subpixel position "a" using pixel values C3 and "b", determine the pixel value associated with subpixel position "c" using pixel values "b" and C4, etc.

The actual filter that the MC module 37 applies to generate interpolated data at sub-pixel locations can have a wide range of implementations. As one example, prediction module 32 may use adaptive interpolation filtering (AIF), as described below, to specify interpolated values. The ITU-T SG16 / Q.6 / VCEG (video coding expert group) is investigating coding technologies that offer higher coding efficiency than H.264 and, in particular, AIF. AIF offers great coding gain over the interpolation filtering used in the H.264 standard, in particular for high-resolution video sequences (e.g. 720i / p or 1080i / p). In AIF, an interpolation filter for each sub-pixel position is analytically calculated for each video frame by minimizing the energy of prediction errors. This helps to take into account the imposition of spectra, quantization errors and motion estimates, camera noise or other artifact contained in the source and reference video frames. Analytically extracted adaptive filtering coefficients for each frame are then predicted, quantized, encoded, and sent in the video bit stream. Some of the techniques of this disclosure may work within the framework of the AIF scheme, as well as many other interpolation schemes.

There are many different types of AIF schemes in accordance with aspects of the present disclosure. For example, the first scheme is a two-dimensional inseparable AIF (NS-AIF), the second is a separable AIF (S-AIF), and the third is an AIF with directional filters (D-AIF). Although each of these AIF schemes uses different interpolation technologies and support, all three AIF schemes can use similar analytical processes to extract filtering coefficients, which is explained below using an inseparable AIF as an example.

, где i, j=0,..., 5, а SP представляет одну из 15 субпикселных позиций ("a"-"o"), показанных на Фиг.3. Следует отметить, что 6 из 15 субпикселных позиций, т.е. "a", "b", "c", "d", "h" и "l", являются одномерными (1D) субпикселными позициями, и модуль 32 прогнозирования может использовать 6-отводный интерполяционный фильтр, чтобы интерполировать такие данные. Субпикселные позиции "a", "b", "c", "d", "h" и "l" являются одномерными в том смысле, что они находятся в горизонтальной или вертикальной строке между двумя целопикселными позициями. Кроме того, допустим, что прогнозирующие пикселы в целопикселных позициях (A1-F6 на Фиг.3) в опорном кадре принимают пикселные значения P _i,j, где i, j = 0…5. Таким образом, A1 принимает значение Р _0,0, …, A6 принимает значение Р _5,0, …, F1 принимает значение Р _5,0, …, и F6 принимает значение Р _5,5. Затем, интерполированное значение p ^SP в субпикселной позиции SP, SP∈{a,…,0}, может вычисляться посредством модуля 32 прогнозирования с использованием следующего уравнения:Suppose a two-dimensional inseparable 6 × 6 filter has coefficients

, where i, j = 0, ..., 5, and SP represents one of the 15 subpixel positions ("a" - "o") shown in FIG. 3. It should be noted that 6 out of 15 subpixel positions, i.e. "a", "b", "c", "d", "h" and "l" are one-dimensional (1D) sub-pixel positions, and prediction unit 32 may use a 6-tap interpolation filter to interpolate such data. The subpixel positions "a", "b", "c", "d", "h" and "l" are one-dimensional in the sense that they are in a horizontal or vertical row between two integer-pixel positions. In addition, suppose that the predictive pixels in integer-pixel positions (A1-F6 in FIG. 3) in the reference frame take pixel values P _{i, j} , where i, j = 0 ... 5. Thus, A1 takes a value of P _0.0 , ..., A6 takes a value of P _5.0 , ..., F1 takes a value of P _5.0 , ..., and F6 takes a value of P _5.5 . Then, the interpolated value p ^SP at the sub-pixel position SP, SP∈ {a, ..., 0}, can be calculated by the prediction unit 32 using the following equation:

Let S _{x, y} be the pixel value in the current video frame at position (x, y) .

является соответствующей позицией пиксела в опорных кадрах. Например, в случае 6-отводного фильтра, FO=6/2-1=2. Для каждой субпикселной позиции SP энергия (e ^SP)2  ошибок прогнозирования между фактическим пикселным значением в текущем кадре и интерполированным значением может накапливаться посредством модуля 32 прогнозирования для всех пикселов, которые имеют точность вектора движения, соответствующую субпикселной позиции SP. Энергия ошибок прогнозирования может вычисляться посредством модуля 32 прогнозирования с использованием следующего уравнения:Where(mvx, mvy) is the motion vector, (mvx, mvy) is the integer component of the motion vector, and FO is the filter offset. Value

 is the corresponding pixel position in the reference frames. For example, in the case of a 6-tap filter, FO = 6 / 2-1 = 2. For each subpixel position SP energy(e ^{SP) 2} prediction errors between the actual pixel value in the current frame and the interpolated value can be accumulated by the prediction unit 32 for all pixels that have a motion vector accuracy corresponding to the sub-pixel position SP. The energy of the prediction errors can be calculated by the prediction unit 32 using the following equation:

. Число уравнений в этом случае равно числу коэффициентов фильтрации, используемых для текущей субпикселной позиции SP. Для каждой двумерной (двумерной) субпикселной позиции "e", "f", "g", "i", "j", "k", "m", "n" и "o", модуль 32 прогнозирования может использовать 6×6-отводный двумерный интерполяционный фильтр. Субпикселные позиции "e", "f", "g", "i", "j", "k", "m", "n" и o" являются двумерными в том смысле, что они не находятся в вертикальной строке или горизонтальной строке между двумя целопикселными позициями. В этом случае, система из тридцати шести уравнений с тридцатью шестью неизвестными может разрешаться посредством MC-модуля 37. Оставшиеся одномерные субпикселные позиции "a", "b", "c", "d", "h", и "l" могут требовать только одномерного интерполяционного фильтра (к примеру, одномерного 6-отводного фильтра). В случае одномерного 6-отводного фильтра система из шести уравнений может разрешаться посредством MC-модуля 37.For each of the subpixel positions ao, the MC module 37 can establish a separate system of equations by calculating the derivative (e ^{SP) 2} relative to the filter coefficients

. The number of equations in this case is equal to the number of filtering coefficients used for the current subpixel position SP . For each two-dimensional (two-dimensional) subpixel position “e”, “f”, “g”, “i”, “j”, “k”, “m”, “n” and “o”, the prediction unit 32 may use 6 × 6-tap two-dimensional interpolation filter. The subpixel positions "e", "f", "g", "i", "j", "k", "m", "n" and o "are two-dimensional in the sense that they are not in a vertical row or horizontal line between two integer-pixel positions. In this case, a system of thirty-six equations with thirty-six unknowns can be resolved by means of the MC module 37. The remaining one-dimensional subpixel positions "a", "b", "c", "d", "h "and" l "may require only a one-dimensional interpolation filter (for example, a one-dimensional 6-tap filter). In the case of a one-dimensional 6-tap filter, a system of six cheers neny may be solved by MC-37 module.

Filter (s) 39 may represent a single filter or a set of many different filters that can be used by the MC module 37 to generate predictive data.

Thus, an exemplary process for extracting and applying AIF filters can have the following steps that can be performed by the prediction module 32:

1. Evaluation of motion vectors (mvx, mvy) for each video block to be encoded. During motion estimation, a non-tunable interpolation filter can be applied (for example, the H.264 / AVC interpolation process).

адаптивной фильтрации для каждой субпикселной позиции SP независимо посредством минимизации энергии ошибок прогнозирования, как указано в двух вышеприведенных уравнениях энергии прогнозирования.2. Using these motion vectors, the accumulation of energy of prediction errors for each sub-pixel position SP over the current video frame. Then the calculation of the coefficients

adaptive filtering for each sub-pixel position SP independently by minimizing the energy of prediction errors, as indicated in the two prediction energy equations above.

3. Assessment of new motion vectors. During the motion estimation process, adaptive interpolation filters calculated in step 2 can be applied. Using adaptive interpolation filters, the motion estimation errors caused by superimposing spectra, camera noise, etc., are reduced and better motion prediction is achieved.

Different AIF schemes may use an analytical process identical to the process defined above. The differences between the various schemes are mainly in the number of unique filtering coefficients used, whether the interpolation filters are separable or inseparable, and in the filter support used (i.e., integer-pixel positions used to interpolate at least part of the sub-pixel positions ) In each of these schemes, certain symmetry restrictions for AIF filters can be imposed to reduce the number of filter coefficients that must be encoded and transmitted in the video bit stream.

For the NS-AIF, for example, the MC module 37 of the prediction module 32 can interpolate the one-dimensional sub-pixel positions "a", "b", "c", "d", "h" and "l" using a one-dimensional 6-tap interpolation filter (also called a 6-position filter, since each tap corresponds to an integer-pixel position), which requires six coefficients. The six coefficients of the 6-position filters used to interpolate the one-dimensional subpixel correspond to one of the integer-pixel positions illustrated in FIG. 3. For example, for the subpixel positions "a", "b" and "c", the six integer pixel positions corresponding to the coefficients are C1, C2, C3, C4, C5 and C6 and for the subpixel positions "d", "h" and "l ", six integer pixel positions corresponding to the coefficients are A3, B3, C3, D3, E3 and F3. These integer-pixel positions represent "filter support" for the interpolation filter.

The prediction unit 32 may interpolate the two-dimensional sub-pixel positions “e”, “f”, “g”, “i”, “j”, “k”, “m”, “n” and o “using a 6 × 6 two-dimensional interpolation filter which requires thirty-six filter coefficients. The thirty-six coefficients of a 6 × 6 two-dimensional interpolation filter used to interpolate the two-dimensional subpixel correspond to integer-pixel positions A1-A6, B1-B6, C1-C6, D1-D6, E1-E6 and F1-F6: These integer-pixel positions represent “filter support” for the interpolation filter. the value is not superimposed, for example, there is no symmetry by coefficients or pixels, video encoder 50 can encode and transmit nine sets of thirty-six coefficients for two-dimensional subpixel positions and six sets of six coefficients for one-dimensional subpixel positions, a total of 360 coefficients. coefficients in the video bit stream can lead to a costly additional number of bits. A high additional number of bits can, in turn, increase the bit rate for a given level of distortion, which is undesirable.

To reduce the additional number of bits associated with the transmission of filter coefficients, certain symmetry constraints may be introduced for interpolation filters to reduce the number of unique filter coefficients transmitted to decoder 28. Two types of symmetry, i.e. pixel symmetry and coefficient symmetry can be superimposed individually or in combination. Pixel symmetry forces an identical set of filtering coefficients (and mirrored, inverted, and / or rotated versions of the set) for different subpixel positions. Pixel symmetry may also be referred to as subpixel symmetry when such pixel symmetry is applied to filter coefficients associated with two or more subpixel locations. The symmetry of the coefficients, on the other hand, forces this interpolation filter to be symmetric in a certain direction (for example, horizontal direction, vertical direction, or both) for different filter support positions relative to other filter support positions for data subpixel values to be interpolated.

является набором коэффициентов фильтрации для субпикселной позиции "a", в таком случае набором коэффициентов фильтрации для субпикселной позиции "c" является

, т.е. идентичный коэффициент в обратном порядке или горизонтально перевернутый. Таким образом, значение P ^a прогнозирующего пиксела в субпикселной позиции "a" и значение p ^c прогнозирующего пиксела в субпикселной позиции "c" могут вычисляться с использованием (8) и (9), соответственно.Pixel symmetry can be used in NS-AIF. Referring again to FIG. 3, let

is the set of filtering coefficients for the subpixel position "a", in which case the set of filtering coefficients for the subpixel position "c" is

, i.e. identical coefficient in the reverse order or horizontally inverted. Thus, the predictive pixel value P ^a at the sub-pixel position “a” and the predictive pixel value p ^c at the sub-pixel position “c” can be calculated using (8) and (9), respectively.

The sub-pixel positions "d" and "l" can be interpolated using a set of interpolation filtering coefficients identical to the set of coefficients of the sub-pixel positions "a" and "c", respectively. In fact, the subpixel positions "a" and "c" may have pixel symmetry relative to the subpixel positions "d" and "l".

One aspect of the present disclosure is to actually remove pixel symmetry between subpixel positions "a" and "c" with respect to subpixel positions "d" and "l". In addition, pixel symmetry can be eliminated for the sub-pixel position "f" relative to the sub-pixel position "i". In addition, pixel symmetry can be eliminated for the sub-pixel position "k" relative to the sub-pixel position "n". In such cases, the diagonal correlation may be low, making the overlapping of diagonal symmetry ineffective or ineffective in these cases.

является набором двумерных коэффициентов фильтрации 6×6 для субпикселной позиции "e". В таком случае набором коэффициентов фильтрации для позиции "g" является

(горизонтально перевернутая версия). Аналогично, набором коэффициентов фильтрации для субпикселной позиции "m" является

(вертикально перевернутая версия), а набором коэффициентов фильтрации для субпикселной позиции "o" является

(сначала горизонтально перевернут, а затем вертикально перевернут). По сути, субпикселные позиции "e", "g", "m" и "o" имеют симметрию по пикселам. Симметрия между субпикселными позициями "b" и "h" и между позициями "f", "i", "k" и n" накладывается аналогичным способом, как в примерах, приведенных выше. При таком ограничении по симметрии по пикселам, остается только пять групп уникальных наборов фильтров, первый набор коэффициентов фильтрации для группы субпикселов "a", "c", "d" и "l", второй набор коэффициентов фильтрации для группы субпикселов "b" и "h", третий набор коэффициентов фильтрации для группы субпикселов "e", "g", "m" и "o", четвертый набор коэффициентов фильтрации для группы "f", "i", "k" и n", и пятый набор коэффициентов фильтрации для группы субпикселов "j".As another example, let

is a set of two-dimensional filter coefficients of 6 × 6 for the sub-pixel position "e". In this case, the set of filtering coefficients for position "g" is

(horizontally flipped version). Similarly, the set of filter coefficients for the sub-pixel position "m" is

(vertically flipped version), and the set of filtering coefficients for the sub-pixel position “o” is

(first horizontally flipped and then vertically flipped). In fact, the sub-pixel positions "e", "g", "m" and "o" have pixel symmetry. The symmetry between the sub-pixel positions "b" and "h" and between the positions "f", "i", "k" and n "is superimposed in the same way as in the examples above. With this restriction on symmetry by pixels, there are only five groups of unique filter sets, the first set of filter coefficients for the group of subpixels "a", "c", "d" and "l", the second set of filter coefficients for the group of subpixels "b" and "h", the third set of filter coefficients for the group of subpixels "e", "g", "m" and "o", the fourth set of filter coefficients for the group "f", "i", "k" and n ", and the fifth on OR the filter coefficients for the group of sub-pixels "j".

In addition, NS-AIF may impose a symmetry constraint on coefficients on some of these filter sets individually or in combination with the sub-pixel symmetry described above. In some cases, diagonal symmetry in coefficients can be deliberately excluded, for example, if all restrictions on diagonal symmetry in coefficients are eliminated. In one case, symmetry in the coefficients is not superimposed on the filter for the first group of subpixel positions, which includes the subpixel locations "a", "c", "d" and "l". However, the second group of sub-pixel positions, which includes the coefficient "b" and "h", may have symmetry in the coefficients in accordance with equation (10).

Similarly, the coefficients for interpolation filters for the third, fourth and fifth groups of subpixel positions, which include the subpixel positions "e", "f" and j ", respectively, can be symmetrical in coefficients as specified in equations (11), ( 12) and (13), respectively.

Under such restrictions on pixel symmetry and symmetry on the coefficients described above, the number of unique filtering coefficients can be reduced from 360 (there is no symmetry restriction) to 6 ( a ) +3 ( b ) +21 ( e ) +18 ( f ) + 6 ( j ) = 54 coefficients, i.e. 6 coefficients for a group including a subpixel position "a", 3 coefficients for a group including a subpixel position "b", 21 coefficients for a group including a subpixel position "e", 18 coefficients for a group including a sub-pixel position of "f", and 6 coefficients for a group including a sub-pixel position of "j". Thus, in the NS-AIF, video encoder 22 can encode and transmit fifty-four coefficients instead of 360, as in the absence of symmetry constraints. As described above, these fifty-four coefficients can be analytically resolved based on the above equation (7). Video encoder 50 can then predict, quantize, encode (for example, using the exponential Golomb code with a sign) and send coefficients in the bitstream. Additional information regarding predictive coding of filter coefficients is explained in more detail below. Coefficient prediction and quantization module 41 may be used to predictively encode and quantize filter coefficients in accordance with this disclosure.

In another AIF scheme, i.e. S-AIF, prediction module 32 may use separable interpolation filters in the horizontal direction and in the vertical direction instead of the inseparable interpolation filters used in the NS-AIF. For one-dimensional subpixel positions, the prediction module 32 (for example, the MC module 37 of the prediction module 32) applies only horizontal directional filters or only vertical directional filters depending on the subpixel location. In one example, horizontal directional filters and vertical directional filters contain 6-position (or 6-tap) filters. Prediction module 32 applies horizontal directional filters for sub-pixel positions “a”, “b” and “c” with integer-pixel positions C1, C2, C3, C4, C5 and C6 (see FIG. 3) as a filter support and applies vertical directional filters filters for sub-pixel positions "d", "h" and "l" with integer-pixel positions A3, B3, C3, D3, E3 and F3 (see Figure 3) as filter support. For the remaining sub-pixel positions, i.e. of two-dimensional sub-pixel positions, the prediction unit 32 first applies horizontal filtering, after which vertical filtering, or vertical filtering, and then horizontal filtering, is performed. On the other hand, the filtering coefficients used for separable horizontal and vertical filters can be calculated in accordance with the above equation (7). Without symmetry, S-AIF can reduce the number of coefficients to be encoded and sent to 144 coefficients from 360 coefficients, as is the case for NS-AIF without symmetry.

и

, соответственно. Для каждой субпикселной позиции, которая имеет вертикальное перемещение, симметрия по субпикселам является аналогичной NS-AIF. Таким образом, коэффициенты для субпикселных позиций "d" и "l", коэффициенты для субпикселных позиций "e" и "m", коэффициенты для субпикселных позиций "f" и "n", и коэффициенты для субпикселных позиций "g" и "o" являются вертикально перевернутыми версиями друг друга, соответственно. Например, если коэффициенты вертикального фильтра для субпикселной позиции "d" - это

, то коэффициенты вертикального фильтра для субпикселной позиции "l" - это

. Таким образом, симметрия по субпикселам, используемая только в S-AIF, имеет вертикальную симметрию по субпикселам в одной размерности. S-AIF, тем не менее, не имеет горизонтальной симметрии по субпикселам. Симметрия по субпикселам в S-AIF может сокращать число коэффициентов, которые должны кодироваться и отправляться, со 144 коэффициентов до 102 коэффициентов.S-AIF may additionally impose one or more symmetry constraints for at least some of the interpolation filters. Unlike NS-AIF subpixel symmetry, with S-AIF, the filter sets used for the subpixel positions “a” and “c” are not symmetrical, i.e. horizontally flipped versions of each other. Instead, the coefficients of the filter sets used for the sub-pixel positions "a" and "c" are extracted independently from each other as

and

, respectively. For each subpixel position that has vertical movement, subpixel symmetry is similar to NS-AIF. Thus, the coefficients for the sub-pixel positions "d" and "l", the coefficients for the sub-pixel positions "e" and "m", the coefficients for the sub-pixel positions "f" and "n", and the coefficients for the sub-pixel positions "g" and "o "are vertically inverted versions of each other, respectively. For example, if the vertical filter coefficients for the subpixel position "d" are

, then the vertical filter coefficients for the subpixel position "l" is

. Thus, sub-pixel symmetry, used only in S-AIF, has vertical sub-pixel symmetry in one dimension. S-AIF, however, does not have horizontal subpixel symmetry. Sub-pixel symmetry in S-AIF can reduce the number of coefficients to be encoded and sent from 144 coefficients to 102 coefficients.

S-AIF, like NS-AIF, can also use coefficient symmetry to further reduce the number of coefficients to be encoded and sent by video encoder 50. The coefficient symmetry used in S-AIF can be identical to the symmetry in NS-AIF for except that only one-dimensional (horizontal or vertical) 6-tap filters are provided in S-AIF. In other words, there are no 6x6 two-dimensional filters. In S-AIF, for each subpixel position of SP, SP∈ {b, h, I, j, k}, the filter coefficients may have symmetry, as indicated in equation (14).

Therefore, both with subpixel symmetry and coefficient symmetry, S-AIF filters can have eleven sets of unique one-dimensional 6-tap filters, five of which are symmetric filters, each of which has three unique coefficients. The remaining six filters can have six unique coefficients. In this case, video encoder 22 can quantize, predict, encode, and send a total of fifty-one unique filter coefficients in the video bit stream.

In another AIF scheme, D-AIF, the prediction unit 32 uses directional filters for two-dimensional subpixel positions "e", "f", "g", "i", "j", "k", "m", "n" and "o". Directional filters may contain six-tap filters, and the coefficients may be symmetric so that only a subset of the coefficients should be transmitted. As described herein, filter support means pixel positions from a reference frame that is used in interpolating a sub-pixel position. The MC module 37 can calculate the sub-pixel positions “e” and “o” using a 6-position (or 6-tap) diagonal interpolation filter that has integer-pixel positions A1, B2, C3, D4, E5 and F6 as filter support. Prediction unit 32 can calculate the sub-pixel positions “g” and “m” using a 6-position diagonal interpolation filter, which has integer-pixel positions A6, B5, C4, D3, E2 and F1 as filter support. Prediction unit 32 can calculate sub-pixel positions “f”, “i”, “j”, “k” and “n” using a 12-position diagonal interpolation filter that has integer-pixel positions A1, B2, C3, D4, E5, F6 , A6, B5, C4, D3, E2 and F1 as filter support.

D-AIF can use the symmetry of subpixels and coefficients identical to the symmetry of subpixels and coefficients described above with respect to NS-AIF. In terms of subpixel symmetry, there are five unique sets of filter coefficients for interpolating fifteen subpixel positions, with the subpixel positions “a”, “c”, “d” and “l” sharing identical filtering coefficients (for example, mirror, inverted and / or rotated versions), sub-pixel positions "e", "g", "m" and "o" share identical filtering coefficients (for example, mirrored, inverted and / or rotated versions), sub-pixel positions "b" and "h" share identical coefficients filtering (where "b" is a horizontal filter and "h" is a vertical filter), the subpixel positions "f", "i", "k" and n "share the same filtering coefficients (for example, mirror, inverted and / or rotated versions), and the subpixel position "j" has its own set of filter coefficients.

From the point of view of symmetry in the coefficients, the filtering coefficients for the first group of subpixels, including "a", "c", "d" and "l", use a one-dimensional 6-position filter having six unique coefficients (i.e., the first group is asymmetric), the filter coefficients for the second group of subpixels "b" and "h" use a one-dimensional 6-position filter having three unique coefficients (ie, b is symmetric), and a third set of filter coefficients for the group of subpixels "e", "g", "m" and "o" use a directional 6-position This filter having six unique coefficients. The fourth set of filtering coefficients for group "f", "i", "k" and n "uses a 12-tap filter with 6 unique coefficients, and the fifth set of filtering coefficients for group" j "having three unique coefficients. The total number of unique the coefficients in the D-AIF scheme is 6 (a) +3 (b) +6 (e) +6 (f) +3 (j) = 24. These filtering coefficients can be predicted, quantized, encoded and sent in the video bit stream.

The MC module 37 of the prediction module 32 may use one or more of the interpolation techniques described herein to increase the accuracy of interpolation of sub-pixel positions and / or to reduce the number of interpolation filter coefficients to be encoded and transmitted. The following describes in more detail the applicable filter support for twelve pixels with a radial shape that can be used for interpolation, technologies that use coefficient symmetry and pixel symmetry to reduce the amount of data needed to be sent between the encoder and decoder to configure filter support for interpolation , technologies for removing some aspects of symmetry relative to traditional technologies and technologies for filtering data in integer-pixel locations in a manner that which is similar to subpixel interpolation. Displacement technologies are also described. In addition, technologies for choosing between interpolation filters and technologies for predictively coding filter coefficients are also described.

4 is a conceptual diagram illustrating twelve pixel filter support with respect to nine sub-pixel locations. According to this disclosure, the MC module 37 may calculate sub-pixel values for the sub-pixel positions "e", "f", "g", "i", "j", "k", "m", "n" and o "based on the hatched integer-pixel values. In this case, the calculation of the sub-pixel values includes the use of an interpolation filter that defines a two-dimensional array of filter support positions corresponding to a set of twelve (or possibly more) integer-pixel positions that surround the sub-pixel positions, as illustrated in Figure 4. Filter support positions are shown with a dash. with uppercase letters and capital letters, and subpixel positions are shown as rectangles with dashed lines and lowercase letters.The MC module 37 can interpolate pixels in this way for each pixel of the video block and can generate a prediction block based on the interpolated subpixel value. Support for the 12-position filter has is radial in shape and has a similar frequency response, like the 36-position filter support used in NS-AIF, but with much less complexity of interpolation.

From figure 3 it follows that the subpixel values may contain fifteen quarter-pixel and half-pixel values. An interpolation filter, which defines a two-dimensional array of filter support positions, is used to specify nine of the fifteen quarter-pixel and half-pixel values, as shown in FIG. 4.

The set of twelve or more integer-pixel positions that surround the sub-pixel positions is shown with shading in FIG. 4 and may include four integer-pixel positions C3, C4, D3 and D4 surrounding nine quarter-pixel and half-pixel values, and eight integer-pixel positions B3 , B4, C2, C5, D2, D5, E3 and E4 surrounding four integer-pixel positions. Each of the filter support positions in the two-dimensional array of filter support positions shown in FIG. 4 is within two integer-pixel positions relative to the sub-pixel positions. Such filtering may be part of motion compensation during the video encoding process by the video encoder 50 (FIG. 2) or during the video decoding process by the video decoder 60 (FIG. 16). During the video encoding process, the prediction unit 32 sends a plurality of sets of coefficient values to the entropy encoding unit 46 as part of the prediction syntax elements. As shown in FIG. 4, the set of twelve or more integer-pixel positions that surround the sub-pixel positions is radial in shape and may contain the set of spatially nearest integer-pixel positions to nine quarter-pixel and half-pixel values specified by an interpolation filter (e.g., a filter ( s) 39 in FIG. 2).

Entropy encoding unit 46 may encode sets of coefficient values and output the coefficient values of the interpolation filter as part of the encoded bitstream, which can then be transferred to another device. Each of a plurality of sets of coefficient values defined with respect to each different sub-pixel location shown in FIG. 4 may define different weightings for twelve or more integer-pixel positions. Pixel symmetry can exist in such a way that M sets of coefficient values give N different weightings for twelve or more integer-pixel positions, with M and N being positive integers, and N is greater than M. In other words, pixel symmetry can allow some of pixel locations based on mirror, inverted, or rotated weightings of other pixel locations so that coefficients should not be encoded for locations that are pixel symmetric with other ubpikselnymi locations.

5 is a conceptual diagram illustrating six pixel horizontal filter support with respect to three horizontal subpixel locations and six pixel vertical filter support with respect to three vertical subpixel locations. Filter support locations are shaded, with uppercase letters, and sub-pixel locations are not shaded, with lowercase letters and dashed lines. The MC module 37 can calculate subpixel values for locations "a", "b" and "c" by applying a linear interpolation filter that defines a one-dimensional matrix of filter support positions C1-C6. The MC module 37 can also calculate subpixel values for locations "d", "h" and "l" by applying a linear interpolation filter that defines a one-dimensional matrix of filter support positions A3-F3. Thus, linear interpolation filters can be used for six of the fifteen quarter-pixel and half-pixel values shown in FIG. 3.

6 is a conceptual diagram illustrating a “five pixel by five pixel” filter support for filtering an integer pixel location. In this case, similar interpolation filtering can be applied to integer-pixel locations to form adjustable integer-pixel values. 6, for example, the integer-pixel value at position C3 can be filtered based on a 5 × 5 matrix of pixels A1-A5, B1-B5, C1-C5, D1-D5 and E1-E5. Other filter supports may be specified for integer filtering. In this case, if the ME module 35 identifies a block of integer pixels, the MC module 37 may filter each pixel based on the 5 × 5 pixel matrix shown in FIG. 6 (or other filter support) to form filtered integer pixel values. Thus, a similar interpolation filtering can be applied to whole pixels, which allows you to improve the coding at time instants of lighting changes or exits from darkening or darkening of the scene. In addition, integer filtering allows you to remove noise and simplify sharpening frames with images and helps to improve the encoding of the exact movement of an object between consecutive video frames, in particular when symmetry does not overlap the filter coefficients. Full pixel filtering can also be extremely useful for improving quality and / or compression for video sequences that have focus changes.

Thus, video encoder 22 and / or video decoder 28 can use similar interpolation filtering with respect to integer-pixel positions to form adjustable integer-pixel values. Video encoder 22 or video decoder 28 may, for example, receive pixel blocks, wherein pixel blocks include integer-pixel values corresponding to integer-pixel positions within the pixel block. Video encoder 22 or video decoder 28 can filter integer-pixel values based on other integer-pixel values within pixel blocks to form adjustable integer-pixel values, wherein adjusted integer-pixel values correspond to integer-pixel positions, and generate a predictive block based on adjusted integer-pixel values. The prediction block can then be used to encode or decode the video block, depending on whether the process is being performed during encoding or decoding.

If the technology is performed during the encoding process, video encoder 22 may form a residual video block as the difference between the current encoded video block and the prediction block. Video encoder 22 may also encode one set of filtering coefficients per coded unit for integer filtering, or possibly encode two or more sets of filtering coefficients per coded unit for integer filtering at various locations within the coded unit. In the example of FIG. 6, a filter support for integer filtering can be defined by a two-dimensional filter, for example, a 5 × 5 offset filter. In addition, coefficient symmetry may exist between at least some of the coefficients used for filtering, so that only a subset of 25 different coefficients is required to determine all 25 coefficients for the filter. Many other filter sizes and shapes can be used in accordance with integer-pixel filtering.

If integer-pixel filtering is performed during the decoding process, the technology may comprise generating the reconstructed video block as the sum of the residual video block associated with the current decoded video block and the prediction block. In this case, the decoding device (for example, the target device 16) may receive one set of filtering coefficients per coded unit for integer filtering, or may receive two or more sets of filtering coefficients per coded unit for integer filtering at various locations within the coded units. Each coded unit may have one or more sets of integer-pixel filtering coefficients, as well as sets of sub-pixel filtering coefficients, for example, eight sets that can be used to form all fifteen sets for quarter-pixel and half-pixel positions.

On the other hand, similar interpolation filtering of integer-pixel values allows you to improve compression specifically during changes in lighting, exits from darkening or darkening the scene, allows you to remove noise and simplify sharpening frames with images, and helps to improve the encoding of the exact movement of an object between consecutive video frames, in particular when symmetry is not superimposed on the filter coefficients. In addition, it has also been found that integer-pixel filtering is extremely useful, for example, in order to improve compression in video sequences that have focus changes.

Traditionally, similar interpolation filtering is not performed for integer-pixel positions in AIF. Instead, the corresponding integer-pixel position in the reference frame is used simply "as is". However, in accordance with the present disclosure, a filter can be used for integer-pixel locations in a manner that is similar to interpolations. Interpolation filtering for integer-pixel positions is technically an erroneous name, since pixel values of integer-pixel positions already exist. Thus, filtering integer-pixel positions is referred to herein as similar interpolation filtering and can be considered as simple filtering of integer-pixel values to form new adjusted pixel values.

Unlike interpolation filtering for sub-pixel positions, where the goal is to obtain non-existent values in the reference frame, pixels at integer positions in the reference frame already exist. The application of filtering to integer-pixel positions in the reference frame offers several advantages. For example, when integer-pixel positions contain an offset, filtering integer-pixel positions can provide better capture of lighting changes between frames. Additionally, filtering for integer-pixel positions allows integer-pixel positions to capture special effects such as dimming and dimming. Filtering for integer-pixel positions also allows you to remove certain noise when transmitting signals (for example, camera noise) and / or perform sharpening if necessary. In addition, if the symmetry for the integer-pixel filter does not overlap, the integer-pixel filter can help capture the exact movement of an object, such as moving an object that is not a multiple of a quarter of a pixel. Finally, integer-pixel filtering can also be used to improve compression of video sequences when changes in focus occur.

In some cases, interpolation filtering of integer-pixel positions may depend on motion vectors. In other words, the MC module 37 of the prediction module 32 can adaptively apply interpolation filtering based on the motion vector identified by the ME module 35. For example, interpolation filtering of integer-pixel positions can be performed when the motion vector indicates an integer-pixel position. Interpolation filtering can result in a adjusted pixel value for an integer-pixel position. When a motion vector indicates a subpixel position, integer-pixel position filtering is not performed relative to integer-pixel values, but filtering can be used to interpolate sub-pixel values. Thus, integer-pixel filtering can be considered adaptive based on motion vectors.

For integer-pixel positions, a 5 × 5 interpolation filter in the form of equation (15) can be retrieved using an analytical process identical to the process used to extract the sub-pixel position filters.

where p ^FP is the filtered integer pixel value, P _{i, j} is the integer pixel value at position (i, j), h ^FP _{i, j} is the coefficient for position (i, j), and o ^FP is the filter offset. In some cases, the interpolation filter for integer-pixel positions may not have a filter offset (i.e. o ^FP = 0). In other cases, the o ^FP filter offset can use any number of technologies.

Different offsets can be set for each subpixel position, for example, fifteen different offsets can be set for each subpixel position shown in FIG. 3. A different offset can be set for whole pixels, giving a total number of offsets of sixteen. The simultaneously pending and transferred application No. ________, addressed to Karczewicz et al., Entitled "OFFSETS AT SUB-PIXEL RESOLUTION", Attorney's case number 081399U2, filed on the same day as this application and incorporated herein by reference, provides many additional information regarding the use of many offsets for different pixels and sub-pixel locations.

To reduce the number of filtering coefficients to be transmitted, symmetry in the coefficients, as in equation (16), can be superimposed on an integer-pixel filter.

Other forms of symmetry by coefficients can also be used. Whether or not symmetry is imposed by the coefficients can be determined by the encoder and signaled to the decoder. If symmetry, as in (16), is used for an integer-pixel filter, then 9 filtering coefficients and 1 offset should probably be transmitted to the decoder.

The encoding of the offsets may be performed by encoding the integer offset first and then encoding each fractional offset. In this case, the integer offset and fractional offsets are encoded differently. The integer offset can be encoded using the exponential signed Golomb code. Depending on the magnitude of the integer bias, fractional biases may be quantized differently from the integer bias. For example, more approximate quantization levels can be applied to fractional biases when the integer bias is large. After quantization, fractional offsets can be encoded using fixed-length codes. As an example, the following pseudo-code can illustrate how a given fractional offset can be encoded using various fixed-length codes defined based on an integer offset value.

Let offsetI be an integer offset

Let offsetF be a fractional offset

Let offsetFbits be the number of bits used to quantize offsetF

Let abs (a) be the absolute value of a

if (abs (offsetI) = 0)

offsetFbits = 5

else if (abs (offsetI) <3)

offsetFbits = 4

else if (abs (offsetI) <7)

offsetFbits = 2

else

offsetFbits = 0

In this case, video encoder 50 may use a coefficient prediction and quantization module 41, which for each of the offset values assigns the first number of bits of the integer part of the given offset value and assigns the second number of bits of the fractional part of this offset value, with the first and second number of bits being determined by based on the value of the integer part. In this case, the quantized versions of the offset values may be sent from the prediction unit 32 as elements of the prediction syntax after the quantization of the offset values is performed according to the above pseudo-code. In any case, video encoder 50 may encode the first number of bits differently from the second number of bits.

7 is a conceptual diagram illustrating four integer-pixel positions and fifteen sub-pixel positions with a hatch to group pixel positions that can use pixel symmetry for filtering coefficients in accordance with this disclosure. In particular, FIG. 7 illustrates the sub-pixel positions “a” through “o” corresponding to the integer-pixel positions C3. The positions in the integer subpixel shown in FIG. 7 correspond to the positions in FIGS. 4 and 5. Thus, the subpixel locations “e”, “f”, “g”, “i”, “j”, “k”, “ m "," n "and o" may have values determined by twelve pixel support, as shown in Figure 4, with subpixel locations "a", "b" and "c" and subpixel locations "d", " h "and" l "may have the meanings determined by the horizontal and linear pixel supports shown in Figure 5.

In this scenario, pixel symmetry can be superimposed so that the filter coefficients for the pixel location “a” are symmetrical with respect to the filter coefficients for the pixel location “c”. Similarly, pixel symmetry can be superimposed so that the filter coefficients for the pixel location “d” are symmetrical with respect to the filter coefficients for the pixel location “l”. The pixel symmetry can be superimposed so that the filter coefficients for the pixel location “e” are symmetrical with respect to the filter coefficients for the pixel locations “g”, “m” and “o” and the pixel symmetry can be superimposed so that the filter coefficients for the location pixels "i" are symmetrical with respect to filtering coefficients for pixel locations "k", and pixel symmetry can be superimposed so that filtering coefficients for pixel locations "f" are symmetric with respect to the filter coefficients for pixel location "n". Therefore, only eight sets of coefficient values may need to be transmitted as part of the video bitstream to enable the decoder to generate complete sets of coefficient values for all fifteen pixel locations. In addition, for any given set of coefficients, symmetry by coefficients can only send a subset of coefficient values, and the decoder can generate a complete set of coefficient values for a given pixel location based on a subset and symmetry by coefficients.

One aspect of the present disclosure is to virtually remove all pixel symmetry between subpixel positions "a" and "c" relative to subpixel positions "d" and "l". In addition, pixel symmetry can be eliminated for the sub-pixel position "f" relative to the sub-pixel position "i". In addition, pixel symmetry can be eliminated for the sub-pixel position "k" relative to the sub-pixel position "n". In such cases, the diagonal correlation may be low, making it ineffective or ineffective to impose diagonal symmetry. Thus, the eight sets of coefficients used to form all fifteen sets of coefficients can be a larger set than in some traditional technologies when symmetry is eliminated in the diagonal dimension for some of the pixel locations. In some cases, pixel diagonal symmetry and coefficient diagonal symmetry may or may not be allowed to allow dispersion in vertical dimensions.

Referring again to FIG. 1, video encoder 22 may determine eight sets of filter coefficients and use eight sets of filter coefficients to generate all fifteen sets of filter coefficients corresponding to fifteen different sub-pixel locations, based on symmetry in the coefficients and pixel symmetry, and output eight sets filter coefficients as part of the encoded bitstream. Eight sets of filter coefficients may be output for each coded unit of the encoded bitstream, or perhaps several cases of eight sets of filter coefficients may be output for different areas of the coded unit.

Fifteen sets correspond to each subpixel quarter-pixel and half-pixel location shown in FIG. 3, and eight sets, coefficient symmetry, and pixel symmetry determine filter supports for each sub-pixel quarter-pixel and half-pixel location. Video decoder 28 of the target device 16 can receive eight sets of filter coefficients as part of the encoded video bitstream, generate fifteen sets of filter coefficients corresponding to fifteen different sub-pixel locations, based on eight sets of filter coefficients, generate interpolated predictive data for video decoding based on one of the fifteen sets of coefficients filtering, while the interpolated predictive data correspond to one of the spots twenty different sub-pixel locations, and decode one or more video blocks based on interpolated predictive data.

At least some of the eight sets of filter coefficients may have coefficient symmetry such that the coefficient symmetry set contains a subset of filter coefficients associated with filter support for a given sub-pixel location associated with this set. Thus, video decoder 28 can generate fifteen sets of filter coefficients, at least in part, on the basis of pixel symmetry between two or more different sub-pixel locations. In addition, video decoder 28 may generate fifteen sets of filter coefficients at least partially based on pixel symmetry between two or more different sub-pixel locations and at least partially based on coefficient symmetry specified for this set so that a set with coefficient symmetry contains a subset of filtering coefficients associated with filter support for a given sub-pixel location associated with this set.

Fifteen sets, on the other hand, can correspond to each quarter-pixel and half-pixel sub-pixel location. Pixel symmetry may exist between fifteen different sets of filter coefficients in the vertical dimension and in the horizontal dimension, but pixel symmetry does not exist in the diagonal dimension for at least some of the fifteen sets of filter coefficients. As explained in more detail below, two out of eight sets can specify three out of fifteen sets for supporting a horizontal filter, two out of eight sets can define three out of fifteen sets for supporting a vertical filter, and four out of eight sets can specify nine out of fifteen sets for two-dimensional filter support . Two-dimensional support may include applicable support for a 12-position filter surrounding nine subpixel positions in radial form.

In particular, the interpolated predictive data may contain fifteen quarter-pixel and half-pixel values, for example, shown in Fig. 7, while the interpolation filter defines a two-dimensional array of twelve filter support positions, for example, shown in Fig. 4, which are used for determining nine out of fifteen quarter-pixel and half-pixel values, and, for example, also shown in FIG. 4. The generation of interpolated predictive data may further include the use of linear interpolation filters that specify one-dimensional matrices of filter support positions for six of the fifteen quarter-pixel and half-pixel values, for example, as shown in FIG.

Horizontal symmetry and the absence of diagonal symmetry can be observed relative to the subpixel positions "a" and "c" or relative to the subpixel positions "i" and "k". In this case, the positions in the integer subpixels "a" and "c" are symmetric along the horizontal axis X, but the symmetry does not exist along the diagonal axis for these pixels so that the subpixel locations "d" and "l" are not symmetrical with respect to the subpixel positions "a" and "c", respectively. Similarly, the sub-pixel positions “i” and “k” are symmetric along the horizontal axis X, but there is no symmetry along the diagonal axis for these pixels, so the sub-pixel locations “f” and “n” are not symmetrical with respect to the sub-pixel positions “i” and "k", respectively.

Pixel symmetry means that the first set of coefficients with respect to the first subpixel position is symmetrical with respect to the second set of coefficients with respect to the second subpixel position. For example, the location of the pixel "a" has pixel symmetry relative to the location of the pixel "c", and the location of the pixel "d" has pixel symmetry relative to the location of the pixel "l". The pixel location “f” has pixel symmetry with respect to the pixel location “n”, and the pixel location “i” has pixel symmetry with respect to the pixel location “k”. The pixel location “e” has pixel symmetry with respect to the pixel locations “g”, “m”, and “o”. The shading shown in FIG. 7 shows this pixel symmetry, for example, with a common shading corresponding to pixel locations that are symmetrical with respect to other pixel locations. In this case, coefficients for eight pixel locations (together with aspects of pixel symmetry) can specify coefficients for all fifteen pixel locations. In addition, within a given set of coefficients, there may be symmetry in the coefficients, so that only a subset of the coefficients in this set of coefficients must be transmitted with a bit stream.

Video encoder 22 may generate encoded video data based on the interpolated predictive data, wherein the interpolated predictive data contains fifteen quarter-pixel and half-pixel values, the interpolation filter defines a two-dimensional array of twelve filter support positions that are used to specify nine of the fifteen quarter-pixel and half-pixel values. The generation of encoded video data based on interpolated predictive data may further include the use of linear interpolation filters, which specify one-dimensional matrices of filter support positions for six of the fifteen quarter-pixel and half-pixel values.

8 is a conceptual diagram illustrating six horizontal linear pixel support positions C1-C6 with respect to the sub-pixel position “b” with a hatch that shows symmetry in coefficients. In this case, coefficient symmetry means that only three filter coefficients are required for C1, C2, and C3 in order to specify a complete set of coefficients for filter support positions C1-C6. C1 is symmetric with C6, C2 is symmetric with C5, and C3 is symmetric with C4. Thus, only three coefficients must be transmitted as part of the encoded video bitstream to specify a set of six coefficients required for interpolating the sub-pixel position “b”.

9 is a conceptual diagram illustrating six positions of supporting horizontal linear pixels with respect to a subpixel, with a hatching that shows the lack of symmetry in the coefficients. Thus, all six coefficients are required to determine a set of coefficients to support the filter relative to the sub-pixel position “a”. However, as indicated above, pixel symmetry means that these identical coefficients with respect to sub-pixel position “a” can also be used to extract filter support for sub-pixel position “c” (see FIG. 7). Thus, only six coefficients must be transmitted as part of the encoded video bitstream to specify two different sets of six coefficients required to interpolate the sub-pixel position “a” and “c”.

10 is a conceptual diagram illustrating six support positions of vertical linear pixels A3, B3, C3, D3, E3 and F3 with respect to the subpixel “h”, with a hatching that shows symmetry in coefficients. In this case, coefficient symmetry means that only three filter coefficients are required for A3, B3 and C3 in order to determine the full set of coefficients for filter support positions A3, B3, C3, D3, E3 and F3. A3 is symmetric with F3, B3 is symmetric with E3, and C3 is symmetric with D3. Thus, only three coefficients must be transmitted as part of the encoded video bitstream to determine the set of six coefficients required for interpolating the sub-pixel position “h”.

11 is a conceptual diagram illustrating six positions of supporting vertical linear pixels with respect to a subpixel, with a hatching that shows the lack of symmetry in the coefficients.

Thus, all six coefficients are required to specify a set of coefficients to support the filter relative to the sub-pixel position “d”. However, as indicated above, pixel symmetry means that these identical coefficients with respect to the sub-pixel position “d” can also be used to extract filter support for the sub-pixel position “l” (see FIG. 7). Thus, only six coefficients must be transmitted as part of the encoded video bitstream to specify two different sets of six coefficients required to interpolate the sub-pixel position “d” and “l”.

12 is a conceptual diagram illustrating twelve 2D pixel support positions (wherein integer-pixel positions are shown with capital letters and hatching) with respect to the sub-pixel position “e”. Hatching shows the lack of symmetry in the coefficients. Thus, all twelve coefficients are required to define a set of coefficients to support the filter relative to the sub-pixel position "e". However, as indicated above, pixel symmetry means that these identical coefficients with respect to the sub-pixel position “e” can also be used to extract filter support for the sub-pixel positions “g”, “m” and “o” (see FIG. 7). Therefore, 12 coefficients determine the sub-pixel position of the filter support "e", and these identical coefficients can be used to extract a set of coefficients for the sub-pixel positions "g", "m" and "o".

13 is a conceptual diagram illustrating twelve positions of supporting two-dimensional pixels (while integer-pixel positions are shown with capital letters) relative to the subpixel "i", with a hatch that shows symmetry in coefficients. In this case, coefficient symmetry means that the filter coefficients for filter support positions B3, B4, C2, C3, C4 and C5 can be used to set filter coefficients for filter support positions E3, E4, D2, D3, D4 and D5. B3 is symmetric with E3, B4 is symmetric with E4, C2 is symmetric with D2, C3 is symmetric with D3, C4 is symmetric with D4, and C5 is symmetric with D5. Thus, only six coefficients must be transmitted as part of the encoded video bitstream to determine the set of twelve coefficients required to interpolate the sub-pixel position “i”. In addition, as indicated above, the subpixel position "i" may have pixel symmetry relative to the subpixel position "k". Therefore, an identical subset of six coefficients can determine complete sets of twelve filtering coefficients for sub-pixel positions “i” and “k”.

14 is a conceptual diagram illustrating twelve 2D pixel support positions (wherein integer-pixel positions are shown with capital letters) relative to the subpixel “f”, with a hatch that shows symmetry in coefficients. In this case, coefficient symmetry means that the filter coefficients for filter support positions C2, D2, B3, C3, D3 and E3 can be used to determine the filter coefficients for filter support positions C5, D5, B4, C4, D4 and E4. C2 is symmetric with C5, D2 is symmetric with D5, B3 is symmetric with B4, C3 is symmetric with C4, D3 is symmetric with D4, and E3 is symmetric with E4. Thus, only six coefficients should be transmitted as part of the encoded video bitstream to determine the set of twelve coefficients required to interpolate the sub-pixel position “f”. Furthermore, as indicated above, the sub-pixel position “f” may have pixel symmetry with respect to the sub-pixel position “n”. Therefore, an identical subset of six coefficients can determine complete sets of twelve filter coefficients for sub-pixel positions “f” and “n”.

15 is a conceptual diagram illustrating twelve positions of supporting two-dimensional pixels (while integer-pixel positions are shown with capital letters) relative to the subpixel "j", with a hatch that shows symmetry in coefficients. In this case, symmetry in the coefficients means that only three filter coefficients are required for B3, C2 and C3 to determine the full set of twelve coefficients for filter support positions B3, B4, C2, C3, C4, C5, D2, D3, D4 , D5, E3 and E4. C3 is symmetric with C4, D3 and D4, C2 is symmetric with D2, C5 and E4, and B3 are symmetric with B4, E3 and D5. Thus, only three coefficients should be transmitted as part of the encoded video bitstream to determine the set of twelve coefficients required to interpolate the sub-pixel position “j”.

FIG. 16 is a block diagram illustrating an example of a video decoder that can decode a video sequence that is encoded as described herein. Video decoder 60 is one example of a specialized computer video device referred to herein as a “coder”. Video decoder 60 includes an entropy decoding module 52 that entropy decodes a received bitstream to generate quantized coefficients and prediction syntax elements. Prediction syntax elements may include a coding mode, one or more motion vectors, information identifying interpolation technology used to generate sub-pixel data, coefficients for use in interpolation filtering, and / or other information associated with generating the prediction block.

Prediction syntax elements, for example, coefficients, are sent to prediction module 55. If prediction is used to code coefficients relative to non-tunable filter coefficients or relative to each other, coefficient prediction and inverse quantization module 53 may decode syntax elements to specify actual coefficients. In addition, if quantization is applied to any prediction syntax, coefficient prediction and inverse quantization module 53 may also remove such quantization. Filtering coefficients, for example, can be predicted encoded and quantized according to the present disclosure, in which case coefficient prediction and inverse quantization module 53 can be used by video decoder 60 to predictively decode and quantize such coefficients.

Prediction unit 55 may generate predictive data based on prediction syntax elements and one or more previously decoded blocks that are stored in memory 62, in a manner similar to that described above with respect to prediction unit 32 of video encoder 50. In particular, prediction unit 55 may perform one or more interpolation filtering techniques of the present disclosure during motion compensation to form a prediction block with specific accuracy, for example, with quarter pixel accuracy. In fact, one or more of the technologies of the present disclosure may be used by video decoder 60 to form a prediction block. Prediction module 55 may include a motion compensation module that includes filters used for interpolation and similar interpolation filtering techniques of the present disclosure. The motion compensation component is not shown in FIG. 16 for simplicity and convenience of illustration.

The inverse quantization unit 56 inversely quantizes, i.e. quantizes block block coefficients. The inverse quantization process may be a process specified for H.264 decoding. Inverse transform unit 58 applies an inverse transform, such as an inverse DCT or a conceptually similar inverse transform process, to transform coefficients to form residual blocks in a pixel domain. An adder 64 sums the residual block with a corresponding prediction block generated by the prediction unit 55 to generate a reconstructed version of the original block encoded by the video encoder 50. If desired, the deblocking filter can also be applied to filter the decoded blocks to remove blocking artifacts. The decoded video blocks are then stored in a reference frame storage unit 62, which provides reference blocks for subsequent motion compensation and also generates decoded video to drive a display device (such as device 28 of FIG. 1).

NS-AIF technologies can calculate pixel values for two-dimensional subpixel positions using an interpolation filter that has 36-position filter support (i.e., 6 × 6 two-dimensional filter support). S-AIF uses separable integer-pixel positions as a filter support for interpolation filtering, first in the horizontal direction and not in the vertical dimension. D-AIF uses a less complex interpolation filter to calculate two-dimensional subpixel positions than NS-AIF. In D-AIF, two-dimensional sub-pixel positions are calculated using an interpolation filter that has either 6-position filter support or 12-position diagonal filter support.

One drawback of using diagonal filter support, as in D-AIF, is that the integer-pixel positions used in filtering are at a considerable distance from the current position to be interpolated. As the distance between the current position to be interpolated and the pixel positions used as filter support increases, the spatial correlation of pixels decreases. In fact, the diagonal filter support used in D-AIF is less suitable for accurate prediction.

In order to provide better prediction using pixels with a higher correlation with the position that should be interpolated (i.e., the positions of the pixel that are closer or shorter than the position that should be interpolated), while maintaining the same low complexity offered by D -AIF, the 12-position filter support shown in FIG. 4 can be used for interpolation. The integer-pixel positions used in supporting the 12-position filter described in the present disclosure include four integer-pixel positions surrounding sub-pixel positions, i.e. integer-pixel positions C3, C4, D3 and D4, which may be referred to as “angular” integer-pixel positions, since they are near the corners of two-dimensional sub-pixel positions. In addition to angular integer-pixel positions, one or more integer-pixel positions directly adjacent to angular integer-pixel positions can also be used to support the filter. Integer-pixel positions directly adjacent to angular integer-pixel positions can include integer-pixel positions immediately above (up or north) or directly below (down or south) angular integer-pixel positions, as well as integer-pixel positions directly to the left (or west) or directly to the right (or east) of angular integer-pixel positions. In the example illustrated in FIG. 4, support for a 12-position filter may include integer-pixel locations B3, B4, C2, C3, C4, C5, D2, D3, D4, D5, E3 and E4. All these integer-pixel positions are within two integer-pixel positions of the sub-pixel to be interpolated.

However, filter support in accordance with the techniques of the present disclosure may include additional integer-pixel locations. For example, filter support may include integer-pixel locations that are within three integer-pixel positions of a sub-pixel position to be interpolated, which form a radial shape.

Even quarter-pixel positions or more accurate resolution can be interpolated based on integer-pixel positions located within at least two integer-pixel positions. Thus, the filter support technologies of the present disclosure provide better prediction of interpolation using integer pixels with a higher correlation with the position to be interpolated, while maintaining the same low complexity offered by D-AIF.

As described above, subpixel symmetry can force an identical set of filtering coefficients (and mirrored, inverted, and / or rotated versions of the set) for a group of one or more subpixel positions. An analytical process, for example, such as the process in equation (5), can be used to extract interpolation filtering coefficients for all sub-pixel positions, for example, fifteen sub-pixel positions in the case of quarter-pixel accuracy. Correlation analysis for fifteen sets of interpolation filtering coefficients reveals a relatively strong correlation between the filtering coefficients of subpixel positions (both one-dimensional and two-dimensional subpixel positions) in one dimension (for example, vertical or horizontal dimension). Conversely, the correlation between the filtering coefficients of subpixel positions in different diagonal directions can be noticeably weaker.

For example, there may be a high correlation between the filtering coefficients for the sub-pixel positions "a" and "c", which are in the horizontal dimension. As another example, there may be a high correlation between the filter coefficients for the sub-pixel positions “d” and “l”, which are in the vertical dimension. However, there may be a slight (if any) correlation between the filter coefficients for the subpixel positions “a” that are in the horizontal dimension and the subpixel position “d” that is in the vertical dimension. Based on these observations of correlations, subpixel symmetry can be designed so that subpixel symmetry for filter coefficients of subpixel positions in diagonal dimensions is not superimposed. This results in eight sets of coefficients, as described herein, which may constitute a larger number of sets than is required for some conventional processes. By eliminating pixel diagonal symmetry for some pixel locations, as described herein, video coding and compression can be improved.

On the other hand, FIG. 7 shows pixel positions with the same hatching that belong to a group of subpixel positions that have subpixel symmetry. In particular, the subpixel positions "a" and "c" form a first group having a symmetry in subpixels, the subpixel positions "d" and "l" form a second group having a symmetry in subpixels, the subpixel positions "e", "g", " m "and" o "form a third group having subpixel symmetry, subpixel positions" f "and" n "form a fourth group having subpixel symmetry, and subpixel positions" i "and" k "form a fifth group having symmetry in subpixels. The subpixel positions “b”, “h”, and “j” are not symmetrical in subpixels with other subpixel positions. In fact, the subpixel positions "b", "h" and "j" can be considered as belonging to their own groups; sixth, seventh and eighth groups, respectively. Each group of coefficients (which may be subsets due to symmetry in the coefficients) can be transmitted once per coded unit or, possibly, repeatedly per coded unit if different types of filtering are specified for different areas or features of the coded unit. Coefficients for similar interpolation integer-pixel filtering can also be sent once or several times per encoded unit.

In NS-AIF and D-AIF, sub-pixel symmetry exists between sub-pixel positions in the same dimension, for example, between two sub-pixel positions in a vertical dimension or two sub-pixel positions in a horizontal dimension. In some traditional NS-AIF and D-AIF, symmetry also exists diagonally between subpixel positions in different dimensions. For example, sub-pixel symmetry exists between the sub-pixel position “a” in the vertical dimension and the sub-pixel position “d” in the horizontal direction in some conventional NS-AIF and D-AIF. With subpixel symmetry in the horizontal, vertical, and diagonal dimensions, only five unique sets of interpolation filtering coefficients can be used for quarter-pixel accuracy.

For some types of S-AIF, on the other hand, subpixel symmetry exists between subpixel positions in the vertical direction (or dimension), but not between subpixel positions in the horizontal dimension. In other words, subpixel symmetry does not always exist for subpixel positions in the same dimension. Subpixel symmetry also does not exist in diagonal dimensions. The symmetry of some S-AIF schemes, therefore, requires more sets of interpolation coefficients. In particular, for quarter-pixel accuracy, some S-AIF schemes require eleven unique sets of interpolation coefficients.

The sub-pixel symmetry scheme described and illustrated in FIG. 7 can lead to more accurate prediction than the sub-pixel symmetry described above for some types of NS-AIF and D-AIF. In particular, the symmetry scheme for subpixels in Fig. 7 imposes symmetry for subpixels in one dimension (for example, in the horizontal direction or vertical direction), but does not impose symmetry for subpixels simultaneously in both dimensions (for example, diagonally).

Due to the failure to apply symmetry on the subpixels for the subpixel positions on the diagonal, a weak correlation between the interpolation filtering coefficients and the subpixel positions in different dimensions is not integrated into the interpolation. Although a larger number of sets of interpolation coefficients may be required (for example, eight instead of five), the resulting interpolated predictive data may be more accurate. Coefficient diagonal symmetry may also not be allowed or eliminated in a similar way.

Prediction unit 32 may also impose symmetry in the coefficients, as described above. In particular, symmetry in the coefficients is imposed for the filtration coefficients in one dimension (for example, in the horizontal or vertical direction), but not for both dimensions in a diagonal way. For example, filtering coefficients for the sub-pixel position “e” are not forced to be set as horizontally symmetrical as in the NS-AIF scheme described above and presented in equation (11). The symmetry of the coefficients is summarized below in equations (17) - (21).

, равны нулю в случаях, в которых 12-позиционный фильтр, описанный подробно относительно Фиг.4, может использоваться. По сути, симметрия по субпикселам и коэффициентам, описанная в этом раскрытии сущности, может использоваться вместе или отдельно от технологий поддержки фильтра. При использовании вместе с поддержкой 12-позиционного фильтра, описанной на Фиг.4, предусмотрено 6(a)+3(b)+6(d)+3(h)+12(e)+6(f)+6(i)+3(j)=45 уникальных коэффициентов для этих субпикселных позиций, которые должны отправляться в декодер, т.е. 6 коэффициентов для группы, включающей в себя субпикселную позицию "a", 3 коэффициента для группы, включающей в себя субпикселную позицию "b", 6 коэффициентов для группы, включающей в себя субпикселную позицию "d", 3 коэффициента для группы, включающей в себя субпикселную позицию "h", 12 коэффициентов для группы, включающей в себя субпикселную позицию "e", 6 коэффициентов для группы, включающей в себя субпикселную позицию "f", 6 коэффициентов для группы, включающей в себя субпикселную позицию "i", и 3 коэффициента для группы, включающей в себя субпикселную позицию "j".It should be noted that for the subpixel positions "f", "i" and "j", some filtering coefficients, i.e.

are equal to zero in cases in which a 12-position filter, described in detail with respect to FIG. 4, can be used. In fact, the symmetry of subpixels and coefficients described in this disclosure can be used together or separately from filter support technologies. When used in conjunction with the 12-position filter support described in FIG. 4, 6 (a) +3 (b) +6 (d) +3 (h) +12 (e) +6 (f) +6 (i ) +3 (j) = 45 unique coefficients for these sub-pixel positions to be sent to the decoder, i.e. 6 coefficients for a group including a subpixel position "a", 3 coefficients for a group including a subpixel position "b", 6 coefficients for a group including a subpixel position "d", 3 coefficients for a group including a subpixel position "h", 12 coefficients for a group including a subpixel position "e", 6 coefficients for a group including a subpixel position "f", 6 coefficients for a group including a subpixel position "i", and 3 coefficient for a group including a subpixel "J" position.

17 is a flowchart illustrating an example operation of a video encoder that uses twelve pixel filter support in accordance with the present disclosure. The technology of FIG. 17 is described in terms of the video encoder 50 of FIG. 2, although other devices may perform similar technologies. As shown in FIG. 17, the MC module 37 of the prediction module 32 receives a block of pixels from the memory 34, which include integer-pixel values corresponding to integer-pixel positions (171). Filter (s) 39 calculates subpixel values based on filter support that contains twelve or more positions that surround subpixel positions (172). As explained in more detail above, support for a 12-position filter can be used to generate nine out of the fifteen possible subpixel interpolations, with linear horizontal and linear vertical filtering can be used to form six out of the fifteen possible subpixel interpolations.

The MC module 37 then generates a prediction block based on the calculated subpixel values (173). In particular, the MC module 37 may generate and output an interpolated prediction block containing interpolated subpixel values. The adder 48 may then encode the current video block based on the interpolated prediction block (174), for example, by subtracting the interpolated prediction block from the encoded video block to form a residual block. The residual block can then be converted and quantized by the transform module 38 and the quantization module 40, respectively. After entropy encoding by means of entropy encoding unit 46, video encoder 50 may output an encoded video bit stream and filter information (175). The filter information, as described herein, may contain eight sets of coefficients used to form all fifteen sets of coefficients for fifteen sub-pixel positions. Filter information may be output once per coded unit, or possibly several times per coded unit, if different areas of the coded unit use different types of sub-pixel interpolation.

18 is a flowchart illustrating an example operation of a video decoder that uses twelve pixel filter support in accordance with the present disclosure. Thus, the process of FIG. 18 can be considered a mutually inverse decoding process for the encoding process of FIG. 17. FIG. 18 is described in terms of the video decoder 60 of FIG. 16, although other devices may perform similar technologies. As shown in FIG. 18, video decoder 60 receives encoded video blocks and filter information (181). Entropy decoding unit 52 may entropy decode this received information. Prediction unit 55 performs interpolation motion compensation according to the techniques of the present disclosure. In particular, the prediction unit 55 obtains a block of pixels from the storage device 62, which include integer-pixel values corresponding to integer-pixel positions (182). Prediction unit 55 may use received motion vectors to determine how to perform interpolation. Based on the motion vectors, the prediction unit 55 can calculate the subpixel based on the filter support, which contains twelve or more positions that surround the subpixel positions (183). Thus, the prediction unit 55 uses interpolation to form a prediction block (184). The prediction block may be used to decode the residual video block by calling adder 64 to add the prediction block to the residual block (185). Various aspects of the interpolation described in this document, including the use of 12-position filter support that surrounds sub-pixel positions in radial form, can improve video encoding by providing better quality interpolated data than traditional technologies.

FIG. 19 is a flowchart illustrating an example operation of a video encoder that uses coefficient symmetry and pixel symmetry in accordance with the present disclosure. The technology of FIG. 19 is described in terms of the video encoder 50 of FIG. 2, although other devices may perform similar technologies. As shown in FIG. 19, the prediction unit 32 of the video encoder 50 sets eight sets of filter coefficients for fifteen subpixel positions based on the symmetry of the coefficients and the symmetry of the pixels, as described herein (201). In particular, prediction module 32 may use pixel symmetry to reduce sets of coefficients from fifteen to eight, and may further reduce the number of coefficients for some or all eight sets based on symmetry in coefficients between the coefficients of these sets. Prediction unit 32 can then encode video using interpolation filters (such as filter (s) 39) and eight sets of filter coefficients (202). Video encoder 50 may output encoded video data and eight sets of filter coefficients (203). Eight sets of coefficients that are encoded and output in the bitstream can be sent with each coded unit (for example, with each frame or series of consecutive macroblocks) so that filtering coefficients that should be used for interpolation can be communicated to the decoding device. Alternatively, different groups of eight sets of coefficients can be encoded and sent with each coded unit to provide different types of interpolations at different locations within the coded unit.

20 is a flowchart illustrating an example operation of a video decoder that uses coefficient symmetry and pixel symmetry in accordance with the present disclosure. In this case, the prediction unit 55 of the video decoder 60 receives eight sets of filter coefficients (191) and generates fifteen sets of filter coefficients based on eight sets, symmetry by coefficients and symmetry by pixels. Prediction unit 60 can then program its interpolation filters and apply such interpolation filters to properly generate interpolated prediction data (193). The video decoder 60 may then decode the video blocks based on the predictive data (194), for example, by calling the adder 64, to add the correct predictive data interpolated by the prediction unit 55 to the residual block to be decoded.

21 is a flowchart illustrating an example operation of a video encoder that uses integer-pixel filtering to generate adjustable integer-pixel values in accordance with the present disclosure. It is explained above as a similar interpolation filtering, since it is similar to interpolation, but does not form values into an integer subpixel. On the contrary, this process generates new integer values that are filtered based on the original integer value and other integer values that surround the original integer value.

FIG. 21 is described from the perspective of the video encoder 50 of FIG. 2, although other devices may perform similar technologies. As shown in FIG. 21, the MC module 37 of the prediction module 32 receives a block of pixels from the memory 34, which include integer-pixel values corresponding to integer-pixel locations (211). Filter (s) 39 of prediction unit 32 filters integer-pixel values based on other integer-pixel values of a pixel block to form adjustable integer-pixel values (212). Prediction unit 32 generates a prediction block based on adjusted integer-pixel values (213), and video encoder 50 encodes a video block based on a prediction block (214), for example, by calling adder 48 to subtract a prediction block from the encoded video block to form a residual block. After transforming and quantizing the residual block by the transform module 38, the quantization module 40, and after entropy encoding by the entropy encoding module 46, the video encoder 50 outputs the encoded video block and filter information (215). As in the case of sub-pixel interpolation, similar interpolation filtering with respect to integer-pixel locations can include outputting and transmitting filter information once per encoded unit or, possibly, several times per encoded unit, if different areas of the encoded unit use different types of integer similar interpolation filtering.

FIG. 22 is a flowchart illustrating an example operation of a video decoder that uses integer-pixel filtering to generate adjustable integer-pixel values in accordance with the present disclosure. The prediction unit 55 of the video decoder 60 receives the encoded video blocks, and also receives filter information (for example, once per encoded unit) (221). The prediction unit 55 obtains a block of pixels from the storage device 62, which include integer-pixel values corresponding to integer-pixel positions (222). Prediction unit 55 activates a filter (not shown) as part of motion compensation to filter integer-pixel values based on other integer-pixel values of the pixel block to form adjustable integer-pixel values (223). Prediction unit 55 generates a prediction block based on adjusted integer-pixel values (224). The video decoder 60 can then decode the video blocks based on the prediction block, for example, when the adder 64 is called, to add the prediction block interpolated by the prediction unit 55 to the residual video block to be decoded (225).

23 is a flowchart illustrating a technology for distortion-based interpolation depending on a transmission rate for encoding a video based on two sets of filter options. In this case, the MC module 37 of the prediction module 32 may generate two or more different sets of predictive data, one set based on a first set of interpolation filter options and another set based on a second set of interpolation filter options. The MC module 37 can then use the analysis based on the distortion depending on the transmission speed to select which interpolation filter gives the best results in terms of speed and distortion. Thus, the MC module 37 not only considers which set of predictive data gives the best results (i.e., the least distortion), but also takes into account the fact that one set of filter options may require a smaller additional number of bits to transmit filtering coefficients to the decoder.

As shown in FIG. 23, the MC module 37 of the prediction module 32 generates first interpolated prediction data for encoding video data based on the first interpolation filter (231). The first interpolation filter, in one example, contains a non-tunable interpolation filter corresponding to a filter specified by a particular video coding standard, such as a filter that corresponds to an ITU-T H.264 interpolation filter. For added accuracy, in some cases, a non-tunable interpolation filter may correspond to an ITU-T H.264 interpolation filter without intermediate rounding of half-pixel values, which are used to form quarter-pixel values. In this case, a non-tunable interpolation filter can generate half-pixel values and can round up such values for interpolation to half-pixel resolution. However, to the extent that such half-pixel values should also be used to interpolate up to quarter-pixel resolution, the MC module 37 can store unrounded versions of half-pixel values and use non-round versions of half-pixel values to interpolate up to a quarter-pixel resolution. The simultaneously pending and transferred application No. ________, addressed to Karczewicz et al., Entitled "ADVANCED INTERPOLATION TECHNIQUES FOR MOTION COMPENSATION IN VIDEO CODING", Attorney's case number 081399U1, which was filed on the same day as this application and which is incorporated into this document by links, provides a wealth of additional information regarding interpolation filtering without intermediate rounding of half-pixel values, which are used to form quarter-pixel values.

Then, the MC module 37 of the prediction module 32 generates second interpolated prediction data for encoding video for the video data based on the second interpolation filter (232). In one example, the second interpolation filter may comprise an adaptive interpolation filter. In this case, according to adaptive interpolation, the MC module 37 may specify filtering coefficients to be used. In particular, the MC module 37 may perform an adaptive interpolation process in which the MC module 37 starts with preset filter coefficients, generates preliminary predictive data and then adjusts the filter coefficients in the adaptive process so that such filter coefficients define more desirable predictive data.

After the MC module 37 of the prediction module 32 generates both the first and second interpolated prediction data, the MC module 37 can choose between the first interpolated prediction data and the second interpolated prediction data based on a distortion analysis depending on the transmission rate (233). Thus, the MC module 37 not only considers which set of predictive data gives the best results (i.e., the least distortion), but the MC module 37 also takes into account different amounts of data (i.e., different bit rates) which should be required for the first interpolation filter relative to the second interpolation filter. Namely, if a non-tunable interpolation filter is used (for example, as the first interpolation filter), video encoder 50 should not encode filter coefficients in the bitstream, whereas if an adaptive interpolation filter is used (for example, as a second interpolation filter), video encoder 50 must encode filter coefficients. Accordingly, an analysis of the distortion depending on the transmission rate can determine which set of predictive data gives the best results (i.e., the least function of the cost of distortion depending on the transmission speed), by taking into account the fact that the use of a non-tunable interpolation filter does not require additional bits to pass filter coefficients to the decoder.

More specifically, the choice between the first interpolated predictive data and the second interpolated predictive data based on a transmission rate distortion analysis (233) may comprise calculating a first distortion cost function as a function of the transmission rate associated with the video data if the video data is encoded through the first interpolated predictive data, calculating a second distortion cost function as a function of the transmission rate associated with the video data, if odannye encoded via the second interpolated predictive data, and selecting between the first interpolated predictive data and the second interpolated predictive data based on the first and second distortion cost functions depending on the transmission rate.

The function of the cost of distortion depending on the transmission rate associated with the video data, if the video data is encoded through the first interpolated predictive data, may contain a first differential metric, for example, the mean square error (MSE) of the pixel values or the sum of the absolute difference (SAD) of the pixel values or the sum the square difference (SSD) of the pixel values indicating the difference between the video data and the first interpolated predictive data, plus a first value that determines the quantities continuously cost associated with encoding of filter coefficients. In this case, if the first interpolation filter is a non-tunable interpolation filter, the first value that quantifies costs can be set to zero. Similarly, the second distortion cost function, depending on the transmission rate, may contain a second differential metric indicating the differences between the video data and the second interpolated predictive data (MSE, SAD or SSD), plus a second value that quantifies the costs associated with encoding the filter coefficients. If the second interpolation filter is an adaptive interpolation filter, the second value, which quantifies the costs associated with encoding the filter coefficients, may contain the number of bits (r) required to code the coefficients of adaptive interpolation filtering, or, possibly, this number (r) times the Lagrange multiplier (λ).

After choosing between the first interpolated predictive data and the second interpolated predictive data based on the analysis of the distortion depending on the transmission rate (233), the MC module 37 of the prediction module 32 encodes the video data based on the selection (234) and encodes the syntax to indicate the choice (235) . The syntax may include a single-bit flag or a multi-bit flag that specifies whether the first interpolation filter or the second interpolation filter should be used by the decoder. The process of FIG. 23 may be repeated for each of a plurality of subpixel locations for each coded unit of a video sequence to indicate whether a first interpolation filter or a second interpolation filter should be used by a decoder for each of the plurality of subpixel locations. Subpixel locations may contain fifteen possible subpixel locations according to interpolation up to quarter pixel resolution, or may contain a different number of subpixel locations. The process of FIG. 23 may also be repeated for an integer-pixel location for each coded unit of a video sequence to indicate whether a first interpolation filter or a second interpolation filter should be used by a decoder for the integer-pixel location.

After the video encoder 50 (FIG. 2) encodes the video data based on the selection (234) and encodes the syntax to indicate the selection (235), the modem 23 and the transmitting device 24 (FIG. 1) can modulate and transmit the encoded video data and the syntax to the target device 17. In case the first interpolation filter is non-tunable and the second interpolation filter is adaptive, the transmitting device 24 may transmit filter coefficients when the syntax indicates that the second interpolated prediction data is used They can be used to generate encoded video data, but may not transmit filtering coefficients when the syntax indicates that the first interpolated predictive data is used to generate encoded video data. Thus, when the first interpolation filter is non-tunable and the second interpolation filter is adaptive, filter coefficients are only sent if the syntax indicates that adaptive interpolation filtering has been used and the decision whether to use adaptive interpolation filtering does not only consider the quality of the prediction filter video, but also the bit rate, which is affected by the presence of filter coefficients in the transmitted bit stream. In other examples, however, the first and second interpolation filters may be non-tunable or may be adaptive.

The technologies of this disclosure can be used in many scenarios, including scenarios where there are more than two sets of filters selected through an encoder. In other words, additional interpolation filters can also be used to generate additional interpolated predictive data, which can also be considered in the analysis of distortion depending on the transmission speed. In other words, the method is not limited to generating only the first and second interpolated predictive data based on two interpolation filters, but can be used to generate any plurality of interpolated predictive data based on any number of interpolation filters. It is important that the interpolated predictive distortion analysis data as a function of the transmission rate is used to identify which filter should be selected. In one example, a method performed by video encoder 50 may comprise generating a plurality of different versions of predictive data for encoding video data based on a plurality of different interpolation filters, selecting from a plurality of different versions of predictive data based on an analysis of distortion depending on a transmission rate, encoding video data based on selection and syntax coding to indicate selection.

In addition to selecting an interpolation filter based on the analysis of the distortion depending on the transmission rate and syntax generation to indicate the choice, the MC module 37 of the prediction module 32 can also perform a similar analysis of the distortion depending on the transmission rate with respect to integer-pixel filtering, which is explained in more detail above. In particular, the MC module 37 of the prediction module 32 may generate two or more different sets of predictive data for integer-pixel locations, for example, one with integer-pixel filtering and one without integer-pixel filtering, and may perform distortion analysis depending on the transmission rate with respect to these two different sets to determine whether or not integer-pixel filtering is required. Accordingly, the MC module 37 of the prediction module 32 may generate another syntax element based on a distortion analysis depending on the transmission rate associated with two different sets of integer predictive data to indicate whether or not integer-pixel filtering should be applied to the coded unit, with the first set integer predictive data is not filtered, and a second set of integer predictive data is filtered. Thus, the decision whether or not to perform integer-pixel filtering can be based not only on the quality of the video encoding, but also on the possible additional number of bits associated with integer-pixel filtering, in particular when integer-pixel filtering should encode and send integer-pixel coefficients filtering used to perform such integer-pixel filtering. Integer filtering can also consider N integer filters (for example, where N is any positive and plural integer). According to the use of N integer filters, the above example should correspond to the case where N is two, and one of the filters does not apply filtering.

Each time interpolation filtering coefficients are actually encoded and sent from source device 12 to target device 16, this disclosure also considers techniques for encoding such interpolation filtering coefficients. The coding described for filtering coefficients improves data compression. In particular, this disclosure discusses prediction technologies for filter coefficients, for example, with respect to non-tunable filter coefficients. In addition, this disclosure discusses prediction technologies for a second set of filter coefficients relative to a first set of filter coefficients. Thus, imperfect symmetry between different filter coefficients can be used to provide data compression. In addition to using such prediction technologies for interpolation filtering coefficients, this disclosure also provides useful quantization and entropy coding technologies for interpolation filtering coefficients based on prefix and suffix coding. The following explanation provides further details on these aspects of this disclosure.

24 is a flowchart illustrating a technique for encoding filter coefficients using predictive coding. In this case, the filtering coefficients used by the video encoder 50 can be predictively encoded with respect to the filtering coefficients of the non-tunable filter, which further improves data compression when the filtering coefficients are sent as part of the encoded bitstream.

As shown in FIG. 24, the MC module 37 of the prediction module 32 has a set of filter coefficients for interpolating the predictive data when encoding video (241). For example, the MC module 37 may identify a set of filter coefficients by performing an adaptive interpolation filtering process. In this case, the MC module 37 may determine a set of filtering coefficients through the adaptive interpolation filtering process and generate predictive data based on the filtering coefficients identified in the adaptive interpolation filtering process. With adaptive interpolation filtering, as explained in more detail in this document, the MC module 37 can implement a two-pass approach in which predictive data is generated based on a non-tunable filter, and then the filter coefficients are adjusted so that the predictive data is set so as to be more similar to encoded video data. The adjusted filter coefficients in this case specify the filter coefficients that are used and encoded into the bit stream.

In order to code filter coefficients such that such filter coefficients can be transmitted as part of the bitstream, coefficient prediction and quantization module 41 may generate residual values associated with a set of filter coefficients based on predictive coding of a set of filter coefficients relative to filter coefficients associated with the non-tunable interpolation filter (242). In particular, coefficient prediction and quantization module 41 can subtract filter coefficients determined during adaptive interpolation filtering from corresponding filter coefficients associated with a non-tunable filter, such as an ITU-T H.264 interpolation filter or an ITU-T H.264 interpolation filter without intermediate rounding of half-pixel values. By encoding and transmitting the residual values, instead of transmitting the actual filtering coefficients, the amount of data transmitted in the bit stream can be reduced. In this case, the decoder can be programmed to know the way the filter coefficients are encoded.

Video encoder 50 may activate coefficient prediction and quantization module 41 to predict and quantize residual coefficients, and entropy encoding module 46 may entropy encode quantized residues (243). Video encoder 50 may then output residual values as part of the encoded bitstream (244). The quantization of residual values associated with filter coefficients may include quantization of residual values in which at least some of the residual values associated with different filter coefficients are assigned different levels of quantization. Thus, coefficient prediction and quantization module 41 may assign more quantization to larger residual coefficients and may assign less quantization to more accurate residual coefficients in order to achieve the desired balance of quantization and accuracy. When more quantization is used, more data is eliminated, but more compression can be achieved. Entropy encoding unit 46 may entropy encode residual values by assigning a prefix code, assigning a suffix code, and assigning a sign value to the residual values after quantizing the residual values. Video encoder 50 may then output residual values as part of the encoded bitstream (244).

27 is a conceptual diagram illustrating some illustrative filtering coefficients that can be predictively encoded. In this case, the filter coefficients O ₁ , O ₂ , O ₃ and O ₄ can specify the filter coefficients associated with the non-tunable interpolation filter. The filtering coefficients X ₁ , X ₂ , X ₃ and X ₄ can specify the desired filtering coefficients, which can be sufficiently similar to the filtering coefficients of a non-tunable interpolation filter. Accordingly, the filter coefficients X ₁ , X ₂ , X ₃ and X ₄ can be predicted based on the filter coefficients O ₁ , O ₂ , O ₃ and O ₄ , respectively. In particular, the first residue may be formed as the difference between X ₁ and O ₁ . Similarly, a second residue may be formed as the difference between X ₂ and O ₂ , a third residue may be formed as the difference between X ₃ and O ₃ , and a fourth residue may be formed as the difference between X ₄ and O ₄ . Residues may contain less data than the original filtering coefficients, thereby contributing to data compression.

In some cases, the set of filtering coefficients comprises a first set of filtering coefficients that specify only a part of the complete set of filtering coefficients associated with the adaptive interpolation filter, and the second set of filtering coefficients can be determined by a decoder based on the first set of filtering coefficients and symmetry by coefficients. For example, filter coefficients X ₁ and X ₂ may be predicted coded based on O ₁ and O ₂ , respectively. In this case, however, X ₃ and X ₄ may be symmetrical with X ₁ and X ₂ , and the decoder may be programmed to know that such symmetry overlaps. Accordingly, using symmetry, the coefficients X ₃ and X ₄ in this simple example can be excluded from the bit stream and can be calculated in the decoder based on the known symmetry by the coefficients after the coefficients X ₁ and X _{2 are} decoded with prediction.

25 is another flowchart illustrating a technique for encoding filter coefficients using predictive coding. In this case, however, two different types of forecasting are used. As shown in FIG. 25, the MC module 37 of the prediction module 32 identifies a set of filter coefficients for interpolating the prediction data in video encoding (251). As in the case of the process of FIG. 24, in FIG. 25, the MC module 37 can identify a set of filter coefficients by performing an adaptive interpolation filtering process. In this case, the MC module 37 may determine a set of filtering coefficients through the adaptive interpolation filtering process and generate predictive data based on the filtering coefficients identified in the adaptive interpolation filtering process. With adaptive interpolation filtering, the MC module 37 may implement a two-pass approach in which predictive data is generated based on a non-tunable filter, and then the filter coefficients are adjusted so that the predictive data is set to be more similar to the encoded video data. The adjusted filter coefficients in this case specify the filter coefficients that are used and encoded into the bit stream.

In order to code filter coefficients so that such filter coefficients can be transmitted as part of the bitstream, coefficient prediction and quantization module 41 may generate a first set of residual values associated with the first set of filter coefficients based on predictive coding for filter coefficients associated with the non-tunable interpolation filter (252). In particular, coefficient prediction and quantization module 41 can subtract the first set of filter coefficients from the corresponding filter coefficients associated with a non-tunable filter, such as an ITU-T H.264 interpolation filter or ITU-T H.264 interpolation filter without intermediate rounding of half-pixel values. Then, coefficient prediction and quantization module 41 may generate a second set of residual values associated with the second set of filtering coefficients based on predictive coding with respect to the first set of filtering coefficients (253). In particular, coefficient prediction and quantization module 41 may subtract a second set of filtering coefficients from mirror or rotated values of the first set of coefficients. Thus, the first set of coefficients is predicted encoded based on non-tunable filter coefficients, and the second set of coefficients are predicted encoded based on the first set of coefficients. By generating residual values instead of using the actual filter coefficients, the amount of data transmitted in the bitstream can be reduced. Furthermore, by using a non-tunable filter to predict the first set of coefficients, and then using the first set of coefficients to predict the second set of coefficients, additional data compression can be achieved relative to the prediction, which is based only on the non-tunable filter.

As in the case of the process of FIG. 24, in FIG. 25, video encoder 50 may activate coefficient prediction and quantization module 41 to predictively encode and quantize residual coefficients, and entropy encoding module 46 may entropy encode quantized residues (254). Video encoder 50 may then output residual values as part of the encoded bitstream (254). On the other hand, the quantization of residual values associated with filter coefficients may include quantization of residual values in which at least some of the residual values associated with different filter coefficients are assigned different levels of quantization. Thus, coefficient prediction and quantization module 41 may assign more quantization to larger residual coefficients and may assign less quantization to more accurate residual coefficients in order to achieve the desired balance of quantization and accuracy. Entropy encoding unit 46 may entropy encode residual values by assigning a prefix code, assigning a suffix code, and assigning a sign value to the residual values after quantizing the residual values. Video encoder 50 may then output residual values as part of the encoded bitstream (255).

и

могут упоминаться как модифицированные коэффициенты фильтрации, которые формируются посредством прибавления деквантованных остатков к коэффициентам прогнозной фильтрации O₁ и O₂.FIG. 28 is a conceptual diagram illustrating some illustrative filtering coefficients that can be predictively encoded according to the process of FIG. 25. In this case, the filter coefficients O ₁ and O ₂ can set the filter coefficients associated with the non-tunable interpolation filter. The filter coefficients X ₁ and X ₂ can specify the first set of required filter coefficients, which can be sufficiently similar to the filter coefficients of a non-tunable interpolation filter. Accordingly, the filter coefficients X ₁ and X ₂ can be predicted based on the filter coefficients O ₁ and O ₂ , respectively. In particular, the first residue may be formed as the difference between X ₁ and O ₁ , and the second residue may be formed as the difference between X ₂ and O ₂ . Residues may contain less data than the original filtering coefficients, thereby contributing to data compression. The residuals are then quantized by coefficient prediction and quantization module 41 and are entropy encoded by entropy encoding module 46.

and

may be referred to as modified filter coefficients, which are formed by adding dequantized residues to predictive filter coefficients O ₁ and O ₂ .

и

, которые задаются на основе коэффициентов X₁ и X₂. В частности, третий остаток может формироваться как разность между Z₁ и

, а четвертый остаток может формироваться как разность между Z₂ и

.

и

могут быть в большей степени аналогичными Z₁ и Z₂, чем O₁ и O₂, следовательно, посредством использования

и

для того, чтобы кодировать с прогнозированием Z₁ и Z₂, дополнительному сжатию данных можно способствовать.Then, the second set of filtering coefficients Z ₁ and Z ₂ can be predicted encoded based on the first set of coefficients X ₁ and X ₂ , for example, specifically from the coefficients

and

which are set based on the coefficients X ₁ and X ₂ . In particular, a third residue may be formed as the difference between Z ₁ and

, and the fourth residue can be formed as the difference between Z ₂ and

.

and

can be more similar to Z ₁ and Z ₂ than O ₁ and O ₂ , therefore, by using

and

in order to encode with prediction of Z ₁ and Z ₂ , additional data compression can be promoted.

26 is a flowchart illustrating a technique for decoding filter coefficients using predictive coding. FIG. 26 is described in terms of the video decoder 60 of FIG. 16. As shown, video decoder 60 receives the residual values associated with the set of filter coefficients (261). Video decoder 60 may entropy decode the residual values through entropy decoding module 52 and may activate coefficient prediction and inverse quantization module 53 to quantize the residual values (262), which are then sent to prediction module 55. Prediction module 56 generates a set of filter coefficients using predictive decoding of residual values (263).

In particular, prediction module 56 can generate a complete set of filter coefficients based on residual values and filter coefficients associated with a non-tunable interpolation filter, for example, as conceptually illustrated in FIG. 27 and allowed above in the context of coding. In some cases, a first set of filtering coefficients may be generated based on residual values and filtering coefficients associated with a non-tunable interpolation filter, and a second set of filtering coefficients may be generated based on symmetry. In other cases, the first set of filter coefficients may be generated based on the residual values and filter coefficients associated with the non-tunable interpolation filter, and the second set of filter coefficients may be formed based on additional residual values and the first set of filter coefficients, for example, as conceptually illustrated on Fig and permitted above in the context of encoding. In any case, the prediction unit 56 of the video decoder 60 applies a set of filtering coefficients to interpolate the predictive data used for predictively decoding the video blocks (264). In particular, prediction unit 56 filters the data to generate interpolated prediction data using prediction decoded filter coefficients so that video blocks can be decoded based on such interpolated prediction data.

On the other hand, a set of predicted decoded filter coefficients may comprise a first set of filter coefficients that specify only part of the complete set of filter coefficients associated with an adaptive interpolation filter. In this case, the coefficient prediction and inverse quantization module 53 of the video decoder 60 may generate a second set of filter coefficients based on the first set of filter coefficients and symmetry by coefficients and apply the first and second sets of filter coefficients to interpolate the predictive data.

In another case, the set of predicted filter coefficients may contain a first set of filter coefficients that specify only part of the complete set of filter coefficients associated with an adaptive interpolation filter. In this case, video decoder 60 may take additional residual values associated with a full set of filtering coefficients. Coefficient prediction and inverse quantization module 53 may generate a second set of filter coefficients using predictive decoding based on additional residual values and a first set of filter coefficients, and prediction module 55 may apply the first and second sets of filter coefficients to interpolate the predictive data.

In some cases, non-tunable interpolation filters based on H.264 / AVC filters can be used to predict one-dimensional filters (which may include filters for the sub-pixel positions a, b, d, h shown in Figs. 8, 9, 10 and 11). For two-dimensional filters, which include filters for the sub-pixel positions e, f, i and j shown in FIGS. 12, 13, 14 and 15, one of the following prediction schemes may be used:

.1. The prediction task is equal to zero (no forecasting),

.

, где

- это (i,j)-тый коэффициент фильтрации в среднем фильтре для субпикселной позиции SP.2. Using prediction based on a non-tunable filter, such as an average filter assembled for a training set, i.e.

where

is the ( i, j ) -th filter coefficient in the middle filter for the sub-pixel position SP .

3. Using possible symmetry in the coefficients and using the already coded coefficients to predict the remaining coefficients.

For one-dimensional filters, any of these three prediction methods can also be applied.

For Prediction FIG. 29 provides an example matrix of integer-pixel filtering coefficients for which prediction techniques may be used for coding. In this example, we can assume that the symmetry does not overlap the integer-pixel filter. Coefficients (h0,0), (h0,1), (h1,0), (h1,1), (h2,0), (h2,1) (h0,2), (h1,2) and (h2 , 2) can be quantized and encoded first. Then, the already encoded upper left coefficients (h0,0), (h0,1), (h1,0), (h1,1) and (h2,0) can be used to predict the right upper coefficients (h0,3 ), (h1,3), (h2,3), (h0,4), (h1,4) and (h2,4). Then, after the upper half of the filter coefficients (h0,0), (h0,1), (h0,2), (h0,3) (h0,4), (h1,0), (h1,1), (h1,2), (h1,3) and (h1,4) are quantized and encoded, they can additionally be used to predict the lower half of the filter coefficients (h3,0), (h3,1), (h3,2 ), (h3,3), (h3,4), (h4,0), (h4,1), (h4,2), (h4,3) and (h4,4). Prediction of other filtering coefficients can be performed in a similar manner. For example, for a sub-pixel position filter “e” (see FIG. 12), which may have a certain diagonal symmetry, the upper right coefficients can be first quantized and encoded and then used to predict the lower left coefficients.

In any case, after predicting the coefficients (for example, by means of the coefficient prediction module 41 and quantizing the prediction module 32), prediction errors are quantized (for example, by the coefficient prediction and quantization module 41). As described above, the so-called "uneven quantization" can be used. In this case, the accuracy of the quantization used by the coefficient prediction and quantization module 41 may depend on the location of the coefficient. It has been found that for coefficients with a smaller magnitude (which are typically coefficients at a greater distance from the center of the filter), better accuracy may be desirable. In contrast, for coefficients with a larger magnitude (which are typically coefficients at a shorter distance to the center of the filter), less accuracy is more desirable.

The following matrices, Q ^1D , Q ^2D , Q ^FP , can be used by coefficient prediction and quantization module 41 to indicate quantization accuracy for coefficients in one-dimensional filters, two-dimensional filters, and an integer-pixel filter, respectively. It should be noted that the number of bits indicated in the matrices may include 1 bit to encode the sign of the corresponding coefficients.

The coefficient prediction and quantization module 41 may encode the prediction errors of the quantized coefficients, i.e. residual coefficients based on a simple prefix encoding scheme. First, the dynamic range of the magnitude of the prediction error is divided into N resolution elements, for example, N = 6. If the dynamic range of the value is [0, ..., 2 ^g-1 -1], where q is the quantization accuracy for a given position of the coefficient (for example, indicated in the matrices above), then each resolution element n, n = 0, ..., N-1 may cover the following ranges:

In this case, the permission element b, b = 0 ..., N-1, to which the input quantity m belongs, can be encoded using a unary prefix code (which takes b + 1 bits). Then, the remainder of the value, mb _start , can be encoded using a suffix code of a fixed length of (g-N + b-1) bits. Finally, the prediction error sign is encoded using 1 bit.

For example, for the average coefficient in a one-dimensional filter, an accuracy of 9 bits can be used by coefficient prediction and quantization module 41 to quantize the prediction error, i.e. g = 9, of which 8 bits can be used to quantize the absolute value of the error. If the absolute value of the error is 97, then it is within the 4th resolution element, i.e. b = 4. Therefore, the unary prefix code “11110” can be encoded to indicate the 4th permission element, and the remainder mb _start = 97-64 = 33 can be encoded using a fixed-length suffix code in (g-N + b-1) = (9-6 + 4-1) = 6 bits. Finally, bit “0” may be encoded to indicate a positive sign.

The technologies of the present disclosure can be implemented in a wide range of devices or devices, including a cordless handset and integrated circuit (IC) or IC set (i.e., chipset). All described components, modules or assemblies are provided in order to emphasize functional aspects, and do not necessarily require implementation through various hardware modules.

The technologies described herein may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, any features described as modules, nodes, or components can be implemented together in an integrated logic device or separately as discrete but interacting logic devices. When implemented in software, technologies can be implemented at least in part by a computer-readable medium containing instructions that, when executed on a processor, perform one or more of the methods described above, the computer-readable medium may comprise a computer-readable storage medium, and may be part of a computer software product, which may include packaging. Computer-readable storage media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash - memory, magnetic or optical storage media, etc. Additionally or alternatively, technologies can be implemented, at least in part, through a computer-readable communication medium that transfers or transmits code in the form of instructions or data structures, and which can be accessed, read, or executed by computer.

The code may be executed by one or more processors, for example, one or more digital signal processors (DSPs), general purpose microprocessors, specialized integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term “processor” as used herein may mean any of the above structure or another structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within specialized software modules or hardware modules configured for encoding or decoding or integrated in a combination video codec. In addition, technologies can be fully implemented in one or more circuits or logic elements.

Various aspects of the disclosure of the invention are described. These and other aspects are within the scope of the attached claims.

Claims (47)

1. A method of video coding, comprising stages in which:
generating many different versions of predictive data for encoding video data based on many different interpolation filters;
choose from a variety of different versions of predictive data based on the analysis of distortion depending on the transmission speed;
encoding the video data based on the selection; and
encode syntax to indicate a choice, a separate syntax element being determined for each of the plurality of sub-pixel locations. 2. The method according to claim 1, in which the formation and selection comprise stages in which:
generating first interpolated predictive data for encoding video data based on the first interpolation filter;
generating second interpolated predictive data for encoding video data based on the second interpolation filter; and
choose between the first interpolated predictive data and the second interpolated predictive data based on the analysis of distortion depending on the transmission speed. 3. The method according to claim 2, in which the first interpolation filter corresponds to a non-tunable interpolation filter. 4. The method according to claim 3, in which the first interpolation filter is made without intermediate rounding of fractional-pixel values, which are used to form other fractional-pixel values. 5. The method according to claim 2, in which the first interpolation filter corresponds to a non-tunable interpolation filter, and the second interpolation filter corresponds to an adaptive interpolation filter. 6. The method according to claim 2, in which the syntax contains a flag that determines whether the first interpolation filter or the second interpolation filter should be used by the decoder. 7. The method of claim 2, wherein a separate syntax element is defined for each of the plurality of subpixel locations for the coded unit to indicate whether a first interpolation filter or a second interpolation filter should be used by a decoder for each of the plurality of subpixel locations. 8. The method of claim 1, wherein a separate syntax element is defined for integer-pixel locations of the video data to indicate whether the first filter or second filter should be used by the decoder for integer-pixel locations. 9. The method according to claim 7, in which the first filter does not apply filtering, and the second filter is associated with adaptive filtering coefficients. 10. The method of claim 2, further comprising transmitting the encoded video data and syntax to indicate whether the first interpolated predictive data or the second interpolated predictive data has been used to generate encoded video data. 11. The method of claim 10, further comprising the steps of: transmitting filter coefficients when the second interpolation filter is adaptive, and the syntax indicates that the second interpolated prediction data has been used to generate encoded video data; and filter coefficients are not transmitted when the first interpolation filter is non-tunable, and the syntax indicates that the first interpolated predictive data was used to generate encoded video data. 12. The method according to claim 2, in which the choice between the first interpolated predictive data and the second interpolated predictive data based on the analysis of distortion depending on the transmission speed contains the steps in which: calculate the first function of the cost of distortion depending on the transmission speed associated with video data if the video data is encoded by first interpolated predictive data; calculating a second distortion cost function depending on the transmission rate associated with the video data if the video data is encoded by the second interpolated predictive data; and choosing between the first interpolated predictive data and the second interpolated predictive data based on the first and second distortion cost functions depending on the transmission rate. 13. A video coding device comprising a video encoder, which:
generates many different versions of predictive data for encoding video data based on many different interpolation filters;
selects from a variety of different versions of predictive data based on an analysis of distortion depending on the transmission rate;
encodes video data based on the selection; and
encodes syntax to indicate a choice, with a separate syntax element being determined for each of the plurality of subpixel locations. 14. The device according to item 13, in which the video encoder: generates the first interpolated predictive data for encoding video data based on the first interpolation filter; generates second interpolated predictive data for encoding video data based on the second interpolation filter; and selects between the first interpolated predictive data and the second interpolated predictive data based on the analysis of distortion depending on the transmission speed. 15. The device according to 14, in which the first interpolation filter corresponds to a non-tunable interpolation filter. 16. The device according to clause 15, in which the non-tunable interpolation filter is made without intermediate rounding of fractional-pixel values, which are used to form other fractional-pixel values. 17. The device according to 14, in which the first interpolation filter corresponds to a non-tunable interpolation filter, and the second interpolation filter corresponds to an adaptive interpolation filter. 18. The device according to 14, in which the syntax contains a flag that determines whether the first interpolation filter or the second interpolation filter should be used by the decoder. 19. The apparatus of claim 14, wherein a separate syntax element is defined for each of the plurality of subpixel locations for the coded unit to indicate whether a first interpolation filter or a second interpolation filter should be used by a decoder for each of the plurality of subpixel locations. 20. The device according to item 13, in which a separate syntax element is defined for integer-pixel locations of the video data to indicate whether the first filter or the second filter should be used by the decoder for integer-pixel locations. 21. The device according to claim 20, in which the first filter does not apply filtering, and the second filter is associated with adaptive filtering coefficients. 22. The apparatus of claim 14, further comprising a transmitter that transmits the encoded video data and syntax to indicate whether the first interpolated predictive data or the second interpolated predictive data has been used to generate encoded video data. 23. The device according to item 22, in which the transmitting device: transmits filtering coefficients when the second interpolation filter is adaptive, and the syntax indicates that the second interpolated predictive data was used to generate encoded video data; and does not transmit filter coefficients when the first interpolation filter is non-tunable, and the syntax indicates that the first interpolated prediction data was used to generate encoded video data. 24. The device according to 14, in which when choosing between the first interpolated predictive data and the second interpolated predictive data based on the analysis of distortion depending on the transmission speed, the device: calculates the first function of the cost of distortion depending on the transmission speed associated with the video data, if video data is encoded by first interpolated predictive data; calculates a second distortion cost function as a function of the transmission rate associated with the video data if the video data is encoded by second interpolated predictive data; and selects between the first interpolated predictive data and the second interpolated predictive data based on the first and second distortion cost functions depending on the transmission rate. 25. The device according to item 13, in which the video encoder contains an integrated circuit. 26. The device according to item 13, in which the video encoder contains a microprocessor. 27. The device according to item 13, and the device is a wireless communication device that includes a video encoder. 28. A video coding device comprising:
means for generating a plurality of different versions of predictive data for encoding video data based on a plurality of different interpolation filters;
means for selecting from a variety of different versions of predictive data based on an analysis of distortion depending on the transmission rate;
means for encoding video data based on the selection; and
means for syntax encoding to indicate a choice, a separate syntax element being determined for each of the plurality of sub-pixel locations. 29. The device according to p. 28, in which the means for forming and means for choosing contain: means for generating the first interpolated predictive data for encoding video data based on the first interpolation filter; means for generating second interpolated predictive data for encoding video data based on the second interpolation filter; and means for choosing between the first interpolated predictive data and the second interpolated predictive data based on a distortion analysis depending on a transmission rate. 30. The device according to clause 29, in which the first interpolation filter corresponds to a non-tunable interpolation filter. 31. The device according to item 30, in which the non-tunable interpolation filter is made without intermediate rounding of fractional-pixel values, which are used to form other fractional-pixel values. 32. The device according to clause 29, in which the first interpolation filter corresponds to a non-tunable interpolation filter, and the second interpolation filter corresponds to an adaptive interpolation filter. 33. The device according to clause 29, in which the syntax contains a flag that determines whether the first interpolation filter or the second interpolation filter should be used by the decoder. 34. The apparatus of claim 29, wherein a separate syntax element is defined for each of the plurality of subpixel locations for the coded unit to indicate whether a first interpolation filter or a second interpolation filter should be used by a decoder for each of the plurality of subpixel locations. 35. The apparatus of claim 28, further comprising means for generating a separate syntax element for integer-pixel locations of the video data to indicate whether the first filter or second filter should be used by the decoder for integer-pixel locations. 36. The device according to clause 35, in which the first filter does not apply filtering, and the second filter is associated with adaptive filtering coefficients. 37. The apparatus of claim 29, further comprising means for transmitting the encoded video data and syntax to indicate whether the first interpolated predictive data or the second interpolated predictive data has been used to generate encoded video data. 38. The device according to clause 37, further comprising: means for transmitting filter coefficients when the second interpolation filter is adaptive, and the syntax indicates that the second interpolated predictive data was used to generate encoded video data; and means for canceling the transmission of filter coefficients when the first interpolation filter is non-tunable, and the syntax indicates that the first interpolated prediction data was used to generate encoded video data. 39. The device according to clause 29, in which the means for choosing between the first interpolated predictive data and the second interpolated predictive data based on the analysis of distortion depending on the transmission speed contains: means for calculating the first function of the cost of distortion depending on the transmission speed associated with video data if the video data is encoded by first interpolated predictive data; means for calculating a second distortion cost function as a function of the transmission rate associated with the video data if the video data is encoded by second interpolated predictive data; and means for choosing between the first interpolated predictive data and the second interpolated predictive data based on the first and second distortion cost functions depending on the transmission rate. 40. A computer-readable storage medium containing instructions that, when executed by a processor, instruct the processor:
generate many different versions of predictive data for encoding video data based on many different interpolation filters;
choose from many different versions of predictive data based on an analysis of distortion depending on the transmission speed;
Encode video based on selection; and
encode syntax to indicate a choice, with a separate syntax element being defined for each of the plurality of subpixel locations. 41. The computer-readable storage medium according to claim 40, wherein the instructions instruct the processor to: generate first interpolated predictive data for encoding video data based on the first interpolation filter; generate second interpolated predictive data for encoding video data based on the second interpolation filter; and choose between the first interpolated predictive data and the second interpolated predictive data based on the analysis of distortion depending on the transmission speed. 42. The computer-readable storage medium according to paragraph 41, wherein the first interpolation filter corresponds to a non-tunable interpolation filter and the second interpolation filter corresponds to an adaptive interpolation filter. 43. The computer-readable storage medium according to paragraph 41, wherein the syntax comprises a flag that determines whether the first interpolation filter or the second interpolation filter is used by the decoder. 44. The computer-readable storage medium of claim 41, wherein a separate syntax element is defined for each of the plurality of subpixel locations for the encoded unit to indicate whether a first interpolation filter or a second interpolation filter should be used by a decoder for each of the plurality of subpixel locations. 45. The computer-readable storage medium according to paragraph 41, wherein the instructions instruct the processor to activate the transmission of encoded video data and syntax to indicate whether the first interpolated predictive data or the second interpolated predictive data has been used to generate encoded video data. 46. The computer-readable storage medium according to item 45, further containing instructions that instruct the processor to activate the transmission of: filter coefficients when the second interpolation filter is adaptive, and the syntax indicates that the second interpolated predictive data was used to generate encoded video data; and without filtering coefficients, when the first interpolation filter is non-tunable, and the syntax indicates that the first interpolated predictive data was used to generate encoded video data. 47. The computer-readable storage medium according to paragraph 41, and when choosing between the first interpolated predictive data and the second interpolated predictive data based on the analysis of distortion depending on the transmission speed, the instructions instruct the processor: to calculate the first function of the cost of distortion depending on the transmission speed, associated with the video data, if the video data is encoded by the first interpolated predictive data; calculate a second distortion cost function as a function of the transmission rate associated with the video data if the video data is encoded by second interpolated predictive data; and choose between the first interpolated predictive data and the second interpolated predictive data based on the first and second distortion cost functions depending on the transmission rate.

Images