RU2787217C1 - Method and device for interpolation filtration for encoding with prediction - Google Patents

Abstract

FIELD: video encoding.

SUBSTANCE: invention relates to video encoding means. A reference sample is obtained. A value of subpixel displacement is obtained. Filtration of the reference sample is performed using 4-diverter interpolation filter to obtain a predicted sample value, wherein filtration coefficients of 4-diverter interpolation filter are determined in the following way: c₀=16 -

, c₁=16 + c0, c₂=c0 + p, c₃=

, wherein a division operation is performed with rounding downwards. At the same time, p is a fractional part of the subpixel displacement value, and c₀, c₁, c₂, and c₃ are filtration coefficients.

EFFECT: increase in the efficiency of video encoding.

12 cl, 27 dwg

Description

The technical field to which the invention belongs

Embodiments of the present disclosure (invention) generally relate to the field of image processing and, in particular, to the encoding and decoding of an image and/or video, and, in particular, to an intra/inter prediction method and apparatus.

State of the art

Video encoding (video encoding and decoding) is used in a wide range of digital video applications, such as digital television broadcast, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat, video conferencing, DVD and Blu-ray discs. , video content collection and editing systems and security camera applications.

The amount of video data required to display even a relatively short video can be significant, which can make it difficult to transfer data over a communication network with limited bandwidth. Thus, in modern communication networks, video data is usually compressed prior to transmission. The size of the video also matters because memory resources may be limited to store the video on the storage device. Video compression devices often use source software and/or hardware to encode video data prior to transmission or storage, thereby reducing the amount of data required to represent digital video images. The compressed data is then received at the destination by a video decompressor, which decodes the video data. Due to limited network resources and ever-increasing demands for higher video quality, advanced compression and decompression technologies are required that increase the compression ratio with little or no degradation in image quality.

Disclosure of the essence of the invention

Embodiments of the present disclosure provide devices and methods for encoding and decoding in accordance with the independent claims.

The above and other objects of the present disclosure are solved by independent claims. Additional forms of implementation are obvious from the dependent claims, description and drawings.

Embodiments of the invention are defined by the features of the independent claims and further preferred embodiments of the embodiments by the features of the dependent claims.

According to an aspect of the present disclosure, a method for intra or inter prediction of video frame processing is provided, comprising the steps of: obtaining a reference sample; obtaining a sub-pixel offset value; and filtering, using the 4-tap sub-pixel interpolation filter, a reference sample to obtain a predicted sample value in which the filter coefficients of the 4-tap sub-pixel interpolation filter satisfy:

,

,

и

являются коэффициентами фильтрации 4-отводного интерполяционного фильтра субпикселя.in which p is the fractional part of the sub-pixel offset value and

,

,

and

are the filter coefficients of the 4-tap sub-pixel interpolation filter.

In one exemplary implementation, reference samples may be denoted ref[x], with

ref[ x ] = p[ −1 − refIdx + x ][ −1 − refIdx ], with x = 0..nTbW + refIdx + 1.

the corresponding array of reference samples and "p" referring to the x-y two-dimensional array p[ x ][ y ] containing the values of the sample. The number of reference samples used may be less than one. In another example, the number of reference samples may be four.

In one exemplary implementation, the resulting sub-pixel offset value may be denoted as

(y + 1 + refIdx ) * intraPredAngle

where "IntraPredAngle" is an intra prediction angle value.

In one exemplary implementation, the predicted sample value "predSamples[ x ][ y ]" can be obtained as follows

) + 32 ) >> 6 )predSamples[ x ][ y ] = Clip1( ( (

) + 32 ) >> 6 )

where fT[ i ] refers to the filter coefficients. These coefficients may be luminance or chrominance filter coefficients for inter prediction related to fG and fC, respectively.

The choice of filter coefficients, both luma and chrominance, can be implemented by using and setting the "flterFlag" flag, for example, as

fT[j] = filterFlag ? fG[ iFact ][ j ] : fC[ iFact ][ j ]

With

iFact = ( ( y + 1 + refIdx ) * intraPredAngle ) & 31.

The value "31" refers to the fractional part of the sub-pixel offset value and can take other values than "31". The values of the filter coefficients fG (luminance) and/or fC (chrominance) are obtained analytically at runtime using the above analytical expression for the filter coefficients of the 4-tap filter. Thus, the filter coefficients are determined according to the sub-pixel offset value.

Therefore, the filter coefficients are obtained without access to the corresponding filter coefficient values from the LUT, but rather are obtained by calculation.

Alternatively, the filter coefficients can be calculated using the above equations and the values can be stored in the LUT.

According to an aspect of the present disclosure, the filter coefficients of a 4-tap sub-pixel interpolation filter are defined in a table as follows:

P

0 sixteen 32 sixteen 0 1 sixteen 32 sixteen 0 2 15 31 17 1 3 15 31 17 1 four fourteen thirty eighteen 2 five fourteen thirty eighteen 2 6 13 29 nineteen 3 7 13 29 nineteen 3 8 12 28 20 four nine 12 28 20 four 10 eleven 27 21 five eleven eleven 27 21 five 12 10 26 22 6 13 10 26 22 6 fourteen nine 25 23 7 15 nine 25 23 7 sixteen 8 24 24 8 17 8 24 24 8 eighteen 7 23 25 nine nineteen 7 23 25 nine 20 6 22 26 10 21 6 22 26 10 22 five 21 27 eleven 23 five 21 27 eleven 24 four 20 28 12 25 four 20 28 12 26 3 nineteen 29 13 27 3 nineteen 29 13 28 2 eighteen thirty fourteen 29 2 eighteen thirty fourteen thirty 1 17 31 15 31 1 17 31 15

According to an aspect of the present disclosure, a method for intra or inter prediction of video frame processing is provided, comprising the steps of: obtaining a reference sample; obtaining a sub-pixel offset value; and filtering using a 4-tap sub-pixel interpolation filter, a reference sample to obtain a predicted sample value, in which the filter coefficients of the 4-tap sub-pixel interpolation filter are determined as follows:

p

0 sixteen 32 sixteen 0 1 sixteen 32 sixteen 0 2 15 31 17 1 3 15 31 17 1 four fourteen thirty eighteen 2 five fourteen thirty eighteen 2 6 13 29 nineteen 3 7 13 29 nineteen 3 8 12 28 20 four nine 12 28 20 four 10 eleven 27 21 five eleven eleven 27 21 five 12 10 26 22 6 13 10 26 22 6 fourteen nine 25 23 7 15 nine 25 23 7 sixteen 8 24 24 8 17 8 24 24 8 eighteen 7 23 25 nine nineteen 7 23 25 nine 20 6 22 26 10 21 6 22 26 10 22 five 21 27 eleven 23 five 21 27 eleven 24 four 20 28 12 25 four 20 28 12 26 3 nineteen 29 13 27 3 nineteen 29 13 28 2 eighteen thirty fourteen 29 2 eighteen thirty fourteen thirty 1 17 31 15 31 1 17 31 15

,

,

и

являются коэффициентами фильтра.where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients.

In one exemplary implementation, filter coefficient values may be stored in a LUT. This means that for sub-pixel interpolation filtering, the corresponding values of the filter coefficients must be obtained from the LUT.

с i = 0, … 3 фильтра. Это называется 4-отводным фильтром и/или 4-точечным фильтром.Fig. 23 shows a flowchart of an intra/inter prediction process. At 2301, a reference sample is obtained. The reference sample may include one or more reference samples. At step 2302, a sub-pixel offset value is obtained. In step 2303, the reference sample is then filtered using a sub-pixel interpolation filter having coefficients

with i = 0, … 3 filters. This is called a 4-tap filter and/or a 4-point filter.

The filter coefficients can be obtained from analytical expressions with a given fractional part p of the sub-pixel offset value. The filter coefficients can also be obtained from the LUT with reference to p values.

According to an aspect of the present disclosure, the p bit is set to zero.

According to an aspect of the present disclosure, p is incremented by 1 and then the least significant bit of the incremented p is set to zero prior to using that value to derive the filter coefficients.

According to an aspect of the present disclosure, obtaining the filter coefficients comprises the step of calculating the filter coefficients in accordance with

фильтра из предшествующего коэффициента

фильтра. Это может дополнительно сократить время для расчета коэффициентов фильтра.This may provide the advantage of calculating the coefficient

filter from previous coefficient

filter. This can further reduce the time to calculate the filter coefficients.

According to an aspect of the present disclosure, a video coding method is provided, the method comprising a process of intra prediction of a video frame block, comprising: filtering, using a 4-tap sub-pixel interpolation filter, a video frame reference sample; wherein the 4-tap sub-pixel interpolation filter is a combination of filters, with filter coefficients of at least one of the filters satisfying:

,

,

и

являются коэффициентами фильтра.where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients.

Fig. 24 shows a flowchart of a video encoding process. In step 2401, an intra prediction process is performed by filtering the block reference sample using a 4-tap sub-pixel interpolation filter.

According to an aspect of the present disclosure, the 4-tap interpolation filter is an anti-aliasing filter that is a parallel combination of two phase dependent linear filters and a low pass filter.

According to an aspect of the present disclosure, the 4-tap sub-pixel interpolation filter is a sharpening filter that is a parallel combination of one phase-dependent linear filter and a power-varying high-pass filter.

According to an aspect of the present invention, the power-changing high-pass filter is a phase-independent finite impulse response high-pass filter FIR whose output is multiplied by a phase-dependent power parameter.

According to any of the preceding aspects of the present disclosure, the least significant bit of p is set to zero.

According to an aspect of the present disclosure, p is incremented by 1 and then the least significant bit of the incremented value is set to zero prior to using that value to derive the interpolation filter coefficients.

According to an aspect of the present disclosure, a clipping operation is performed for the output of a power varying high pass filter.

According to an aspect of the present disclosure, an apparatus for intra or inter prediction of a video frame is provided, including: a reference sample acquisition unit configured to acquire a reference sample; a sub-pixel offset value obtaining unit, configured to obtain sub-pixel offset values; and a 4-tap sub-pixel interpolation filter configured to filter the reference sample to obtain a predicted sample value in which the filter coefficients of the 4-tap sub-pixel interpolation filter satisfy:

,

,

и

являются коэффициентами фильтра 4-отводного интерполяционного фильтра субпикселя.where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients of the 4-tap sub-pixel interpolation filter.

According to an aspect of the present disclosure, the filter coefficients of a 4-tap sub-pixel interpolation filter are defined in a table as follows:

p

0 sixteen 32 sixteen 0 1 sixteen 32 sixteen 0 2 15 31 17 1 3 15 31 17 1 four fourteen thirty eighteen 2 five fourteen thirty eighteen 2 6 13 29 nineteen 3 7 13 29 nineteen 3 8 12 28 20 four nine 12 28 20 four 10 eleven 27 21 five eleven eleven 27 21 five 12 10 26 22 6 13 10 26 22 6 fourteen nine 25 23 7 15 nine 25 23 7 sixteen 8 24 24 8 17 8 24 24 8 eighteen 7 23 25 nine nineteen 7 23 25 nine 20 6 22 26 10 21 6 22 26 10 22 five 21 27 eleven 23 five 21 27 eleven 24 four 20 28 12 25 four 20 28 12 26 3 nineteen 29 13 27 3 nineteen 29 13 28 2 eighteen thirty fourteen 29 2 eighteen thirty fourteen thirty 1 17 31 15 31 1 17 31 15

According to an aspect of the present disclosure, a device for intra or inter prediction of a video frame is provided, comprising: a reference sample acquisition unit configured to obtain a reference sample; a sub-pixel offset value obtaining unit, configured to obtain the sub-pixel offset value; and a 4-tap sub-pixel interpolation filter configured to filter the reference sample to obtain a predicted sample value, wherein the filter coefficients of the 4-tap sub-pixel interpolation filter are determined as follows:

p

0 sixteen 32 sixteen 0 1 sixteen 32 sixteen 0 2 15 31 17 1 3 15 31 17 1 four fourteen thirty eighteen 2 five fourteen thirty eighteen 2 6 13 29 nineteen 3 7 13 29 nineteen 3 8 12 28 20 four nine 12 28 20 four 10 eleven 27 21 five eleven eleven 27 21 five 12 10 26 22 6 13 10 26 22 6 fourteen nine 25 23 7 15 nine 25 23 7 sixteen 8 24 24 8 17 8 24 24 8 eighteen 7 23 25 nine nineteen 7 23 25 nine 20 6 22 26 10 21 6 22 26 10 22 five 21 27 eleven 23 five 21 27 eleven 24 four 20 28 12 25 four 20 28 12 26 3 nineteen 29 13 27 3 nineteen 29 13 28 2 eighteen thirty fourteen 29 2 eighteen thirty fourteen thirty 1 17 31 15 31 1 17 31 15

,

,

и

являются коэффициентами фильтра.where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients.

Fig. 25 shows a diagram of an intra/inter prediction processing unit 2500, which includes a reference sample acquisition unit 2510, a subpixel offset value acquisition unit 2520, and a subpixel interpolation filter 2530.

According to an aspect of the present disclosure, a device for encoding a video is provided, comprising: an intra prediction processing unit configured to perform intra prediction of a block of a video frame; and a 4-tap sub-pixel interpolation filter configured to filter a video frame reference sample; wherein the 4-tap sub-pixel interpolation filter is a combination of filters, with filter coefficients of at least one of the filters, satisfying:

,

,

и

являются коэффициентами фильтра.where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients.

Fig. 26 shows a diagram of a video encoding unit 2600, which includes an intra prediction processing unit 2610 and a sub-pixel interpolation filter 2620.

According to an aspect of the present disclosure, an encoder (20) is provided, containing a processing circuit for performing the method according to any of the preceding aspects of the present disclosure.

According to an aspect of the present disclosure, a decoder (30) is provided, comprising a processing circuit for performing a method according to any of the preceding aspects of the present disclosure.

According to an aspect of the present disclosure, a computer program product is provided that contains program code for performing a method in accordance with any of the preceding aspects of the present disclosure.

According to an aspect of the present disclosure, a decoder is provided, comprising: one or more processors; and a persistent computer-readable medium coupled to the processors and storing programs for execution by the processors, wherein the programs, when executed by the processors, configure a decoder to execute a method in accordance with any of the preceding aspects of the present disclosure.

According to an aspect of the present disclosure, an encoder is provided, comprising: one or more processors; and a persistent computer-readable medium coupled to the processors and storing programs for execution by the processors, wherein the programs, when executed by the processors, configure an encoder to execute a method in accordance with any of the preceding aspects of the present disclosure.

The present invention according to any of the preceding aspects can provide a more efficient method for intra/inter prediction of a video frame. This is due to the fact that the filter coefficients of the interpolation filter are obtained in an analytical way, that is, the said coefficients are calculated during processing. This avoids storing the coefficients in a lookup table (LUT), requiring access time to read the coefficients from memory.

фильтра являются линейными на позиции р дробной выборки, и используются операции деления на 2, соответствующая операция может быть эффективно выполнена, используя быструю низкоуровневую битовую операцию. Соответствующее время для выполнения битовой операции и для расчета коэффициентов фильтра короче времени доступа к хранимым коэффициентам из LUT. Таким образом, задержка обработки сокращается.Thus, the prediction becomes more efficient and requires less memory. It also provides an inexpensive implementation of the prediction. Moreover, since the analytic coefficients

filters are linear at position p of a fractional sample, and divide-by-2 operations are used, the corresponding operation can be efficiently performed using a fast low-level bit operation. The corresponding time to perform a bit operation and to calculate the filter coefficients is shorter than the time to access the stored coefficients from the LUT. Thus, the processing delay is reduced.

Moreover, the specific analytical structure of the filter coefficients may provide the advantage of implementing a low complexity filter block(s). As previously described, the filter's frequency response (ie, frequency response) for different subpixel positions follows the amplitude-to-phase relationship and prevents artifacts in the frequency response, particularly at high frequencies. The linearity of the filter coefficients can provide the benefit of equipment reuse.

The following is a detailed description of one or more embodiments with reference to the accompanying drawings. Other features, objects and advantages will be apparent from the description, drawings and claims.

The invention may be implemented in hardware and/or software.

For purposes of clarity, any of the above embodiments may be combined with any or more of the other above embodiments to provide a new embodiment within the scope of this disclosure.

These and other features will be better understood from the following detailed description with reference to the accompanying drawings and claims.

Brief description of the drawings

The following is a detailed description of embodiments of the invention with reference to the accompanying drawings, in which:

fig. 1A is a block diagram showing an example of a video coding system capable of implementing embodiments of the invention;

fig. 1B is a block diagram showing another example of a video coding system capable of implementing embodiments of the invention;

fig. 2 is a block diagram showing an example of a video encoder capable of implementing embodiments of the invention;

fig. 3 is a block diagram showing an exemplary structure of a video encoder capable of implementing embodiments of the invention;

fig. 4 is a block diagram illustrating an example of an encoding device or a decoding device;

fig. 5 is a block diagram illustrating another example of an encoding device or a decoding device;

fig. 6 is a diagram showing intra prediction angular directions and their associated intra prediction modes in HEVC;

fig. 7 is a diagram showing intra prediction angular directions and associated intra prediction modes in JEM;

fig. 8 is a diagram showing intra prediction angular directions and associated intra prediction modes in the VTM-3.0 and VVC draft v.3 specifications;

fig. 9 is a diagram showing an example of a filter combination for smoothing, where a set of reference samples is input to the interpolation filter;

fig. 10 is a diagram showing an exemplary filter combination for sharpening interpolation filtering, including a clipping operation;

fig. 11 is a diagram showing an exemplary filter combination for sharpening interpolation filtering with an alternative clipping position;

fig. 12A is a diagram showing an exemplary filter combination for sharpening interpolation filtering using a variable power filter;

fig. 12B is a diagram showing another exemplary filter combination for sharpening interpolation filtering using a variable power filter;

fig. 13-21 are diagrams showing various combinations of LUTs and analytical representations of interpolation filter coefficients;

fig. 22 is a diagram showing an exemplary implementation of the coefficient calculation method;

fig. 23 is a flowchart of an intra/inter prediction process;

fig. 24 is a block diagram of a video coding algorithm;

fig. 25 is a block diagram of an intra/inter prediction processing unit including a reference sample acquisition unit, a subpixel offset acquisition unit, and a subpixel interpolation filter;

fig. 26 is a block diagram of a video encoding unit including an inter prediction processing unit and a sub-pixel interpolation filter.

In the following description, identical reference numerals refer to identical or at least functionally equivalent features, unless expressly indicated otherwise.

Implementation of the invention

In the following description, reference is made to the accompanying drawings, which form part of the present disclosure and show, by way of illustration, specific aspects of embodiments of the invention or specific aspects in which embodiments of the present disclosure may be used. Obviously, embodiments of the invention can be used in other aspects and contain structural or logical changes not shown in the drawings. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

For example, it is obvious that the present invention with the described method can also be true for the corresponding device or system configured to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more blocks, such as functional blocks, to perform the one or more method steps described (for example, one block performing one or more steps, or a plurality of blocks , each of which performs one or more of a plurality of steps) even if such one or more units are not explicitly described and illustrated in the drawings. On the other hand, for example, if a particular device is described in terms of one or more blocks, such as functional blocks, the corresponding method may include one step to perform the functionality of one or more blocks (for example, one step to perform the functionality of one or more blocks). blocks, or a plurality of steps, each of which performs the functionality of one or more of the plurality of blocks) even if such one or more steps is not explicitly described or illustrated in the drawings. In addition, it is understood that features of the various exemplary embodiments and/or aspects described herein may be combined with one another unless specifically noted otherwise.

Video coding generally refers to the processing of a sequence of images that form a video or video sequence. Instead of the term "image", the term "frame" or "image" can be used as synonyms in the field of video coding. The video coding used in the present invention (or the present disclosure) indicates either video coding or video decoding. Video encoding is performed at the source side, typically involving processing (eg, by compressing) the source video images to reduce the amount of data needed to represent the video images (for more efficient storage and/or transmission). Video decoding is performed on the destination side and usually contains inverse processing compared to the encoder for recovering video images. Embodiments relating to "encoding" video images (or images in general, as will be explained later) should be understood to refer to either "encoding" or "decoding" a video sequence. The combination of an encoding part and a decoding part is also called CODEC (Coding and Decoding).

In the case of lossless video coding, the original video images can be reconstructed, that is, the reconstructed video images have the same quality as the original video images (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, additional compression is performed, for example by quantization, to reduce the amount of data representing video images that cannot be fully reconstructed in the decoder, i.e., the quality of the reconstructed video images is lower or worse compared to the quality of the original video images.

Several video coding standards belong to the group of "hybrid lossy video codecs" (ie, combine spatial and temporal sample-domain prediction and 2D transform coding to apply transform-domain quantization). Each picture of a video sequence is usually divided into a set of non-overlapping blocks, and encoding is usually done at the block level. In other words, in the encoder, video is usually processed, that is, encoded, at the block (video block) level, for example, using spatial (inner picture) prediction and temporal (outer picture) prediction to generate a prediction block, subtracting the prediction block from the current block ( block currently being processed/to be processed) to obtain a residual block, transforming the residual block, and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compressed), while the decoder partially performs inverse processing compared to with the encoder to the encoded or compressed block to reconstruct the current block for presentation. In addition, the encoder duplicates the processing cycle of the decoder so that both will generate identical predictions (eg, intra and inter prediction) and/or re-restore for processing, ie, encoding, subsequent blocks.

In the following embodiments, with reference to FIG. 1-3, the encoder 20, the decoder 30 and the encoding system 10 are described.

Fig. 1A is a block diagram illustrating an example of a coding system 10, such as a video coding system 10 (or coding system 10 for short), that may use the methods of this present disclosure. Video encoder 20 (eg, encoder 20) and video decoder 30 (eg, decoder 30) of video coding system 10 are examples of devices that can be configured to perform methods in accordance with the various examples described in the present invention.

As shown in FIG. 1A, the encoding system 10 includes a source device 12 configured to provide encoded image data 21 to, for example, destination device 14 for decoding encoded image data 13.

The source device 12 includes an encoder 20 and may additionally, i. e. possibly contain an image source 16, a preprocessor 18 (pre-processor), for example, an image preprocessor 18 and a communication interface or communication unit 22.

Image source 16 may comprise or be any type of image capture device, such as for capturing a real image and/or any kind of image or commentary (for screen encoding, some texts on the screen are also considered part of the image or image to be encoded), device generating, such as a computer graphics processor for generating a computer animated image, or any device for receiving and/or rendering a real image, a computer animated image (e.g., screen content, a virtual reality (VR) image), and/or any combination thereof (e.g., an image augmented reality (AR)). The image source can be any type of memory or storage that stores any of the above images.

Unlike the preprocessor 18 and the processing performed by the preprocessor 18, the image or image data 17 may also be referred to as raw image or original image data 17.

The preprocessor 18 is configured to receive (raw) image data 17 and perform pre-processing of the image data 17 to obtain a pre-processed image 19 or pre-processed image data 19. Pre-processing is performed by the preprocessor 18, which includes, for example, cropping, color format conversion (eg from RGB to YCbCr), color correction or denoising. Obviously, preprocessor 18 could be a possible component.

Video encoder 20 is configured to receive pre-processed image data 19 and provide encoded image data 21 (more details will be described below, for example, with reference to FIG. 2).

Communication interface 22 of source device 12 may be configured to receive encoded image data 21 and transmit encoded image data 21 (or any further processed version thereof) over communication channel 13 to another device, such as destination device 14 or any other device for storage or direct recovery.

Destination device 14 includes a decoder 30 (eg video decoder 30) and may additionally, i. e. possibly contain a communication interface or a communication unit 28, a post-processor 32 (post-processing unit 32) and a display device 34.

The communication interface 28 of the destination device 14 is configured to receive the encoded image data 21 (or any further processed version thereof), for example, directly from the source device 12 or from any other source, for example, a storage device, for example, an encoded image data storage device, and provide encoded picture data 21 to decoder 30.

Communication interface 22 and communication interface 28 may be configured to transmit or receive encoded image data 21 or encoded data 13 via a direct link between source device 12 and destination device 14, such as a direct wired or wireless connection, or over any network, such as wired or wireless network or any combination thereof, or any private and public network or any combination thereof.

The communication interface 22 may, for example, be configured to pack the encoded image data 21 into an appropriate format, such as packets, and/or process the encoded image data using any kind of transmission coding or processing for transmission over a communication channel or communication network.

Communication interface 28 analogous to communication interface 22 may, for example, be configured to receive transmitted data and process transmitted data using any kind of appropriate transmission decoding or processing and/or decompression to obtain encoded image data 21.

Both the communication interface 22 and the communication interface 28 can be configured as unidirectional communication interfaces, as indicated by the arrow for the encoded image data 13 in FIG. 1A pointing from source device 12 to destination device 14 or, as bi-directional communication interfaces, and may be configured to send and receive messages, for example, to establish a connection, acknowledge and exchange any other information related to the communication channel and /or data transmission, for example, transmission of encoded image data.

The decoder 30 is configured to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (more details will be described below, for example, based on FIG. 3 or FIG. 5).

The post processor 32 of the destination device 14 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. Post-processing performed by block 32 post-processing may include, for example, color format conversion (for example, from YCbCr to RGB), color correction, cropping or resampling, or any other processing, for example, to prepare the decoded image data 31 for display, for example, using the display device 34.

The display device 34 of the destination device 14 is configured to receive the image data 33 after being processed to display the image to, for example, a user or a viewer. The display device 34 may be or comprise any kind of display for presenting the reconstructed image, such as a built-in or external display or monitor. The displays may, for example, consist of liquid crystal displays (LCD), organic light emitting diode (OLED) displays, plasma displays, projectors, micro LED displays, liquid crystals on silicon (LCoS), digital light processor (DLP) or any other display.

Although FIG. 1A depicts source device 12 and destination device 14 as separate devices, device embodiments may also comprise both or both source device 12 functionality or corresponding functionality and destination device 14 or corresponding functionality. In such embodiments, source device 12 or related functionality and destination device 14 or related functionality may be implemented using the same hardware and/or software, or using separate hardware and/or software, or any combination thereof.

As will be apparent to those skilled in the art based on the description, the presence and (precise) separation of the functionality of the various blocks or functions in source device 12 and/or destination device 14, as shown in FIG. 1A may vary depending on the actual device and implementation.

Encoder 20 (eg, video encoder 20) and decoder 30 (eg, video decoder 30), or both encoder 20 or decoder 30, may be implemented by a processing circuit as shown in FIG. 1B, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. Encoder 20 may be implemented by processing circuitry 46 to implement various modules, as discussed with respect to encoder 20 in FIG. 2 and/or any other coding system or subsystem described herein. Decoder 30 may be implemented by processing circuitry 46 to implement various modules, as discussed with respect to decoder 30 in FIG. 3 and/or any other decoder system or subsystem described herein. The processing circuit may be configured to perform various operations as described below. As shown in FIG. 5, if the methods are partially implemented in software, the device may store instructions for the software on a suitable persistent computer-readable storage medium and may execute the instructions in hardware using one or more processors to execute the methods of this disclosure. Either of video encoder 20 and video decoder 30 may be integrated as part of a combined encoder/decoder (CODEC) in a single device, such as shown in FIG. 1b.

Source device 12 and destination device 14 may comprise any of a wide range of devices, including any kind of portable or stationary devices, such as laptops or laptops, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, game consoles, video streaming devices (eg, content service servers or content delivery servers), broadcast receiver, broadcast transmitter, and the like. and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

In some cases, the video coding system 10 illustrated in FIG. 1A is merely an example, and the methods of the present disclosure may apply to video encoding settings (eg, video encoding or video decoding) that do not necessarily involve any communication between encoding and decoding devices. In other examples, data is retrieved from local memory, transferred over a network, and the like. The video encoding device may encode and store data in memory, and/or the video decoding device may retrieve and decode data from memory. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data in memory and/or retrieve and decode data from memory.

For convenience of description, embodiments of the invention are described herein, for example, with reference to High Efficiency Video Coding (HEVC) or Universal Video Coding (VVC) reference software, the next generation video coding standard developed by the Collaborative Video Coding Group (JCT-VC). ) the ITU-T Video Coding Expert Group (VCEG) and the ISO/IEC Moving Picture Expert Group (MPEG). For a person skilled in the art it is obvious that the embodiments of the invention are not limited to HEVC or VVC.

Encoder and encoding method

Fig. 2 shows a block diagram of an exemplary video encoder 20 that is configured to implement the methods of the present disclosure. In the example in FIG. 2, video encoder 20 comprises an input 201 (or input interface 201), a residual calculation block 204, a transform processing block 206, a quantization block 208, an inverse quantization block 210, and an inverse transform processing block 212, a reconstruction block 214, a loop filter block 220, a buffer 230 decoded pictures (DPB), mode selection block 260, entropy encoding block 270, and output block 272 (or output interface 272). The mode selector 260 may include an inter-predictor 244, an intra-predictor 254, and a splitter 262. Inter prediction block 244 may include a motion estimator and a motion compensation block (not shown). Video encoder 20 shown in FIG. 2 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

The residual calculator 204, the transform processing block 206, the quantizer 208, the mode selector 260 form the direct signal path of the encoder 20, while, for example, the inverse quantizer 210, the inverse transform processor 212, the reconstruction block 214, the buffer 216, the loop filter 220, a decoded picture buffer (DPB) 230, inter predictor 244, and intra prediction 254 form an encoder signal return path, in which the encoder return signal path corresponds to the decoder signal path (see decoder 30 in FIG. 3). An inverse quantization block 210, an inverse transform processing block 212, a reconstruction block 214, a loop filter 220, a decoded picture buffer (DPB) 230, an inter prediction block 244, and an intra prediction block 254 are also related to the formation of the “embedded decoder” of the video encoder 20.

Images & image sharing (images and blocks)

Encoder 20 is configured to receive, for example, via input 201 an image 17 (or image data 17), for example an image of a sequence of images constituting a video or video sequence. The received image or image data may also be a pre-processed image 19 (or pre-processed image data 19). For ease of description, hereinafter referred to as picture 17. Picture 17 is also referred to as the current picture or the picture to be encoded (in particular, when encoding video to distinguish the current picture from other pictures, for example, previously encoded and/or decoded pictures of one and the same video sequence, that is, a video sequence that also contains the current image).

A (digital) image is, or can be viewed as, a two-dimensional array or matrix of samples with intensity values. A sample in an array can also be called a pixel (short for picture element) or a pixel. The number of samples in the horizontal and vertical direction (or axis) of the array or image determines the size and/or resolution of the image. Three color components are typically used to represent a color, that is, an image can be represented or contain three arrays of samples. In RBG format or color space, an image contains a corresponding array of red, green, and blue color samples. However, in video encoding, each pixel is usually represented in a luma and chrominance format or color space, such as YCbCr, which contains a luma component labeled Y (sometimes L is also used instead) and two chrominance components labeled Cb and Cr. The Y component of luminance (or briefly, luminance) represents the brightness or intensity of the gray level (eg, as in a grayscale image), while the two chrominance components (or briefly, chrominance) Cb and Cr represent components of chrominance or color information. Accordingly, an image in YCbCr format contains an array of luma samples of luminance sample values (Y) and two chrominance sample arrays of chrominance values (Cb and Cr). RGB images can be converted to YCbCr format and vice versa, a process also known as color conversion. If the image is monochrome, it can only contain an array of brightness samples. Accordingly, the image may be, for example, an array of luminance samples in monochrome format, or an array of luminance samples and two corresponding arrays of chrominance samples in 4:2:0, 4:2:2 and 4:4:4 color format.

Embodiments of encoder 20 may include a splitter (not shown in FIG. 2) configured to split image 17 into multiple (generally non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264/AVC), or coding tree blocks (CTBs) or coding tree blocks (CTUs) (H.265/HEVC and VVC). The partition block may be configured to use the same block size for all images of the video sequence and the corresponding grid defining the block size, or to change the block size between images, subsets or groups of images and divide each image into appropriate blocks.

In additional embodiments, the video encoder may be configured to directly receive block 203 of image 17, such as one, several or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be encoded.

Like image 17, image block 203 is again, or can be viewed as, a two-dimensional array or matrix of samples with intensity values (sample values), albeit smaller than image 17. In other words, block 203 may contain, for example, one array of samples (for example , a luma array in the case of a monochrome image 17 or a luminance or chrominance array in the case of a color image) or three arrays of samples (for example, a luminance array and two chrominance arrays in the case of a color image 17) or any other number and/or types of arrays, depending on the applied color format. The number of samples in the horizontal and vertical direction (or axis) of block 203 determines the size of block 203. Accordingly, the block may, for example, be an array of MxN samples (M-column per N-row) or an MxN array of transform coefficients.

Embodiments of video encoder 20, as shown in FIG. 2 may be further configured to encode image 17 block by block, such as splitting and encoding block 203.

Remainder Calculation

Residual block 204 is configured to calculate residual block 205 (also referred to as residual 205) based on image block 203 and prediction block 265 (more details about predictor 265 are provided later), for example, by subtracting the sample values of predictor 265 from the sample values. image block 203, sample by sample (pixel by pixel) to obtain a residual block 205 in the sample area.

transformation

Transform processing unit 206 is configured to apply transforms, such as discrete cosine transform (DCT) or discrete sine transform (DST), on sample values of residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as residual transform coefficients and represent the residual block 205 in the transform region.

The transform processing unit 206 may be configured to apply DCT/DST integer approximations such as the transforms specified for HEVC/H.265. Compared to the orthogonal DCT, such integer approximations are usually scaled by a certain factor. To preserve the norm of the residual block, which is processed by direct and inverse transformations, additional scaling factors are applied during the transformation. The scaling factors are usually chosen based on certain constraints, such as the scaling factors being a power of two for the shift operation, the bit width of the transform coefficients, the trade-off between precision and implementation cost, and so on. Specific scaling factors are, for example, specified for the inverse transform, for example, by the inverse transform processing unit 212 (and the corresponding inverse transform, for example, by the inverse transform processing unit 312 in the decoder 30), and the corresponding scaling factors for the forward transform, for example, by the transform processing unit 206 in encoder 20, respectively.

Embodiments of video encoder 20 (respectively, transform processing unit 206) may be configured to output transform parameters, such as the type of transform or transform, such as directly or encoded or compressed by entropy encoding unit 270, such that, for example, video decoder 30 may receive and use conversion parameters for decoding.

Quantization

Quantizer 208 is configured to quantize transform coefficients 207 to obtain quantized transform coefficients 209, for example by applying scalar quantization or vector quantization. The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209.

The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207 . For example, an n-bit transform factor may be rounded down to an m-bit transform factor during quantization, where n is greater than m. The degree of quantization can be changed by adjusting the quantization parameter (QP). For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. Smaller quantization step sizes correspond to finer quantization, while larger quantization step sizes correspond to coarser quantization. The applicable quantization step size may be indicated by a quantization parameter (QP). The quantization parameter may, for example, be an index to a predetermined set of applicable quantization step sizes. For example, small quantization parameters may correspond to fine quantization (small quantization step sizes), and large quantization parameters may correspond to coarse quantization (large quantization step sizes), or vice versa. Quantization may include division by the quantization step size and corresponding or inverse dequantization, for example by inverse quantizer 210, may include multiplication by the quantization step size. Embodiments according to some standards, such as HEVC, may be configured to use a quantization parameter to determine the quantization step size. Typically, the quantization step size can be calculated based on the quantization parameter using a fixed-point approximation of an equation involving division. Additional scaling factors may be introduced for quantization and dequantization to recover the residual block norm, which may be modified due to the scaling used in the fixed-point approximation of the equation for the quantization step size and quantization parameter. In one exemplary implementation, inverse transform scaling and dequantization may be combined. Alternatively, customized quantization tables can be used and signaled from the encoder to the decoder, for example, in a bitstream. Quantization is a lossy operation in which the loss increases with increasing quantization step sizes.

Embodiments of video encoder 20 (respectively, quantizer 208) may be configured to output quantization parameters (QPs), for example, directly or encoded by entropy coding unit 270, such that, for example, video decoder 30 may receive and apply quantization parameters for decoding.

Inverse quantization

Inverse quantizer 210 is configured to apply the inverse quantization of quantizer 208 to the quantized coefficients to obtain dequantized coefficients 211, for example, by applying the inverse quantization scheme applied by quantizer 208 based on or using the same quantization step size as block 208 quantization. The dequantized coefficients 211 may also be referred to as the dequantized residual coefficients 211 and correspond, although not usually identical to the transform coefficients due to quantization loss, to the transform coefficients 207.

Reverse transformation

The inverse transform processing unit 212 is configured to apply an inverse transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the region samples. The recovered residual block 213 may also be referred to as a transformation block 213 .

Recovery

Restoration block 214 (e.g., adder or adder 214) is configured to add transform block 213 (i.e., reconstructed residual block 213) to prediction block 265 to obtain reconstructed block 215 in the sample area, for example, by addition, sample by sample, the sample values of the reconstructed residual block 213 and the sample values of the prediction block 265 .

Filtration

The loop filter block 220 (or "loop filter" 220 for short) is configured to filter the reconstructed block 215 to obtain a filtered block 221 or, typically, to filter the reconstructed samples to obtain filtered samples. The edge filter block is designed, for example, to smooth out pixel transitions or otherwise improve video quality. The loop filter block 220 may comprise one or more loop filters, such as a deblocking filter, an adaptive shift (SAO) filter, or other filters, such as a two-sided filter or an adaptive loop filter (ALF), or sharpening or smoothing filters, or collaborative filters. or any combination of them. Although the loop filter unit 220 is shown in FIG. 2 as a loop filter, in other configurations, the loop filter unit 220 may be implemented as a post loop filter. The filtered block 221 may also be referred to as the filtered reconstructed block 221.

Embodiments of video encoder 20 (respectively, loop filter unit 220) may be configured to output loop filter parameters (such as adaptive sample offset information), for example, directly or encoded by entropy encoding unit 270, such that, for example, decoder 30 may receive and apply the same loop filter parameters or corresponding loop filters for decoding.

Decoded picture buffer

Decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use in encoding video data by video encoder 20. DPB 230 may be formed by any of a variety of memories, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The decoded picture buffer (DPB) 230 is configured to store the filtered block 221. The decoded picture buffer 230 can be further configured to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221 of the same current picture or different pictures, such as , previously reconstructed images, and may provide complete previously reconstructed, i.e., decoded, images (and corresponding reference blocks and samples) and/or partially reconstructed current image (and corresponding reference blocks and samples), for example, for inter prediction. The decoded picture buffer (DPB) 230 is also configured to store one or more unfiltered reconstructed blocks 215 or, in general, unfiltered reconstructed samples, for example, if the reconstructed block 215 is not filtered by the loop filter block 220 or any other further processed version of the reconstructed blocks or samples. .

Mode selection (separation and prediction)

The mode selection section 260 comprises a splitter 262, an inter prediction section 244, and an intra prediction section 254, and is configured to receive or obtain original image data, such as the original block 203 (the current block 203 of the current image 17) and reconstructed image data, such as filtered and/or unfiltered reconstructed samples or blocks of the same (current) picture and/or from one or more previously decoded pictures, eg decoded picture buffer 230 or other buffers (eg, linear buffer, not shown). The reconstructed image data is used as reference image data for prediction, such as inter prediction or intra prediction, to obtain prediction block 265 or predictor 265.

The mode selector 260 may be configured to determine or select a split for the current block prediction mode (including no split) and the prediction mode (e.g., intra or inter prediction mode) and generate a corresponding prediction block 265 that is used to calculate the residual block. 205 and to restore the restored block 215.

Embodiments of mode selector 260 may be configured to select a split and prediction mode (e.g., from those supported or available to mode selector 260) that provides the best match, or in other words, minimum residual (minimum residual means better compression). for transmission or storage), or minimal signaling overhead (minimum signaling overhead means better compression for transmission or storage), or which takes into account or balances both. The mode selector 260 may be configured to determine the division and prediction mode based on rate distortion optimization (RDO), that is, select the prediction mode that provides the minimum rate distortion. Terms such as "best", "minimum", "optimum", etc. in this context does not necessarily refer to the general "best", "minimum", "optimal", etc., but may also refer to termination performance or selection criteria such as a value exceeding or falling below a threshold, or other restrictions , potentially leading to a "non-optimal choice" but reducing complexity and processing time.

In other words, partitioner 262 may be configured to partition block 203 into smaller blocks or subblocks (which again form blocks), for example, iteratively using quadtree partitioning (QT), binary partitioning (BT), or ternary tree (TT), or any combination thereof, and to perform, for example, prediction for each of the partitions of blocks or subblocks, wherein the mode selection comprises selecting a tree structure of the partitioned block 203 and the prediction modes are applied to each of the partitions or subblocks.

The following explains in more detail the splitting process (for example, splitter 260) and prediction processing (inter predictor 244 and intra predictor 254) performed by exemplary video encoder 20.

Separation

Partitioner 262 may divide (or split) the current block 203 into smaller sections, such as smaller square or rectangular blocks. These smaller blocks (which may also be referred to as sub-blocks) can be further subdivided into even smaller sections. This is also called tree splitting or hierarchical tree splitting, in which a root block, e.g. at root level 0 of the tree (hierarchy level 0, depth 0) can be recursively split, e.g. split into two or more next lower tree level boxes, e.g. , nodes at level 1 of the tree (hierarchy level 1, depth 1) in which these blocks can be split again into two or more next lower level blocks, e.g. tree level 2 (hierarchy level 2, depth 2), etc. . until the split is complete, for example, when a termination criterion is met, such as the maximum tree depth or the minimum block size is reached. Blocks that are not split further are also called leaf blocks or leaf nodes of the tree. A tree using two partitions is called a binary tree (BT), a tree using three partitions is called a ternary tree (TT), and a tree using four partitions is called a quadtree (QT).

As mentioned earlier, the term "block" as used herein may be part, specifically, part of a square or rectangular shape of an image. With reference to, for example, HEVC and VVC, a block may either correspond to a coding tree block (CTU), a coding unit (CU), a prediction block (PU), and a transform unit (TU), and/or correspond to blocks, for example, a coding tree block (CTB), coding block (CB), transform block (TB), or prediction block (PB).

For example, a coding tree unit (CTU) may be either a luminance sample CTB, two corresponding chrominance sample CTBs of an image that has three sample arrays, or a monochrome image sample CTB or an image that is encoded using three separate color planes and the syntax structures used for coding samples. Accordingly, a coding tree block (CTB) may be a block of NxN samples for some value of N, such that splitting a component into CTBs is a split. A coding unit (CU) may be or comprise a luminance sample coding unit, two corresponding chrominance sample coding units of an image that has three sample arrays, or a monochrome image sample coding unit or an image that is encoded using three separate color planes and the syntax structures used for coding samples. Accordingly, a coding block (CB) may be a block of MxN samples for some values of M and N, so that the division of the CTB into coding blocks is a division.

In embodiments, for example in accordance with HEVC, a coding tree unit (CTU) may be partitioned into CUs using a square tree structure referred to as a coding tree. The decision as to whether to encode an image region using extrinsic (temporal) or intrinsic (spatial) prediction is made at the CU layer. Each CU may be further partitioned into one, two or four PUs according to the partition type of the PU. Within one PU, the same prediction process is applied, and the corresponding information is transmitted to the decoder based on the PU. After obtaining the residual block by applying a prediction process based on the division type of the PU, the CU may be divided into transform units (TUs) according to a different quadtree structure similar to the coding tree for the CU.

In embodiments, for example, in accordance with the latest video coding standard currently being developed, called universal video coding (VVC), square tree and binary tree (QTBT) splitting using coding block splitting. In the QTBT block structure, the CU can be either square or rectangular. For example, a coding tree unit (CTU) is first divided by a quadtree structure. The leaf nodes of a quadtree can be further subdivided using a binary tree or ternary (or ternary) tree structure. The leaf nodes of the quadtree are called coding units (CUs) and segmentation is used for prediction and transformation without any additional partitioning. This means that in most cases CU, PU and TU have the same block size in the QTBT coding block structure. In parallel, many partitions have also been proposed, for example, ternary tree partitioning has been used in conjunction with the QTBT block structure.

In one example, mode selector 260 of video encoder 20 may be configured to perform any combination of the separation methods described herein.

As described above, encoder 20 is configured to determine or select the best or optimal prediction mode from a set of (predefined) prediction modes. The set of prediction modes may include, for example, intra prediction modes and/or inter prediction modes.

Internal prediction

The intra prediction mode set may comprise 35 different intra prediction modes, such as non-directional modes such as DC (or average) mode and planar mode, or directional modes such as those defined in HEVC, or may comprise 67 different intra prediction modes such as , non-directional modes such as DC (or medium) mode and planar mode, or directional modes such as those defined for VVC.

The intra prediction block 254 is configured to use the reconstructed samples of adjacent blocks of the same current picture to generate the intra prediction block 265 according to the intra prediction mode of the intra prediction mode set.

Block 254 intra prediction (or, in General, block 260 mode selection) is further configured to output parameters of intra prediction (or, in General, information indicating the selected intra prediction mode for the block) in block 270 entropy encoding in the form of elements 266 syntax for including an encoded image in data 21 so that, for example, video decoder 30 can receive and use prediction parameters for decoding.

external prediction

The set (or possible) inter prediction modes depends on the available reference pictures (i.e., previous at least partially decoded pictures, e.g. stored in DBP 230) and other inter prediction parameters, e.g. , for example, an area search box around the area of the current block, a reference frame is used to find the best match of the reference block and/or, for example, whether pixel interpolation is applied, such as half/half pixel, quarter pixel and/or 1/16 pixel interpolation or not.

In addition to the aforementioned prediction modes, a skip mode and/or a direct mode may be applied.

Inter prediction block 244 may include a motion estimator (ME) and a motion compensation (MC) block (both not shown in FIG. 2). The motion estimator is configured to receive or obtain an image block 203 (the current image block 203 of the current image 201) and a decoded image 231 or at least one or a plurality of previously reconstructed blocks, e.g., reconstructed blocks of one or a plurality of other/different previously decoded 231 images for motion estimation. For example, the video sequence may comprise the current picture and previously decoded pictures 231 or, in other words, the current picture and previously decoded pictures 231 may be part of or form a sequence of pictures that make up the video sequence.

The encoder 20 may, for example, be configured to select a reference block from a plurality of reference blocks of the same or different pictures from a plurality of other pictures and provide a reference picture (or reference picture index) and/or a shift (spatial shift) between position (x, y coordinates ) of the reference block and the position of the current block as inter-prediction parameters to the motion estimation block. This shift is also called motion vector (MV).

The motion compensation block is configured to obtain, for example, receive an inter prediction parameter, and perform inter prediction based on or using the inter prediction parameter to obtain inter prediction block 265. The motion compensation performed by the motion compensator may comprise sampling or generating a prediction block based on the motion vector/block determined by the motion estimate, possibly performing interpolation to sub-pixel precision. Interpolation filtering can generate additional pixel samples from known pixel samples, thus potentially increasing the number of prediction block candidates that can be used to encode an image block. After receiving the motion vector for the PU of the current picture block, the motion compensator may locate the prediction block pointed to by the motion vector in one of the reference picture lists.

The motion compensation block may also generate syntax elements associated with blocks and a video segment for use by video decoder 30 when decoding image blocks of a video segment.

Entropy coding

The entropy encoding unit 270 is configured to apply an entropy encoding algorithm or schemes (e.g., variable length coding (VLC) schemes, context adaptive VLC scheme (CALVC), arithmetic coding scheme, context adaptive binary arithmetic coding (CABAC) based on context adaptive binary arithmetic coding (SBAC), probability interval entropy coding (PIPE), or other entropy coding technologies or methods) or bypass scheme (no compression) on quantized residual coefficients 209, inter prediction parameters, intra prediction parameters, and/or loop filter parameters and/or other syntax elements for obtaining encoded image data 21 that can be output by output 272, for example in the form of encoded bitstream 21, so that, for example, decoder 30 can receive and use parameters for decoding. The encoded bitstream 21 may be transmitted to video decoder 30 or stored in memory for later transmission or retrieval by video decoder 30.

Other structural variations of video encoder 20 may be used to encode the video stream. For example, non-transform-based encoder 20 may quantize the residual signal directly without transform processor 206 for specific blocks or frames. In another implementation, encoder 20 may have quantizer 208 and inverse quantizer 210 combined into a single block.

Decoder and decoding method

In FIG. 3 shows an example of a video decoder 30 that is configured to implement the methods of the present disclosure. Video decoder 30 is configured to receive encoded picture data 21 (e.g., encoded bitstream 21), such as encoded by encoder 20, to obtain a decoded picture 331. The encoded picture data or bitstream contains information for decoding encoded picture data, such as data that are image blocks of the encoded video segment and associated syntax elements.

In the example in FIG. 3, the decoder 30 includes an entropy decoding block 304, an inverse quantization block 310, an inverse transform processing block 312, a reconstruction block 314 (e.g., adder 314), a loop filter 320, a decoded picture (DBP) buffer 330, an inter prediction block 344, and a block 354 internal prediction. The inter prediction block may be or include a motion compensation block. Video decoder 30 may, in some examples, perform a decoding process that is generally the reverse of the encoding step described with respect to video encoder 100 in FIG. 2.

As explained with respect to the encoder 20, the inverse quantization block 210, the inverse transform processing block 212, the reconstruction block 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction block 344, and the intra prediction block 354 are also called "inline decoder" generation. of the video encoder 20. Accordingly, the inverse quantizer 310 may be identical in function to the inverse quantizer 110, the inverse transform processor 312 may be identical in function to the inverse transform processor 212, the restorer 314 may be identical in function to the restorer 214, the loop filter 320 may be identical in function to the loop filter 220 and the buffer 330 of the decoded image may be identical in function to the buffer 230 of the decoded image. Therefore, the explanations provided for the respective blocks and functions of the video encoder 20 apply to the respective blocks and functions of the video decoder 30, respectively.

Entropy decoding

The entropy decoding unit 304 is configured to parse the bitstream 21 (or, more generally, the encoded image data 21) and perform, for example, entropy decoding of the encoded image data 21 to obtain, for example, quantized coefficients 309 and/or decoded coding parameters ( not shown in Figure 3), such as any or all inter prediction parameters (eg, reference picture index and motion vector), intra prediction parameters (eg, intra prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and /or other syntax elements. Entropy decoding unit 304 may be configured to apply decoding algorithms or schemes corresponding to coding schemes as described with respect to entropy encoding unit 270 of encoder 20. Entropy decoding unit 304 may be further configured to provide inter prediction parameters, an intra prediction parameter, and/ or other syntax elements to mode select block 360 and other parameters to other blocks of decoder 30. Video decoder 30 may receive syntax elements at the video segment level and/or video block level.

Inverse quantization

The inverse quantizer 310 may be configured to receive quantization parameters (QPs) (or inverse quantization related information in general) and quantized coefficients from the encoded picture data 21 (e.g., by parsing and/or decoding, e.g., by entropy decoding block 304) and apply, based on the quantization parameters, inverse quantization to the decoded quantized coefficients 309 to obtain dequantized coefficients 311, which may also be referred to as transform coefficients 311. The inverse quantization process may include using a quantization parameter determined by video encoder 20 for each video block in a video segment to determine the degree of quantization, and likewise the degree of inverse quantization.

Reverse transformation

The inverse transform processor 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply transforms to the dequantized coefficients 311 to obtain reconstructed residual blocks 213 in the sample area. The restored residual blocks 213 may also be referred to as transform blocks 313 . The transformation may be an inverse transformation, eg, inverse DCT, inverse DST, inverse integer transformation, or a conceptually similar inverse transformation process. The inverse transform processing unit 312 may be further configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., by parsing and/or decoding, e.g., by entropy decoding block 304) to determine the transform to be applied to the dequantized coefficients 311.

Recovery

A reconstructor 314 (e.g., adder 314) is configured to add the reconstructed residual block 313 to the prediction block 365 to obtain a reconstructed block 315 in the sample area, for example, by adding the sample values of the reconstructed residual block 313 and the sample values of the prediction block 365.

Filtration

The loop filter block 320 (either in the encoding loop or after the encoding loop) is configured to filter the reconstructed block 315 to obtain a filtered block 321, for example, to smooth pixel transitions or otherwise improve video quality. The loop filter block 320 may comprise one or more loop filters, such as a deblocking filter, an adaptive shift filter (SAO), or one or more other filters, such as a bidirectional filter, an adaptive loop filter (ALF), an anti-aliasing filter, a sharpening filter, or collaborative filters or any combination thereof. Although in FIG. 3 shows the loop filter block 320 as a loop filter, in other configurations the loop filter block 320 may be implemented as a post loop filter.

Decoded image buffer

The decoded video blocks 321 in that frame or picture are then stored in the decoded picture buffer 330, which stores the decoded pictures 331 as reference pictures, used for subsequent motion compensation for other pictures and/or outputting the corresponding picture for display.

The decoder 30 is configured to output the decoded image 311, for example, through the output 312 for presentation or viewing to the user.

Prediction

Inter prediction block 344 may be identical to inter prediction block 244 (in particular, a motion compensation block) and intra prediction block 354 may be identical to inter function prediction block 254 and performs split or split decisions and predictions based on split and/or prediction parameters. or corresponding information received from the encoded image data 21 (eg, by parsing and/or decoding, eg, by entropy decoding block 304). The mode selector 360 may be configured to perform prediction (intra or inter prediction) for each block based on the reconstructed images, blocks, or corresponding samples (filtered or unfiltered) to obtain prediction block 365.

When a video segment is encoded as an intra-coded (I) segment, the intra prediction block 354 of the mode selector 360 is configured to generate a prediction block 365 for a picture block of the current video segment based on the signaled intra prediction mode and data from previously decoded blocks of the current picture. When a video image is encoded as an inter-coded (i.e., B or P) segment, the inter prediction block 344 (e.g., motion compensation block) of the mode select block 360 is configured to generate prediction blocks 365 for the video block of the current video segment based on motion vectors and others. syntax elements received from block 304 entropy decoding. For inter prediction, prediction blocks may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct reference frame lists, list 0 and list 1 using default construction methods based on reference pictures stored in DPB 330.

The mode selector 360 is configured to determine prediction information for the video block of the current video segment by analyzing motion vectors or other syntax elements, and uses the prediction information to generate prediction blocks for the current video block being decoded. For example, the mode selector 360 uses some of the received syntax elements to determine the prediction mode (eg, intra or inter prediction) used to encode the video blocks of a video segment, the inter prediction segment type (eg, B-segment, P-segment, or GPB- segment), construction information for one or more reference picture lists for the segment, motion vectors for each externally coded video block of the segment, inter prediction status for each externally coded video block of the segment, and other information for decoding video blocks in the current video segment.

Other variants of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter block 320 . For example, the non-transform-based decoder 30 can perform the inverse transform of the residual signal quantization directly without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, video decoder 30 may have an inverse quantizer 310 and an inverse transform processor 312 combined into one block.

It should be understood that in encoder 20 and decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after the interpolation filtering, motion vector output, or loop filtering, an additional operation such as clipping or shifting may be performed on the result of the interpolation filtering, motion vector output, or loop filtering processing.

It should be noted that additional operations can be applied to the received motion vectors of the current block (including, but not limited to, affine mode control point motion vectors, sub-block motion vectors of affine, planar, ATMVP modes, temporal motion vectors, etc.). . For example, the motion vector value is limited to a given range according to its representing bit. If the representing bit of the motion vector is bitDepth, then the range is -2^(bitDepth-1)~2^(bitDepth-1)-1, in which "^" means exponentiation. For example, if bitDepth is set to 16, then the range is -32768 ~ 32767; if bitDepth is set to 18, then the range is -131072 ~ 131071. For example, the value of the received motion vector (for example, MVs of four 4x4 subblocks within one 8x8 block) is limited such that the maximum difference between the integer parts of four 4x4 subblock MVs does not exceed N pixels, for example, no more than 1 pixel.

The present invention provides two methods for limiting a motion vector according to bitDepth.

Method 1: Remove MSB (Most Significant Bit) Overflow Through Expiration Operations

ux = (mvx + 2 ^bitDepth )% 2 ^bitDepth … (1)

mvx = (ux >= 2 ^{bitDepth -1} )? (ux - 2 ^bitDepth ): ux ... (2)

uy = (mvy + 2 ^bitDepth )% 2 ^bitDepth … (30)

mvy = (uy >= 2 ^{bitDepth -1} )? (uy - 2 ^bitDepth ): uy … (4)

in which mvx is the horizontal component of the motion vector of the image block or sub-block, mvy is the vertical component of the motion vector of the image block or sub-block, and ux and uy indicate an intermediate value;

For example, if the mvx value is -32769, after formula (1) and (2) is applied, the resulting value is 32767. In a computer system, decimal numbers are stored as a binary component. The binary component of -32769 is 1.0111.1111.1111.1111 (17 bits) and the MSB is discarded, so the resulting binary component is 0111.1111.1111.1111 (decimal 32767), which is the same as the output value after applying formulas (1) and (2).

ux = (mvpx + mvdx + 2 ^bitDepth )% 2 ^bitDepth ... (5)

mvx = (ux >= 2 ^{bitDepth -1} )? (ux - 2 ^bitDepth ): ux ... (6)

uy = (mvpy + mvdy + 2 ^bitDepth )% 2 ^bitDepth ... (7)

mvy = (uy >= 2 ^{bitDepth -1} )? (uy - 2 ^bitDepth ): uy …. (8)

Operations can be applied during summation of mvp and mvp as shown in formula (5) to (8).

Method 2: Remove MSB overflow via value constraint

vx = Clip3(-2 ^bitDepth-1 , 2 ^bitDepth-1 -1, vx)

vy = Clip3(-2 ^bitDepth-1 , 2 ^bitDepth-1 -1, vy)

in which vx is the horizontal component of the motion vector of the image block or sub-block, vy is the vertical component of the motion vector of the image block or sub-block; x, y and z correspond to the three input values of the MV clipping process and the Clip3 function definition is as follows:

Clip3(x, y, z) =

Fig. 4 is a diagram of a video encoding apparatus 400 according to an embodiment of the present disclosure. The video encoding device 400 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, video encoding device 400 may be a decoder, such as video decoder 30 of FIG. 1A or an encoder such as video encoder 20 of FIG. 1A.

The video encoding device 400 includes input ports 410 and receiver (Rx) units 420 for receiving data; a processor, logical unit, or central processing unit (CPU) 430 for processing data; transmitter (Tx) blocks 440 and data output ports 450; and a memory 460 for storing data. The video encoder 400 may also include optical-electrical (OE) components and electrical-optical (EO) components connected to input ports 410, receiver units 420, transmitter units 440, and output ports 450 for outputting or inputting optical or electrical signals.

Processor 430 is implemented in hardware and software. Processor 430 may be implemented as one or more CPU chips, cores (eg, as a multi-core processor), FPGAs, ASICs, and DSPs. Processor 430 is connected to input ports 410, receiver units 420, transmitter units 440, output ports 450, and memory 460. Processor 430 includes an encoder 470. Encoding module 470 implements the disclosed embodiments described above. For example, encoding module 470 implements, processes, prepares, or provides various encoding operations. Thus, the use of the encoding module 470 provides a significant improvement in the functionality of the video encoding device 400 and causes the video encoding device 400 to be converted to another state. Alternatively, encoding module 470 is implemented as instructions stored in memory 460 and executed by processor 430.

The memory 460 contains one or more disks, tape drives, and solid state drives and can be used as an overflow storage device, for storing programs when such programs are selected for execution, and for storing instructions and data that are read during program execution. Memory 460 may be volatile and/or non-volatile and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and/or static random access memory (SRAM).

Fig. 5 is a simplified block diagram of a device 500 that can be used as either or both source device 12 and destination device 14 of FIG. 1 according to an exemplary embodiment.

Processor 502 in device 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or a plurality of devices capable of manipulating or processing information that currently exists or will be developed in the future. Although the disclosed implementations may be implemented with a single processor, as shown with processor 502, for example, speed and efficiency advantages may be achieved using more than one processor.

In an implementation, memory 504 in device 500 may be read only memory (ROM) or random access memory (RAM). The memory 504 may be any other suitable type of storage device. Memory 504 may include code and data 506 accessed by processor 502 using bus 512. Memory 504 may further include an operating system 508 and application programs 510, application programs 510 include at least one program, which allows the processor 502 to perform the methods described herein. For example, application programs 510 may include applications 1 through N, which further include a video encoding application that performs the methods described herein.

The device 500 may also include one or more output devices such as a display 518. The display 518 may be, in one example, a touch display that combines a display with a touch element capable of recognizing touch inputs. Display 518 may be connected to processor 502 via bus 512.

Although depicted as a single bus, bus 512 of device 500 may be comprised of multiple buses. In addition, secondary storage 514 may be directly connected to other components of the device 500 or may be accessed via a network and may contain one integrated unit, such as a memory card, or more units, such as multiple memory cards. Thus, device 500 can be implemented in a wide variety of configurations.

Fig. 6 illustrates a design of a plurality of intra prediction modes used in the UIP HEVC design. For luma blocks, the intra prediction modes may contain up to 36 intra prediction modes, which may include three non-directional modes and 33 directional modes. The non-directional modes may comprise a mean prediction mode (DC) and a chrominance prediction mode from (LM) a luminance prediction mode. The planar prediction mode can perform prediction by assuming that the amplitude surface of the block with horizontal and vertical components is sloped from the block boundary. The DC prediction mode can perform prediction by assuming a flat block surface with a value corresponding to the average value of the block boundary. The LM prediction mode can perform prediction as long as the chrominance value for the block matches the luma value for the block.

In FIG. 7 shows an example of 67 intra prediction modes, for example, as proposed for VVC, a plurality of intra prediction modes of 67 intra prediction modes include: planar mode (index 0), dc mode (index 1), and angular modes with indices 2-66, in which the lower left corner mode in FIG. 7 refers to index 2 and the index numbering is incremented until index 66 is equal to the upper right corner mode in FIG. 7.

As shown in FIG. 8, since the second version of VVC, several modes are used corresponding to the oblique directions of intra prediction, including wide-angle (shown as dotted lines). For any of these modes, sample predictions in block interpolation of a set of adjacent reference samples should be performed if the corresponding position on the side of the block is fractional. HEVC and VVC use linear interpolation between two adjacent reference samples. JEM uses more complex 4-pin interpolation filters. The filter coefficients are selected as either Gaussian or Cubic depending on the width or height value. The decision whether to use the width or height is consistent with the decision to select the main reference side: when the intra prediction mode is greater than or equal to the diagonal mode, then the top side of the reference samples is selected as the main reference side, and the width value is selected to determine the use of the interpolation filter . Otherwise, the base seat is selected from the left side of the block and the height controls the filter selection process. In particular, if the selected side length is less than or equal to 8 samples, 4-tap cubic interpolation is applied. Otherwise, the interpolation filter is a 4-tap Gaussian.

Like inter prediction, intra prediction may require interpolation filtering when intra-block samples are predicted according to the fractional slope mode. If a linear filter is used for this purpose, the filter coefficients can be easily calculated if the predicted sample in the block is in a fractional (subpixel) position with respect to the reference samples. Thus, a linear filter does not require a LUT (lookup table) to store its coefficient. However, it can be used and directly computed. However, if the prediction module uses multi-tap (e.g. 4-tap or 8-tap) interpolation filters, then a LUT may be required to store the coefficient of the interpolation filters, as this is implemented in the inter-prediction module, where the 8-tap DCT-IF is stored in the table for luminance 4-tap DCT-IF for chrominance, as shown in Table 1 and Table 2, respectively.

Table 1. Specification of coefficients f _L [p] of interpolation luma prediction filter for each 1/16 position p of the fractional sample.

Positions p fractional sample Interpolation filter coefficients _fL [p][0] _fL [p][1] _fL [p][2] _fL [p][3] _fL [p][4] _fL [p][5] _fL [p][6] _fL [p][7] 1 0 1 −3 63 four −2 1 0 2 −1 2 −5 62 8 −3 1 0 3 −1 3 −8 60 13 −4 1 0 four −1 four −10 58 17 −5 1 0 five −1 four −11 52 26 −8 3 −1 6 −1 3 −9 47 31 −10 four −1 7 −1 four −11 45 34 −10 four −1 8 −1 four −11 40 40 −11 four −1 nine −1 four −10 34 45 −11 four −1 10 −1 four −10 31 47 −9 3 −1 eleven −1 3 −8 26 52 −11 four −1 12 0 1 −5 17 58 −10 four −1 13 0 1 −4 13 60 −8 3 −1 fourteen 0 1 −3 8 62 −5 2 −1 15 0 1 −2 four 63 −3 1 0

Table 2 Specification of interpolation chroma prediction interpolation filter coefficients f _C [p] for each 1/32 position p of the fractional sample.

Positions p fractional sample Interpolation filter coefficients f _C [ p ][ 0 ] f _C [ p ][ 1 ] f _C [ p ][ 2 ] f _C [ p ][ 3 ] 1 −1 63 2 0 2 −2 62 four 0 3 -2 60 7 −1 four −2 58 10 −2 five −3 57 12 −2 6 −4 56 fourteen −2 7 −4 55 15 −2 8 −4 54 sixteen −2 nine −5 53 eighteen −2 10 −6 52 20 −2 eleven −6 49 24 −3 12 −6 46 28 −4 13 −5 44 29 −4 fourteen −4 42 thirty −4 15 −4 39 33 −4 sixteen −4 36 36 −4 17 −4 33 39 −4 eighteen −4 thirty 42 −4 nineteen −4 29 44 −5 20 −4 28 46 −6 21 −3 24 49 −6 22 −2 20 52 −6 23 −2 eighteen 53 −5 24 −2 sixteen 54 −4 25 −2 15 55 −4 26 −2 fourteen 56 −4 27 −2 12 57 −3 28 −2 10 58 −2 29 −1 7 60 −2 thirty 0 four 62 −2 31 0 2 63 −1

A specific set of coefficients can be defined, as shown in Table 3.

Table 3 Specification of the intra prediction interpolation coefficients fC and fG as described in the VVC version 3 specification.

Positions p fractional sample Interpolation filter coefficients fC Interpolation filter coefficients fG f _C [p][0] f _C [p][1] f _C [p][2] f _C [p][3] fG[p][0] fG[p][1] fG[p][2] fG[p][3] 0 0 64 0 0 sixteen 32 sixteen 0 1 −1 63 2 0 15 29 17 3 2 −2 62 four 0 15 29 17 3 3 −2 60 7 −1 fourteen 29 eighteen 3 four −2 58 10 −2 13 29 eighteen four five −3 57 12 −2 13 28 nineteen four 6 −4 56 fourteen −2 13 28 nineteen four 7 −4 55 15 −2 12 28 20 four 8 −4 54 sixteen −2 eleven 28 20 five nine −5 53 eighteen −2 eleven 27 21 five 10 −6 52 20 −2 10 27 22 five eleven −6 49 24 −3 nine 27 22 6 12 −6 46 28 −4 nine 26 23 6 13 −5 44 29 −4 nine 26 23 6 fourteen −4 42 thirty −4 8 25 24 7 15 −4 39 33 −4 8 25 24 7 sixteen −4 36 36 −4 8 24 24 8 17 −4 33 39 −4 7 24 25 8 eighteen −4 thirty 42 −4 7 24 25 8 nineteen −4 29 44 −5 6 23 26 nine 20 −4 28 46 −6 6 23 26 nine 21 −3 24 49 −6 6 22 27 nine 22 −2 20 52 −6 five 22 27 10 23 −2 eighteen 53 −5 five 21 27 eleven 24 −2 sixteen 54 −4 five 20 28 eleven 25 −2 15 55 −4 four 20 28 12 26 −2 fourteen 56 −4 four nineteen 28 13 27 −2 12 57 −3 four nineteen 28 13 28 −2 10 58 −2 four eighteen 29 13 29 −1 7 60 −2 3 eighteen 29 fourteen thirty 0 four 62 −2 3 17 29 15 31 0 2 63 −1 3 17 29 15

An intra-predictive convolution sample is computed with coefficients determined according to the sub-pixel offset and filter type as follows:

указывает побитовую операцию сдвига вправо.In this equation

specifies a bitwise right shift operation.

извлечен из таблицы поиска (LUT). В таблице 3 приведен пример значений, хранящихся в соответствии с текущей структурой VVC, описанной в проекте спецификации версии 3 (DJVET-L1001 «Универсальное кодирование видео (проект 3)». Осуществляется выбор между сглаживающим (fG), повышения резкости (fG) и интерполяционными фильтрами с использованием условий MDIS (зависимый от режима внутреннего сглаживания). Таким образом, fC или fG могут использоваться для генерирования предсказанных выборок блока.Set of coefficients

retrieved from a lookup table (LUT). Table 3 shows an example of the values stored according to the current VVC structure described in the Draft Specification Version 3 (DJVET-L1001 Universal Video Coding (Draft 3)). filters using MDIS (internal smoothing mode dependent) conditions.Thus, fC or fG can be used to generate predicted block samples.

дополнительно обрезается до допустимого диапазона значений, который либо определяется с использованием SPS, или получен из битой глубины выбранного компонента.If a sharpening filter is selected, the predicted sample

optionally clipped to a valid range of values, which is either determined using SPS or derived from the bit depth of the selected component.

For some use cases, saving explicit LUTs for interpolation filters can be advantageously avoided because LUT processing requires additional memory and thus increases power consumption as well as chip size. The 1st aspect is critical for mobile applications. The 2nd aspect adversely affects the price.

For the mentioned use cases, it is reasonable to analytically determine the interpolation filter coefficients. Instead of extracting from the LUT, the filter coefficients can be calculated from the input position p of the fractional sample.

According to an embodiment of the present disclosure, a method for intra or inter prediction of video frame processing is provided, comprising the steps of: obtaining a reference sample; obtaining a sub-pixel offset value; and filtering, using the 4-tap sub-pixel interpolation filter, the reference sample to obtain a predicted sample value in which the filter coefficients of the 4-tap sub-pixel interpolation filter satisfy:

,

,

, и

являются коэффициентами фильтра 4-отводного интерполяционного фильтра субпикселя.Wherein p is the fractional part of the sub-pixel offset value and

,

,

, and

are the filter coefficients of the 4-tap sub-pixel interpolation filter.

In one exemplary implementation, reference samples may be denoted ref[ x ], with

ref[ x ] = p[ −1 − refIdx + x ][ −1 − refIdx ], with x = 0..nTbW + refIdx + 1.

The corresponding array of reference samples and "p" refers to x-y two-dimensional arrays p[ x ][ y ] containing sample values. The number of reference samples used may be at least one. In another example, the number of reference samples may be four.

In one exemplary implementation, the resulting sub-pixel offset value may be denoted

(y + 1 + refIdx ) * intraPredAngle

where "IntraPredAngle" is an intra prediction angle value.

In one exemplary implementation, the predicted sample value "predSamples[ x ][ y ]" can be obtained using

) + 32 ) >> 6 )predSamples[ x ][ y ] = Clip1( ( (

) + 32 ) >> 6 )

where fT[ i ] denotes the filter coefficients. These coefficients may be inter prediction luminance or chrominance filter coefficients related to fG and fC, respectively.

The choice of filter coefficients being chrominance or luma can be implemented by using and setting the "filterFlag" flag, for example, as

fT[ j ] = filterFlag ? fG[ iFact ][ j ] : fC[ iFact ][ j ]

With

iFact = ( ( y + 1 + refIdx ) * intraPredAngle ) & 31.

The value "31" refers to the fractional part of the sub-pixel offset value and can take other values than "31". Filter coefficient values fG (luminance) and/or fC (chrominance) are obtained analytically at runtime using the above analytical expression for the 4-tap filter coefficients. Thus, the filter coefficients are determined according to the sub-pixel offset value.

Therefore, the filter coefficients are obtained without access to the corresponding filter coefficient values from the LUT, but rather are obtained by calculation.

Alternatively, the filter coefficients can be calculated using the equations above and the values can be stored in the LUT.

According to an embodiment of the present disclosure, the filter coefficients of the 4-tap sub-pixel interpolation filter are defined in the table as follows:

p

0 sixteen 32 sixteen 0 1 sixteen 32 sixteen 0 2 15 31 17 1 3 15 31 17 1 four fourteen thirty eighteen 2 five fourteen thirty eighteen 2 6 13 29 nineteen 3 7 13 29 nineteen 3 8 12 28 20 four nine 12 28 20 four 10 eleven 27 21 five eleven eleven 27 21 five 12 10 26 22 6 13 10 26 22 6 fourteen nine 25 23 7 15 nine 25 23 7 sixteen 8 24 24 8 17 8 24 24 8 eighteen 7 23 25 nine nineteen 7 23 25 nine 20 6 22 26 10 21 6 22 26 10 22 five 21 27 eleven 23 five 21 27 eleven 24 four 20 28 12 25 four 20 28 12 26 3 nineteen 29 13 27 3 nineteen 29 13 28 2 eighteen thirty fourteen 29 2 eighteen thirty fourteen thirty 1 17 31 15 31 1 17 31 15

According to an aspect of the present disclosure, a method for intra or inter prediction of video frame processing is provided, comprising the steps of: obtaining a reference sample; obtaining a sub-pixel offset value; and filtering using a 4-tap sub-pixel interpolation filter, a reference sample to obtain a predicted sample value, in which the filter coefficients of the 4-tap sub-pixel interpolation filter are determined as follows:

p

0 sixteen 32 sixteen 0 1 sixteen 32 sixteen 0 2 15 31 17 1 3 15 31 17 1 four fourteen thirty eighteen 2 five fourteen thirty eighteen 2 6 13 29 nineteen 3 7 13 29 nineteen 3 8 12 28 20 four nine 12 28 20 four 10 eleven 27 21 five eleven eleven 27 21 five 12 10 26 22 6 13 10 26 22 6 fourteen nine 25 23 7 15 nine 25 23 7 sixteen 8 24 24 8 17 8 24 24 8 eighteen 7 23 25 nine nineteen 7 23 25 nine 20 6 22 26 10 21 6 22 26 10 22 five 21 27 eleven 23 five 21 27 eleven 24 four 20 28 12 25 four 20 28 12 26 3 nineteen 29 13 27 3 nineteen 29 13 28 2 eighteen thirty fourteen 29 2 eighteen thirty fourteen thirty 1 17 31 15 31 1 17 31 15

,

,

и

являются коэффициентами фильтра 4-отводного интерполяционного фильтра субпикселя.where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients of the 4-tap sub-pixel interpolation filter.

In one exemplary implementation, filter coefficient values may be stored in a LUT. This means that for sub-pixel interpolation filtering, the corresponding values of the filter coefficients must be extracted from the LUT.

с i = 0, … 3. Это называется 4-отводным фильтром и/или 4-точечным фильтром.Fig. 23 shows a flowchart of an intra/inter prediction process. At step 2301, a reference sample is obtained. The reference sample may include one or more reference samples. At step 2302, a sub-pixel offset value is obtained. In step 2303, the reference sample is then filtered using a sub-pixel interpolation filter having filter coefficients

with i = 0, … 3. This is called a 4-tap filter and/or a 4-point filter.

The filter coefficients can be obtained from analytical expressions with a given fractional part p of the sub-pixel offset value. The filter coefficients can also be obtained from the LUT with reference to p values.

According to an embodiment of the present disclosure, the p bit is set to zero.

According to an embodiment of the present disclosure, p is incremented by 1 and subsequently the least significant bit of the incremented p is set to zero before said p is used to derive the filter coefficients.

According to an embodiment of the present disclosure, obtaining the filter coefficients comprises the step of calculating the filter coefficients according to

фильтра от предшествующего коэффициента

фильтра. Это может дополнительно уменьшить время для расчета коэффициентов фильтра.This may provide the advantage of calculating the coefficient

filter from the previous coefficient

filter. This can further reduce the time to calculate the filter coefficients.

The present invention allows using both approaches to obtain the filter coefficients: obtaining from the LUT and direct calculation in accordance with the proposed equations.

One of the signs of the interpolation filter is the linear dependence of its phase shift at the position p of the fractional sample. Embodiments of the invention use this relationship to obtain the value of the filter coefficient using simple operations on position p of the fractional sample.

The linear relationship is used to obtain the smoothing coefficients fG of the interpolation filter. Deriving the sharpening coefficients fC of the interpolation filter is based on deriving the absolute value of a linear expression based on p.

According to an embodiment of the present disclosure, a video coding method is provided, the method comprising intra prediction processing of a video frame block, comprising: filtering using a 4-tap interpolation filter of a sub-pixel of a reference sample of a video frame; wherein the 4-tap sub-pixel interpolation filter is a combination of filters, with filter coefficients of at least one of the filters, satisfying:

,

,

и

являются коэффициентами фильтра. where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients.

Fig. 24 shows a flowchart of a video encoding process. In step 2410, intra prediction processing is performed by filtering the block reference sample using a 4-tap sub-pixel interpolation filter.

According to an embodiment of the present disclosure, the 4-tap sub-pixel interpolation filter is an anti-aliasing filter that is a parallel combination of two phase dependent linear filters and a low pass filter.

According to an embodiment of the present disclosure, the 4-tap sub-pixel interpolation filter is a sharpening filter that is a parallel combination of one phase-dependent linear filter and a power-varying high-pass filter.

According to an aspect of the present invention, the power-changing high-pass filter is a phase-independent finite impulse response high-pass filter FIR whose output is multiplied by a phase-dependent power parameter.

According to any of the preceding aspects of the present disclosure, the least significant bit of p is set to zero.

According to an aspect of the present disclosure, p is incremented by 1 and then the least significant bit of the incremented value is set to zero prior to using that value to derive the interpolation filter coefficients.

According to an aspect of the present disclosure, a clipping operation is performed for the output of a power varying high pass filter.

Embodiments of the invention express the filter coefficient as a combination of two or more FIR (finite impulse response) filters. This combination may contain the following options:

- sequential combination, that is, convolution of coefficients;

- a parallel combination, that is, a weighted sum of coefficients.

Each of these filters may or may not depend on the phase parameter P, which is a function of the fractional sample position p. The coefficients of the filters belonging to the combination have a simple representation from the phase parameter P.

In one embodiment, in FIG. 9, a set of reference samples is used in the interpolation filter.

In one embodiment, for the case of a smoothing interpolation filter, the value of the phase parameter P is the same as the value of the position p of the fractional sample. In FIG. 9, blocks F1 and F2 represent linear interpolation filters. The coefficients for each of these filters are expressed from the phase parameter P as follows:

In the example here and below, the split operation can be defined with or without rounding, i.e.:

or

In which "a" is the denominator of the division, and "n" is the power of the two parameters of the denominator.

Block F3 is a two-tap low-pass filter with constant coefficients:

,

,

.

.

In the example, filters F1..F3 are not normalized, F3 has a higher DC gain than F1 and F3 has a higher DC gain than F2.

The output of filters F1 .. F3 is summed and normalized. The normalization (">>" block) can be performed, for example, by shifting to the right or division with rounding, as described above.

The resulting equation for this combination (equivalent to a 4-tap filter) is expressed as:

This embodiment provides a LUT-based implementation. According to this implementation, coefficient values can also be determined using the LUT. The values stored in this LUT are determined using Table 4 (for both cases: with and without rounding).

Table 4 - Specification of the smoothing filter intra prediction interpolation filter coefficient.

p with rounding No rounding

0 sixteen 32 sixteen 0 sixteen 32 sixteen 0 1 15 31 17 1 sixteen 32 sixteen 0 2 15 31 17 1 15 31 17 1 3 fourteen thirty eighteen 2 15 31 17 1 four fourteen thirty eighteen 2 fourteen thirty eighteen 2 five 13 29 nineteen 3 fourteen thirty eighteen 2 6 13 29 nineteen 3 13 29 nineteen 3 7 12 28 20 four 13 29 nineteen 3 8 12 28 20 four 12 28 20 four nine eleven 27 21 five 12 28 20 four 10 eleven 27 21 five eleven 27 21 five eleven 10 26 22 6 eleven 27 21 five 12 10 26 22 6 10 26 22 6 13 nine 25 23 7 10 26 22 6 fourteen nine 25 23 7 nine 25 23 7 15 8 24 24 8 nine 25 23 7 sixteen 8 24 24 8 8 24 24 8 17 7 23 25 nine 8 24 24 8 eighteen 7 23 25 nine 7 23 25 nine nineteen 6 22 26 10 7 23 25 nine 20 6 22 26 10 6 22 26 10 21 five 21 27 eleven 6 22 26 10 22 five 21 27 eleven five 21 27 eleven 23 four 20 28 12 five 21 27 eleven 24 four 20 28 12 four 20 28 12 25 3 nineteen 29 13 four 20 28 12 26 3 nineteen 29 13 3 nineteen 29 13 27 2 eighteen thirty fourteen 3 nineteen 29 13 28 2 eighteen thirty fourteen 2 eighteen thirty fourteen 29 1 17 31 15 2 eighteen thirty fourteen thirty 1 17 31 15 1 17 31 15 31 0 sixteen 32 sixteen 1 17 31 15

From Table 4, you can see that the coefficients are within the range [0, 31]. This fact explains the technical benefit of the invention, which is the use of 16-bit multipliers for 8-bit and 10-bit images, which are and will be used, at least in the medium term. A typical implementation of the invention will contain four parallel multiplication operations, in which the multiplication operands have bit depths, a maximum of 6 for filter coefficients and 10 for samples. The result of the multiplication will not exceed a 16-bit value, which makes the proposed coefficients implementation-friendly.

In the inter prediction mode, motion compensation of the motion block is performed. The motion compensation may comprise an interpolation filtering step, which is similar to intra prediction filtering. Another useful feature of the invention is that the coefficients of the interpolation filters have the same precision (ie, the bit depth of the coefficient value is the same) for the intra and inter prediction modes. If the accuracy of the interpolation filter coefficients is higher in the case of intra prediction, which can adversely affect the computational complexity of the video codec. The reason is that intra-prediction is by nature sequential, since it requires reconstructed samples of previous blocks. In contrast, extrinsic prediction can be performed in parallel. Therefore, if the interpolation accuracy is higher for intra prediction than for inter prediction, which may deepen the implementation mismatch between the intra and inter prediction pipelines. This leveling of accuracy avoids such negative consequences.

Another advantage of the present invention is the reduction in coefficient diversity. Given that pairs of adjacent rows in Table 4 are identical, a practical implementation of the invention would use a LUT that has a total of 16 rows (rows corresponding to even p values in the "no rounding" column and rows corresponding to odd p values in the "rounded" column ) instead of 32 lines.

The technical effect of the invention is the possibility of implementing at least the following:

- implementation without LUT, which calculate the filter coefficients using p-values, in which the calculation is based on the analytical equations described above; and

is an LUT-based implementation that derives the filter coefficients from the LUT using an index value based on the p-value.

The precision of the p-value decreases as the coefficient values are calculated. In particular, the value of the coefficients are obtained based on the result of an integer division by 2, i.e., p/2, which can be implemented as a right-shift by 1, which is simply implemented in the hardware. For a LUT-based implementation, the precision is reduced, resulting in a reduction in the memory required to store the LUT. In hardware, this will also reduce the number of LUT address bus wires.

According to an embodiment of the present disclosure, an apparatus for intra or inter prediction of video frame processing is provided, including: a reference sample acquisition unit configured to acquire a reference sample; a sub-pixel offset value obtaining unit, configured to obtain the sub-pixel offset value; and a 4-tap sub-pixel interpolation filter configured to filter the reference sample to obtain a predicted sample value in which the filter coefficients of the 4-tap sub-pixel interpolation filter satisfy:

,

,

и

являются коэффициентами фильтра 4-отводного интерполяционного фильтра субпикселя. where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients of the 4-tap sub-pixel interpolation filter.

According to an embodiment of the present disclosure, the filter coefficients of the 4-tap sub-pixel interpolation filter are defined in the table as follows:

p

0 sixteen 32 sixteen 0 1 sixteen 32 sixteen 0 2 15 31 17 1 3 15 31 17 1 four fourteen thirty eighteen 2 five fourteen thirty eighteen 2 6 13 29 nineteen 3 7 13 29 nineteen 3 8 12 28 20 four nine 12 28 20 four 10 eleven 27 21 five eleven eleven 27 21 five 12 10 26 22 6 13 10 26 22 6 fourteen nine 25 23 7 15 nine 25 23 7 sixteen 8 24 24 8 17 8 24 24 8 eighteen 7 23 25 nine nineteen 7 23 25 nine 20 6 22 26 10 21 6 22 26 10 22 five 21 27 eleven 23 five 21 27 eleven 24 four 20 28 12 25 four 20 28 12 26 3 nineteen 29 13 27 3 nineteen 29 13 28 2 eighteen thirty fourteen 29 2 eighteen thirty fourteen thirty 1 17 31 15 31 1 17 31 15

According to an aspect of the present disclosure, a device for intra or inter prediction of a video frame is provided, comprising: a reference sample acquisition unit configured to obtain a reference sample; a sub-pixel offset value obtaining unit, configured to obtain the sub-pixel offset value; and a 4-tap sub-pixel interpolation filter configured to filter the reference sample to obtain a predicted sample value, wherein the filter coefficients of the 4-tap sub-pixel interpolation filter are determined as follows:

p

0 sixteen 32 sixteen 0 1 sixteen 32 sixteen 0 2 15 31 17 1 3 15 31 17 1 four fourteen thirty eighteen 2 five fourteen thirty eighteen 2 6 13 29 nineteen 3 7 13 29 nineteen 3 8 12 28 20 four nine 12 28 20 four 10 eleven 27 21 five eleven eleven 27 21 five 12 10 26 22 6 13 10 26 22 6 fourteen nine 25 23 7 15 nine 25 23 7 sixteen 8 24 24 8 17 8 24 24 8 eighteen 7 23 25 nine nineteen 7 23 25 nine 20 6 22 26 10 21 6 22 26 10 22 five 21 27 eleven 23 five 21 27 eleven 24 four 20 28 12 25 four 20 28 12 26 3 nineteen 29 13 27 3 nineteen 29 13 28 2 eighteen thirty fourteen 29 2 eighteen thirty fourteen thirty 1 17 31 15 31 1 17 31 15

,

,

и

являются коэффициентами фильтра. where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients.

Fig. 25 shows a diagram of an intra/inter prediction processing unit 2500, which includes a reference sample acquisition unit 2510, a subpixel offset value acquisition unit 2520, and a subpixel interpolation filter 2530.

According to an aspect of the present disclosure, a device for encoding a video is provided, comprising: an intra prediction processing unit configured to perform intra prediction of a block of a video frame; and a 4-tap sub-pixel interpolation filter configured to filter a video frame reference sample; wherein the 4-tap sub-pixel interpolation filter is a combination of filters, with filter coefficients of at least one of the filters, satisfying:

,

,

и

являются коэффициентами фильтра.where p is the fractional part of the subpixel offset value and

,

,

and

are the filter coefficients.

Fig. 26 shows a diagram of a video encoding unit 2600, which includes an intra prediction processing unit 2610 and a sub-pixel interpolation filter 2620.

According to an aspect of the present disclosure, an encoder (20) is provided, containing a processing circuit for performing the method according to any of the preceding aspects of the present disclosure.

According to an aspect of the present disclosure, a decoder (30) is provided, comprising a processing circuit for performing a method according to any of the preceding aspects of the present disclosure.

According to an aspect of the present disclosure, a computer program product is provided that contains program code for performing a method in accordance with any of the preceding aspects of the present disclosure.

According to an aspect of the present disclosure, a decoder is provided, comprising: one or more processors; and a persistent computer-readable medium coupled to the processors and storing programs for execution by the processors, wherein the programs, when executed by the processors, configure a decoder to execute a method in accordance with any of the preceding aspects of the present disclosure.

According to an aspect of the present disclosure, an encoder is provided, comprising: one or more processors; and a persistent computer-readable medium coupled to the processors and storing programs for execution by the processors, wherein the programs, when executed by the processors, configure an encoder to execute a method in accordance with any of the preceding aspects of the present disclosure.

The present invention according to any of the preceding aspects can provide a more efficient method for intra/inter prediction of a video frame. This is due to the fact that the filter coefficients of the interpolation filter are obtained in an analytical way, that is, the said coefficients are calculated during processing. This avoids storing the coefficients in a lookup table (LUT), requiring access time to read the coefficients from memory.

фильтра являются линейными на позиции р дробной выборки, и используются операции деления на 2, соответствующая операция может быть эффективно выполнена, используя быструю низкоуровневую битовую операцию. Соответствующее время для выполнения битовой операции и для расчета коэффициентов фильтра короче времени доступа к хранимым коэффициентам из LUT. Таким образом, задержка обработки сокращается.Thus, the prediction becomes more efficient and requires less memory. It also provides an inexpensive implementation of the prediction. Moreover, since the analytic coefficients

filters are linear at position p of a fractional sample, and divide-by-2 operations are used, the corresponding operation can be efficiently performed using a fast low-level bit operation. The corresponding time to perform a bit operation and to calculate the filter coefficients is shorter than the time to access the stored coefficients from the LUT. Thus, the processing delay is reduced.

Moreover, the specific analytical structure of the filter coefficients may provide the advantage of implementing a low complexity filter block(s). As previously described, the filter's frequency response (ie, frequency response) for different subpixel positions follows the amplitude-to-phase relationship and prevents artifacts in the frequency response, particularly at high frequencies. The linearity of the filter coefficients can provide the benefit of equipment reuse.

In one embodiment, for the case of an interpolation sharpening filter, the power parameter S is defined as follows:

This power parameter S is determined to obtain the maximum value at the half-pixel position (p = 16). For subpixel positions close to integer, the power parameter S has a lower value. An exemplary filter combination is shown in FIG. 10. Block F1 in this figure is a linear filter with its coefficients defined as follows:

,

,

.

.

Block F2 is a high pass filter having the following coefficients:

The output of block F2 is multiplied by the power parameter S. The result of the multiplication is additionally summed with the output of the linear filter F1, and the resulting sum is normalized.

The resulting equation for this combination (equivalent to a 4-tap filter) is expressed as:

For this equation, a LUT-based implementation is also possible. Table 5 illustrates the values of the coefficients.

Table 5 Specification of the interpolation filter of the sharpening filter coefficient intra prediction

p

0 0 64 0 0 1 0 62 2 0 2 0 60 four 0 3 -1 59 7 -1 four -1 57 nine -1 five -1 55 eleven -1 6 -2 54 fourteen -2 7 -2 52 sixteen -2 8 -3 51 nineteen -3 nine -3 49 21 -3 10 -3 47 23 -3 eleven -four 46 26 -four 12 -four 44 28 -four 13 -four 42 thirty -four fourteen -five 41 33 -five 15 -five 39 35 -five sixteen -6 38 38 -6 17 -five 35 39 -five eighteen -five 33 41 -five nineteen -four thirty 42 -four 20 -four 28 44 -four 21 -four 26 46 -four 22 -3 23 47 -3 23 -3 21 49 -3 24 -3 nineteen 51 -3 25 -2 sixteen 52 -2 26 -2 fourteen 54 -2 27 -1 eleven 55 -1 28 -1 nine 57 -1 29 -1 7 59 -1 thirty 0 four 60 0 31 0 2 62 0

Because filter F2 has negative coefficients, the normalized result is further clipped to fit the range of valid values that can either be signaled in the SPS or obtained from the bit depth of the samples.

An alternative implementation is to indicate an alternative position of the clipping operation (see Fig. 11). This alternative is based on the fact that the F1 linear filter may not have an output value that is lower or higher than any of its input values.

In one embodiment, the pruning unit operates using the following steps.

Step one is to determine whether to clip on the minimum threshold or the maximum threshold.

Stage two depends on the outcome of stage one. In particular, clipping is performed on either the minimum or maximum threshold value.

этапа два (и блока отсечения) может быть определен следующим образом:The minimum threshold applies when the clipping block input is negative. Otherwise, the maximum threshold is applied. Output

stage two (and the clipping block) can be defined as follows:

рассчитывается вывод минимального порогового значения

следующим образом:For input value

the output of the minimum threshold value is calculated

in the following way:

The maximum threshold output is calculated as follows:

является максимальным значением диапазона допустимого значения выборки.wherein

is the maximum value of the range of valid sample value.

и

являются входными опорными выборками, показанными на фиг. 11, которые являются входом линейного фильтра F1.Values

and

are the input reference samples shown in FIG. 11, which are the input of the line filter F1.

рассчитывается следующим образом:In one embodiment, in the case of performing a cut after a multiplication operation,

calculated as follows:

In some embodiments, filter F3 in FIG. 9, filter F2 in FIG. 10 and FIG. 11 may not depend on the parameter P of the phase. These embodiments simplify implementation because the output of these filters may be the same for each input row of predicted samples, thus this step can be performed before starting the interpolation process, for example, in the reference sample filtering step. This approach increases concurrency and thus reduces directional filtering latency.

Another alternative implementation (see FIG. 12) uses a variable power filter and not just a filter output multiplier. Fig. 12A shows the case where the output of the high pass filter F2 is clipped. Fig. 12B shows the case where output clipping of the normalized output of the combination of F1 and F2 is performed. An example of adjustable power filters could be: a two-way filter, an inverse transform filter, etc.

Another embodiment uses resolution reduction p, that is, only 16 out of 32 entries. This is achieved, for example, by setting the least significant bit of p to zero.

As shown in FIG. 13-21, various combinations using LUTs and analytic representations of the interpolation filter coefficients are possible for both intra and inter prediction. For the examples of FIG. 14 illustrates a use case where coefficients for smoothing an interpolation filter used in the case of intra prediction are calculated, while coefficients for other interpolation filters for both intra prediction and inter prediction are stored in LUTs. Additionally, the video coding specification may include both LUTs and analytic representations of the interpolation filter coefficients for various implementations. If both LUT-based representations and analytic representations are described, they must provide identical coefficients.

Fig. 22 is an implementation of the coefficient calculation method as shown in FIG. 1.

In the equation

The power parameter S is defined as follows:

This implementation corresponds to another way of obtaining the power parameter S (denoted as the resulting value "q" in Fig. 22). The value of the fractional sample position p p is sent to the Demux demultiplexer, which is controlled by the input 1-bit signal SEL, which is set to the value of the 5th least significant bit of p (which also has index 4 if indexing starts from zero). If this bit is set to "1", then the SEL signal indicates that the value of p is greater than 16, and the following value should be calculated:

= − ( not (31 xor

) ),

= − ( not (31xor

) ),

Where "not" and "xor" are the bitwise NOT and XOR operations, respectively. This expression refers to q = (32 - p) and can be written in C/C++ programming languages as

= -(~(0x1F ^

)).

= -(~(0x1F ^

)).

Otherwise (that is, if the input signal SEL of the Demux is "0"), any calculations are bypassed and signal p is sent to the MUX as signal q. The output signal r of the MUX multiplexer is passed to the module, which calculates the value t as follows:

t = ((r << 1) + r) >> 3,

Where "<<" and ">>" are offsets to the right and left, respectively.

In fact, this value of t is the power parameter S. Further optimization is possible by performing the steps shown in FIG. 22, in parallel with one of the linear filter coefficients:

z = (64 - (p << 1))

.Once the z and S values have been calculated, they can be added to each other to get

.

Although embodiments of the invention have been primarily described in terms of video coding, it should be noted that the embodiments of encoding system 10, encoder 20, and decoder 30 (and system 10, respectively) and other embodiments described herein can also be performed. capable of processing or encoding a still image, that is, processing or encoding a single image independently of any preceding or subsequent image, as in video encoding. In general, if the image encoding processing is limited to one image 17, only inter prediction blocks 244 (encoder) and 344 (decoder) may be unavailable. All other functions (also referred to as tools or technologies) of video encoder 20 and video decoder 30 can equally be used for still picture processing, e.g. /312, split 262/362, intra prediction 254/354, and/or loop filtering 220, 320 and entropy coding 270 and entropy decoding 304.

Embodiments of, for example, encoder 20 and decoder 30, as well as the functions described herein, for example, in relation to encoder 20 and decoder 30, may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted over a communication medium as one or more instructions or code and executed by a hardware processor. The computer-readable medium may include a computer-readable medium that corresponds to a tangible medium, such as a data medium, or a communication medium, including any medium that facilitates the transfer of a computer program from one place to another, for example, in accordance with a communication protocol. Thus, computer-readable media can typically correspond to (1) tangible computer-readable storage media that are non-volatile, or (2) communication media such as a signal or carrier wave. Storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementing the methods described in the present invention. The computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk or other magnetic storage devices, flash memory, or any other media which can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. Additionally, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are considered as the communication medium. However, it should be understood that computer-readable storage media and storage media do not include connections, carrier waves, signals, or other temporary media, but instead refer to non-volatile, tangible storage media. A disc as used herein includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk, and a Blu-ray disc, where the discs usually reproduce data magnetically and the discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer readable media.

The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Accordingly, the term "processor" as used herein may refer to any of the aforementioned structure, or any other structure suitable for implementing the described methods. Additionally, in some aspects described herein, functionality may be provided within dedicated hardware and/or software modules capable of encoding and decoding or contained in a combined codec. Additionally, the methods may be fully implemented in one or more circuits or logic elements.

The methods of the present disclosure may be implemented in a wide variety of devices or appliances, including a wireless telephone, an integrated circuit (IC), or a set of ICs (eg, a chipset). The present invention describes various components, modules, or blocks to represent the functional aspects of devices capable of performing the disclosed technologies, but not necessarily requiring implementation in different hardware blocks. Rather, as described above, the various blocks may be combined into a codec hardware block or provided by a set of cooperating hardware blocks including one or more processors, as described above, in combination with suitable software and/or firmware.

,

,

и

, = p / 2, р является дробной частью значения смещения субпикселя.In summary, the present invention relates to intra or inter prediction for video encoding and decoding. For this purpose, the apparatus and methods obtain a reference sample and a sub-pixel offset value. A 4-tap sub-pixel interpolation filter is used to obtain the predicted value for filtering the reference sample. The filter coefficients of the 4-tap sub-pixel interpolation filter are determined according to the sub-pixel offset value, such as

,

,

and

, = p / 2, p is the fractional part of the sub-pixel offset value.

Additional embodiments are summarized in the following paragraphs:

Item 1: The video coding technology intra or inter prediction method, the method comprises:

obtaining a reference sample (ie, at least one reference sample in the example, the number of reference samples may be four);

obtaining a sub-pixel offset value; and

filtering, using the sub-pixel interpolation filter, the reference sample to obtain a predicted sample value, in which the filter coefficients of the sub-pixel interpolation filter are obtained in accordance with the sub-pixel offset value.

Claim 2: The method of claim 1, wherein the filter coefficient of the sub-pixel interpolation filter is obtained depending on the fractional part of the sub-pixel offset value.

Claim 3: The method of claim 1 or claim 2, wherein the sub-pixel interpolation filter is an equivalent 4-tap filter.

Claim 4: The method of claim 3, wherein the filter coefficients of the sub-pixel interpolation filter are obtained according to

in which p is the fractional part of the sub-pixel offset value.

Claim 5: The method of claim 3, wherein the filter coefficients of the sub-pixel interpolation filter are obtained according to

where the power parameter S is defined as follows:

in which p is the fractional part of the sub-pixel offset value.

Item 6: The method according to any one of paragraphs. 1-5, in which a bit (eg, the least significant bit or least significant bit) of the sub-pixel offset p is set to zero.

Item 7: The method according to any one of paragraphs. 1-5, in which the offset value p is incremented by 1, and thereafter the least significant bit of the incremented value is set to a value of zero prior to using this value to obtain the filter coefficients.

Item 8: A video coding method, the method comprises block intra-prediction processing comprising interpolated sub-pixel filtering applied to the reference samples;

wherein the sub-pixel interpolation filter is a combination of filters of at least one of the filters determined according to the sub-pixel offset.

Claim 9: The method of claim 8, wherein the interpolation filter is an anti-aliasing filter that is a parallel combination of two phase dependent linear filters and a low pass filter.

Claim 10: The method of claim 8, wherein the interpolation filter is a sharpening filter that is a parallel combination of one phase-dependent linear filter and a power-varying high-pass filter.

Claim 11: The method of claim 10, wherein the power variable high pass filter is a phase dependent high pass filter whose output is multiplied by a phase dependent power parameter.

Item 12: The method of claim 9, wherein the filter combination is the equivalent of a 4-tap filter having coefficients given according to the equations:

where p is the position p of the fractional sample.

Claim 13: The method of claim 11 wherein the filter combination is the equivalent of a 4-tap filter having coefficients given according to the equations:

in which the power parameter S is defined as follows:

Claim 14: The method according to any of the preceding claims, wherein the least significant bit of position p of the fractional sample is set to zero.

Item 15: The method according to any one of paragraphs. 8-13, in which the position p of the fractional sample is incremented by 1, and thereafter the least significant bit of the incremented value is set to zero before being used to obtain the interpolation filter coefficients.

Claim 16: The method according to claim 10 or claim 11, wherein the clipping operation is performed to output a power-changing high-pass filter.

Item 17: Encoder (20) containing the processing circuit for performing the method according to any one of paragraphs. 1-16.

Item 18: Decoder (30) containing the processing circuit for performing the method according to any one of paragraphs. 1-16.

Item 19: A computer program product containing program code for performing the method according to any one of paragraphs. 1-16.

Item 20: A decoder comprising:

one or more processors; and

a computer-readable medium coupled to the processors and storing programs for execution by the processor, wherein the programs, when executed by the processor, cause the decoder to execute the method of any one of claims. 1-16.

Item 21: An encoder containing:

one or more processors; and

a computer-readable medium coupled to the processors and storing programs for execution by the processor, wherein the programs, when executed by the processor, cause the encoder to execute the method of any one of claims. 1-16.

Item 22: A method for intra or inter prediction of video coding technology, the method comprises:

obtaining a reference sample (ie, at least one reference sample in the example, the number of reference samples may be four);

getting the position of a fractional sample; and

filtering using the reference sample sub-pixel interpolation filter to obtain a predicted sample value in which the filter coefficients of the sub-pixel interpolation filter are obtained according to

являются коэффициентами фильтра.where S is the power parameter, p is the position of the fractional sample,

are the filter coefficients.

Claim 23: The method of claim 22, wherein the method further comprises:

obtaining the value of the SEL signal by taking the N-th most significant bit of the position p of the fractional sample, N being a positive integer;

obtaining an intermediate value q, in which q is set to the value of the fractional sample position p when the signal SEL is zero; or q is set to - (not (31 xor p)) when the SEL signal is non-zero;

obtaining the power parameter S using one additional operation and shift operations:

<< 1) +

) >> 3S = ((

<< 1) +

) >> 3

Item 24: The method according to item 23, where N is 5.

Item 25: The method according to any one of paragraphs. 22-24, in which the sub-pixel interpolation filter is the equivalent of a 4-tap filter.

интерполяционного фильтра и гдеItem 26: The method according to any one of paragraphs. 22-25, in which the power parameter S is calculated in parallel with the coefficient z of the linear filter, in which the coefficient z of the linear filter is used to obtain the coefficient

interpolation filter and where

Item 27: The method according to any one of paragraphs. 22-25, in which a bit (eg, the least significant bit or lower bit) of the position p of the fractional sample is set to zero.

Item 28: The method according to any one of paragraphs. 22-26, in which the position p of the fractional sample is incremented by 1 and then the least significant bit of the incremented value is set to zero before the value is used to obtain the filter coefficients.

Item 29: A video coding method, the method comprises block intra prediction processing comprising sub-pixel interpolation filtering applied to the reference samples;

wherein the sub-pixel interpolation filter is a combination of filters of at least one of the filters determined according to the position of the fractional sample.

Claim 29: The method of claim 28, wherein the interpolation filter is an anti-aliasing filter that is a parallel combination of two phase dependent linear filters and a low pass filter.

Claim 30: The method of Claim 29, wherein the interpolation filter is a sharpening filter that is a parallel combination of one phase-dependent linear filter and a power-varying high-pass filter.

Claim 31: The method of claim 30, wherein the power varying high pass filter is an FIR phase independent high pass filter whose output is multiplied by a phase dependent power parameter.

Item 32: Encoder (20) containing the processing circuit for performing the method according to any one of paragraphs. 22-31.

Item 33: decoder (30) containing the processing circuit for performing the method according to any one of paragraphs. 22-31.

Item 34: A computer program product containing program code for performing the method according to any one of paragraphs. 22-31.

Item 35: A decoder comprising:

one or more processors; as well as

a computer-readable medium coupled to the processors and storing programs for execution by the processor, wherein the programs, when executed by the processor, cause the decoder to execute the method of any one of claims. 22-31.

Item 36: An encoder containing:

one or more processors; and

a computer-readable medium coupled to the processors and storing programs for execution by the processor, wherein the programs, when executed by the processor, cause the encoder to execute the method of any one of claims. 22-31.

List of reference positions

Fig. 1A

10 video encoding system

12 source device

13 communication channel

14 destination device

16 image source

17 image data

18 preprocessor

19 preprocessed image data

20 video encoder

21 encoded image data

22 communication interface

28 communication interface

30 video decoder

31 decoded image data

32 postprocessor

33 post-processed image data

34 display device

Fig1B.

40 video encoding system

41 imaging device(s)

42 antenna

43 processor(s)

44 memory

45 display device

46 processing scheme

20 video encoder

30 video decoder

Fig. 2.

17 image (data)

19 preprocessed image (data)

20 encoder

21 encoded image data

201 inputs (interface)

204 calculation of remainder [block or step]

206 conversion processing unit

208 quantization block

210 inverse quantizer

212 inverse conversion processing unit

214 recovery block

220 loop filter unit

230 Decoded Picture Buffer (DPB)

260 mode selection block

270 entropy coding block

272 output (interface)

244 inter prediction block

254 intra prediction block

262 separation unit

203 image block

205 residual block

213 restored residual block

215 restored block

221 filter blocks

231 inventions decoded

265 prediction block

266 syntax elements

207 conversion factors

209 quantum coefficients

211 dequantized coefficients

Fig. 3

21 encoded image data

30 video decoder

304 entropy decoding block

309 quantized coefficients

310 inverse quantizer

311 dequantized coefficients

312 inverse conversion processing unit

313 rebuilt residual block

314 recovery block

315 rebuilt block

320 loop filter

321 filtered blocks

330 DBP decoded picture buffer

331 decoded pictures

360 mode application block

365 prediction block

366 syntax elements

344 inter prediction block

354 intra prediction block

Fig. four.

400 video encoder

410 input ports / input ports

420 RX receiver blocks

430 processor

440 TX transmitter units

450 output ports / output ports

460 memory

470 encoding module

Fig. five.

500 source device or destination device

502 processor

504 memory

506 code and data

508 operating system

510 application programs

512 tire

518 display

Fig. 23

2300 flowchart of the intra/inter prediction processing method

Fig. 24

2400 flowchart of a video encoding method

Fig. 25.

2500 internal/external prediction module

2510 reference sample acquisition block

2520 sub-pixel offset value acquisition block

2530 subpixel interpolation filter

Fig. 26

2600 video encoding module

2610 Intra prediction processing block

2620 subpixel interpolation filter

Claims (39)

1. A method for processing intra prediction or inter prediction of a video frame, comprising the steps of: get a reference sample; get the subpixel offset value and filtering, using a 4-tap interpolation filter, the reference sample to obtain a predicted sample value, wherein The filter coefficients of a 4-tap interpolation filter are defined as follows:

where p is the fractional part of the sub-pixel offset value, and c ₀ , c ₁ , c ₂ , and c ₃ are the filter coefficients. 2. The method of claim 1, wherein the least significant bit of p is set to zero. 3. The method of claim 1, wherein p is incremented by 1 and subsequently the least significant bit of the incremented p is set to zero before said p is used to derive the filter coefficients. 4. The method according to claim 1, in which the filter coefficients are calculated in accordance with

, c ₁ \u003d 16 + c ₀ , c ₂ \u003d c ₀ + p,

and the division operation is performed with rounding down. 5. An intra prediction or inter prediction processing device for a video frame, comprising: a reference sample acquisition unit configured to obtain a reference sample; a sub-pixel offset value obtaining unit, configured to obtain a sub-pixel offset value; and A 4-tap interpolation filter configured to filter the reference sample to obtain a predicted sample value, wherein The filter coefficients of a 4-tap interpolation filter are defined as follows:

where p is the fractional part _of the sub-pixel offset value, and _c0 , c1, c2, and _{c3 are} the filter coefficients. 6. The apparatus of claim 5, wherein the least significant bit p is set to zero. 7. The apparatus of claim 5, wherein p is incremented by 1 and then the least significant bit of the incremented p is set to zero before said p is used to obtain the filter coefficients. 8. The device according to claim 5, in which the filter coefficients are calculated according to

, c ₁ \u003d 16 + c ₀ , c ₂ \u003d c ₀ + p

and the division operation is performed with rounding down 9. Decoder containing: one or more processors and a permanent computer-readable medium connected to the processors and storing programs that cause, when executed by the processors, the execution by the decoder of the method according to any one of paragraphs. 1-4. 10. Permanent media containing a bit stream decoded by the method according to any one of paragraphs. 1-4. 11. A computer-readable medium that stores a program recorded thereon, wherein the program causes the computer to execute the method according to any one of claims. 1-4. 12. An encoder containing: one or more processors and a permanent computer-readable medium connected to the processors and storing programs that cause, when executed by the processors, the encoder to execute the method according to any one of paragraphs. 1-4.

Images