KR20220024716A - Single-index quantization matrix design for video encoding and decoding - Google Patents

Abstract

Different quantization matrices corresponding to different block sizes, color components and prediction modes may be transmitted. To signal the coefficients of the quantization matrices more efficiently, in one implementation, based on a size identifier (sizeId) related to the CU size in which larger sizes are listed first, and a matrix type (matrixTypeId) in which luma QMs are listed first, The unified matrix identifier matrixId is used. For example, the unified identifier is derived as follows: matrixId = N * sizeId + matrixTypeId, where N is the number of possible type identifiers, for example N = 6. This single identifier allows to reference any previously transmitted matrix when using prediction (copy), and transmitting larger matrices first avoids interpolation in the prediction process. If the block uses the intra block copy prediction mode, the QM identifier can be derived as if the block uses the inter prediction mode.

Description

Single-index quantization matrix design for video encoding and decoding

The present embodiments generally relate to a method and apparatus for quantization matrix design in video encoding or decoding.

To achieve high compression efficiency, image and video coding schemes generally use prediction and transformation to leverage spatial and temporal redundancy in video content. In general, intra or inter prediction is used to exploit intra or inter picture correlation, then the differences between the original block and the predicted block, often denoted as prediction errors or prediction residuals, are transformed, quantized, and entropy coded. do. To reconstruct the video, the compressed data is decoded by inverse processes corresponding to entropy coding, quantization, transform, and prediction.

According to one embodiment, there is provided a video decoding method, the method comprising: obtaining a single identifier for a quantization matrix based on a block size, a color component and a prediction mode of a block to be decoded in a picture; decoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix; obtaining the quantization matrix based on the reference quantization matrix; dequantizing transform coefficients for the block in response to the quantization matrix; and decoding the block in response to the inverse quantized transform coefficients.

According to another embodiment, there is provided a method for video encoding, the method comprising: accessing a block to be encoded in a picture; accessing a quantization matrix for the block; obtaining a single identifier for the quantization matrix based on a block size, a color component, and a prediction mode of the block; encoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix; quantizing transform coefficients for the block in response to the quantization matrix; and entropy encoding the quantized transform coefficients.

According to another embodiment, there is provided an apparatus for video decoding comprising one or more processors, the one or more processors generating a single identifier for a quantization matrix based on a block size, a color component and a prediction mode of a block to be decoded in a picture. obtain; decode a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix; obtain the quantization matrix based on the reference quantization matrix; inverse quantize transform coefficients for the block in response to the quantization matrix; and decode the block in response to the inverse quantized transform coefficients.

According to another embodiment, there is provided an apparatus for video encoding comprising one or more processors, the one or more processors accessing blocks to be encoded in a picture; access a quantization matrix for the block; obtain a single identifier for the quantization matrix based on a block size, a color component, and a prediction mode of the block; encode a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix; quantize transform coefficients for the block in response to the quantization matrix; and entropy encode the quantized transform coefficients.

According to another embodiment, there is provided an apparatus for video decoding, the apparatus comprising: means for obtaining a single identifier for a quantization matrix based on a block size, a color component and a prediction mode of a block to be decoded in a picture; means for decoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix; means for obtaining the quantization matrix based on the reference quantization matrix; means for inverse quantizing transform coefficients for the block in response to the quantization matrix; and means for decoding the block in response to the inverse quantized transform coefficients.

According to another embodiment, there is provided an apparatus for video encoding, the apparatus comprising: means for accessing a block to be encoded in a picture; means for accessing a quantization matrix for the block; means for obtaining a single identifier for the quantization matrix based on a block size, a color component, and a prediction mode of the block; means for encoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix; means for quantizing transform coefficients for the block in response to the quantization matrix; and means for entropy encoding the quantized transform coefficients.

One or more embodiments also provide a computer program comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform an encoding method or a decoding method according to any of the foregoing embodiments. One or more of the present embodiments also provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to the methods described above. One or more embodiments also provide a computer-readable storage medium having stored thereon a bitstream generated according to the methods described above. One or more embodiments also provide a method and apparatus for transmitting or receiving a bitstream generated according to the methods described above.

1 illustrates a block diagram of a system in which aspects of the present embodiments may be implemented.
2 illustrates a block diagram of one embodiment of a video encoder.
3 illustrates a block diagram of one embodiment of a video decoder.
4 illustrates that transform coefficients are inferred to be zero for block sizes greater than 32 in VVC Draft 5;
5 illustrates a fixed prediction tree described in JCTVC-H0314.
6 illustrates prediction from a larger magnitude (decimation), according to one embodiment.
7 illustrates a combination of prediction from a larger size and decimation for a rectangular block, according to one embodiment.
8 illustrates a QM derivation process for a rectangular block for chroma, according to one embodiment.
9 illustrates a QM derivation process (adaptation to 4:2:2 format) for a rectangular block for chroma, according to one embodiment.
10 illustrates a QM derivation process (adaptation to 4:4:4 format) for a rectangular block for chroma.
11 illustrates a flowchart for parsing a scaling list data syntax structure, according to an embodiment.
12 illustrates a flow diagram for encoding a scaling list data syntax structure, according to an embodiment.
13 illustrates a flow diagram for a QM derivation process, according to one embodiment.

1 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 100 may be implemented as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers, such as various including, but not limited to, electronic devices. Elements of system 100, alone or in combination, may be implemented as a single integrated circuit, multiple ICs, and/or discrete components. For example, in at least one embodiment, the processing and encoder/decoder elements of system 100 are distributed across a plurality of ICs and/or discrete components. In various embodiments, system 100 is communicatively coupled to other systems, or other electronic devices, for example, via a communications bus or via dedicated input and/or output ports. In various embodiments, system 100 is configured to implement one or more of the aspects described herein.

System 100 includes, for example, at least one processor 110 configured to execute instructions loaded therein to implement various aspects described herein. Processor 110 may include embedded memory, input/output interfaces, and various other circuits as known in the art. System 100 includes at least one memory 120 (eg, a volatile memory device and/or a non-volatile memory device). System 100 includes storage device 140, which includes non-volatile memory and / or may include volatile memory. Storage device 140 may include, as non-limiting examples, an internal storage device, an attached storage device, and/or a network accessible storage device.

System 100 includes, for example, an encoder/decoder module 130 configured to process data to provide encoded video or decoded video, wherein the encoder/decoder module 130 includes its own processor and It may contain memory. Encoder/decoder module 130 represents a module(s) that may be included in a device to perform encoding and/or decoding functions. As is known, a device may include one or both of encoding and decoding modules. Further, the encoder/decoder module 130 may be implemented as a separate element of the system 100 , or may be integrated into the processor 110 as a combination of hardware and software as known to those skilled in the art. there is.

Program code to be loaded onto processor 110 or encoder/decoder 130 to perform various aspects described herein may be stored in storage device 140 and subsequently executed by processor 110 . may be loaded onto the memory 120 for According to various embodiments, one or more of the processor 110 , the memory 120 , the storage device 140 , and the encoder/decoder module 130 is one or more of the various items during performance of the processes described herein. can be saved. These stored items include input video, decoded video or portions of decoded video, bitstream, matrices, variables, and intermediate or final results from processing of equations, formulas, operations and arithmetic logic. can, but is not limited to.

In various embodiments, memory internal to processor 110 and/or encoder/decoder module 130 is used to store instructions and to provide working memory for necessary processing during encoding or decoding. However, in other embodiments, memory external to the processing device (eg, the processing device may be either the processor 110 or the encoder/decoder module 130 ) is used for one or more of these functions. do. The external memory may be memory 120 and/or storage device 140 , such as dynamic volatile memory and/or non-volatile flash memory. In various embodiments, an external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, a high-speed external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2, HEVC or VVC.

Input to elements of system 100 may be provided through various input devices, as indicated by block 105 . These input devices include (i) an RF portion that receives an RF signal transmitted over the air, for example by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and/or (iv) an HDMI input terminals, but are not limited thereto.

In various embodiments, the input devices of block 105 have associated respective input processing elements as is known in the art. For example, the RF portion may include (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a band of frequencies), (ii) downconverting the selected signal; (iii) selecting a (eg) signal frequency band, which may be referred to as a channel in certain embodiments by bandlimiting back to a narrower band of frequencies, (iv) demodulating the down-converted and band-limited signal; elements suitable for (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF portion of various embodiments may contain one or more elements for performing these functions, eg, frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error corrector. and demultiplexers. The RF portion may include a tuner that performs various such functions, including, for example, downconverting the received signal to a lower frequency (eg, an intermediate frequency or near baseband frequency) or to baseband. can In one set top box embodiment, the RF portion and its associated input processing element receive an RF signal transmitted over a wired (eg, cable) medium, and perform filtering, downconversion, and filtering back to a desired frequency band. frequency selection by Various embodiments rearrange the order of the foregoing (and other) elements, remove some of these elements, and/or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, eg, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF portion includes an antenna.

Additionally, the USB and/or HDMI terminals may include respective interface processors for connecting the system 100 to other electronic devices via USB and/or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 110 as desired. . Similarly, aspects of USB or HDMI interface processing may be implemented within processor 110 or within separate interface ICs as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements including, for example, a processor 110 and an encoder/decoder 130 operating in combination with memory and storage elements. , process the datastream as needed for presentation on the output device.

The various elements of system 100 may be provided within an integrated housing. Within the integrated housing, the various elements are interconnected and connected using a suitable connection arrangement 115 , for example, an internal bus such as is known in the art, including an I2C bus, wiring, and printed circuit boards. Data can be transmitted between

The system 100 includes a communication interface 150 that enables communication with other devices via a communication channel 190 . Communication interface 150 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 190 . Communication interface 150 may include, but is not limited to, a modem or network card, and communication channel 190 may be implemented within, for example, wired and/or wireless media.

Data is streamed to system 100 using a Wi-Fi network, such as IEEE 802.11, in various embodiments. The Wi-Fi signal of these embodiments is received via a communication interface 150 and communication channel 190 adapted for Wi-Fi communication. The communication channel 190 of these embodiments is to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications. normally connected. Other embodiments provide the streamed data to the system 100 using a set top box that delivers the data via the HDMI connection of the input block 105 . Still other embodiments use the RF connection of the input block 105 to provide streamed data to the system 100 .

System 100 may provide an output signal to various output devices including display 165 , speakers 175 , and other peripheral devices 185 . Other peripheral devices 185 include, in various examples of embodiments, one or more of a standalone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 100 . . In various embodiments, control signals are communicated to system 100 and display 165 using signaling such as AV.Link, CEC, or other communication protocols that enable device-to-device control with or without user intervention. ), speakers 175 , or other peripheral devices 185 . Output devices may be communicatively coupled to system 100 via dedicated connections via respective interfaces 160 , 170 , 180 . Alternatively, the output devices may be connected to the system 100 using the communication channel 190 via the communication interface 150 . Display 165 and speakers 175 may be integrated into a single unit with other components of system 100 in an electronic device, eg, a television. In various embodiments, display interface 160 includes a display driver, eg, a timing controller (T Con) chip.

For example, if the RF portion of input 105 is part of a separate set-top box, display 165 and speaker 175 may alternatively be separate from one or more of the other components. In various embodiments where display 165 and speakers 175 are external components, the output signal may be provided via dedicated output connections including, for example, HDMI ports, USB ports, or COMP outputs. .

2 illustrates an example video encoder 200 , such as a High Efficiency Video Coding (HEVC) encoder. 2 may also illustrate an encoder using techniques similar to HEVC, such as an encoder for which improvements to the HEVC standard are made or a Versatile Video Coding (VVC) encoder being developed by the Joint Video Exploration Team (JVET).

In this application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “encoded” or “coded” may be used interchangeably, and the terms “image”, “picture” " and "frame" may be used interchangeably. Generally, though not necessarily, the term "reconstructed" is used at the encoder side, while "decoded" is used at the decoder side.

Before being encoded, the video sequence is subjected to pre-encoding processing 201 , eg, color transformation (eg, RGB 4:4:4 to YCbCr 4:2:0 transformation) to the input color picture. or performing remapping of the input picture components to obtain a signal distribution that is more resilient to compression (eg, using a histogram equalization of one of the color components). Metadata is associated with preprocessing and can be attached to the bitstream.

In the encoder 200 , the picture is encoded by encoder elements as described below. The picture to be encoded is partitioned 202 and processed, for example, into units of CUs. Each unit is encoded using, for example, either intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction 260 . In inter mode, motion estimation 275 and compensation 270 are performed. The encoder determines whether to use either intra mode or inter mode to encode the unit ( 205 ), and indicates the intra/inter decision by, for example, a prediction mode flag. Prediction residuals are calculated, for example, by subtracting 210 the predicted block from the original image block.

The prediction residuals are then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded 245 to output the bitstream. The encoder can skip the transform and apply the quantization directly to the untransformed residual signal. The encoder can bypass both transform and quantization, ie the residual is coded directly without applying transform or quantization processes.

The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are inverse quantized ( 240 ) and inverse transformed ( 250 ) to decode the prediction residuals. Combining the decoded prediction residuals with the predicted block (255), the image block is reconstructed. In-loop filters 265 are applied to the reconstructed picture to, for example, perform deblocking/Sample Adaptive Offset (SAO) filtering to reduce encoding artifacts. The filtered image is stored in the reference picture buffer 280 .

3 illustrates a block diagram of an example video decoder 300 . At the decoder 300 , the bitstream is decoded by decoder elements as described below. The video decoder 300 generally performs a decoding pass reciprocal to the encoding pass as described in FIG. 2 . Encoder 200 also performs video decoding, typically as part of encoding the video data.

In particular, the input of the decoder comprises a video bitstream, which may be generated by the video encoder 200 . The bitstream is first entropy decoded 330 to obtain transform coefficients, motion vectors, and other coded information. The picture partition information indicates how the picture is partitioned. Accordingly, the decoder may divide the picture according to the decoded picture partitioning information ( 335 ). The transform coefficients are inverse quantized (340) and inverse transformed (350) to decode the prediction residuals. Combining the decoded prediction residuals with the predicted block ( 355 ), the image block is reconstructed. The predicted block may be obtained 370 from intra prediction 360 or motion-compensated prediction (ie, inter prediction) 375 . In-loop filters 365 are applied to the reconstructed image. The filtered image is stored in the reference picture buffer 380 .

The decoded picture is subjected to post-decoding processing 385 , for example, inverse color conversion (eg, YCbCr 4:2:0 to RGB 4:4:4 conversion) or pre-encoding processing 201 . Inverse remapping that performs the reverse of the remapping process may be further performed. The post-decoding process may use metadata derived from the pre-encoding process and signaled in the bitstream.

The HEVC specification allows the use of quantization matrices in the inverse quantization process, where the transformed coefficients are scaled by the current quantization step and further scaled by the quantization matrix (QM) as follows:

d[ x ][ y ] = Clip3( coeffMin, coeffMax, ( ( TransCoeffLevel[ xTbY ][ yTbY ][ cIdx ][ x ][ y ] * m[ x ][ y ] * levelScale[ qP%6 ] << ( qP / 6 ) ) + ( 1 << ( bdShift - 1 ) ) ) >> bdShift )

here:

· TransCoeffLevel [...] are the transformed coefficient absolute values for the current block identified by its spatial coordinates xTbY, yTbY and its component index cIdx.

· x and y are horizontal/vertical frequency indices.

· qP is the current quantization parameter.

· Multiplication by left shift by levelScale[qP%6] and (qP/6) is equivalent to multiplication by quantization step qStep=(levelScale[qP%6] << (qP/6)).

· m[...][...] is a two-dimensional quantization matrix. Here, since the quantization matrix is used for scaling, it may also be referred to as a scaling matrix.

· bdShift is an additional scaling factor to account for the image sample bit depth. The term (1 << (bdShift-1)) serves the purpose of rounding to the nearest integer.

· d[...] are the resulting inverse quantized transformed coefficient absolute values.

The syntax used by HEVC to transmit quantization matrices is described below:

The following may be noted:

· A different matrix is specified for each transform size (sizeId). In the scaling list data syntax structure, the scaling matrix is scanned into a 1-D scaling list (eg, ScalingList).

· For a given transform size, six matrices are specified for intra/inter coding and Y/Cb/Cr components.

· A matrix can be any of the following:

o If scaling_list_pred_mode_flag is equal to 0, it is copied from a previously transmitted matrix of the same size (reference matrixId is obtained as matrixId-scaling_list_pred_matrix_id_delta)

o Copied from default values specified in the standard (if both scaling_list_pred_mode_flag and scaling_list_pred_matrix_id_delta are 0)

o Using exp-Golomb entropy coding, it is fully specified in right-to-right diagonal scanning order in DPCM coding mode.

· For block sizes greater than 8x8, only 8x8 coefficients are transmitted to signal the quantization matrix to save coded bits. The coefficients are then interpolated using zero-hold (ie, repetition), except for the DC coefficient that is explicitly transmitted.

The use of HEVC-like quantization matrices was adopted in VVC draft 5 based on the contribution JVET-N0847 (O. Chubach, et al., "CE7-related: Support of quantization matrices for VVC," JVET-N0847, Geneva, CH, See March 2019). The scaling_list_data syntax is adapted to the VVC codec as shown below.

In the design of VVC draft 5 with JVET-N0847 adoption, as in HEVC, QM is identified by two parameters, matrixId and sizeId. This is illustrated in the two tables below.

Table 1: Block size identifiers (JVET-N0847)

Table 2: QM Type Identifier (JVET-N0847)

Note: MODE_INTRA QMs are also used for MODE_IBC (Intra Block Copy).

Combinations of both identifiers are shown in the table below:

Table 3: (matrixId, sizeId) combinations (JVET-N0847)

As in HEVC, for block sizes greater than 8x8, only 8x8 coefficients and DC coefficients are transmitted. A QM of the correct size is reconstructed using zero-hold interpolation. For example, for a 16x16 block, all coefficients are repeated twice in both directions, and then the DC coefficients are replaced with the transmitted coefficients.

For rectangular blocks, the size (sizeId) maintained for QM selection is the largest of the larger dimensions, ie, width and height. For example, for a 4x16 block, a QM for a 16x16 block size is selected. The reconstructed 16x16 matrix is then vertically decimated by a factor of 4 to obtain the final 4x16 quantization matrix (ie, 3 out of 4 lines are skipped).

Hereinafter, with respect to sizeId and the square block size in which it is used, the QMs for a given family of block sizes (square or rectangular) are referred to as size-N. For example, for block sizes 16x16 or 16x4, QMs are identified as size-16 (sizeId 4 in VVC Draft 5). The size-N notation is used to distinguish the exact block shape and number of signaled QM coefficients (limited to 8x8 as shown in Table 3).

Also, in VVC Draft 5, for size-64, the QM coefficients for the lower right quadrant are not transmitted (they are inferred to be zero and are referred to as “zero-outs” hereinafter). This is implemented by the condition "x>=4 && y>=4" in the scaling_list_data syntax. This prevents transmission of QM coefficients that are never used by the transform/quantization process. Indeed, in VVC, for transform block sizes greater than 32 in any dimension (64xN, Nx64, N <= 64), no transformed coefficient with an x/y frequency coordinate greater than or equal to 32 is transmitted and It is inferred to be zero, and consequently, no quantization matrix coefficients are needed to quantize it. This is illustrated in Figure 4, where the hatched regions correspond to transform coefficients inferred to be zero.

Compared to HEVC, VVC requires more quantization matrices due to the larger number of block sizes. However, in VVC Draft 5, QM prediction is still limited to copies of identical block-size matrices, which may result in wasted bits. Also, the syntax related to QMs is more complicated because of using block size 2x2 only for chroma and using block size 64x64 only for luma in VVC. Also, as in HEVC, JVET-N0847 describes a specific matrix derivation process for each block size.

, CA, USA, February 2012 참조)에서 탐구되었다.Some QM prediction techniques are used during HEVC standardization, for example, JCTVC-E073 (see J. Tanaka, et al., “Quantization Matrix for HEVC,” JCTVC-E073, Geneva, CH, March 2011) and JCTVC-H0314 (Y Wang, et al., “Layered quantization matrices representation and compression,” JCTVC-H0314, San Jos

, CA, USA, February 2012).

JCTVC-E073: QMs are transmitted in a specific parameter set (QMPS). Within QMPS, QMs are transmitted in increasing size order (sizeId/matrixId similar to HEVC). Prediction (=copy) from any previously coded QM coefficients including the previous QMPS is proposed. Up-conversion with linear interpolation is used to adapt from a smaller reference QM, whereas simple downsampling is used to adapt from a larger reference QM. It was finally rejected during HEVC standardization.

JCTVC-H0314: QMs are transmitted in order from largest to smallest. As shown in FIG. 5 , using a fixed prediction tree (no explicit reference indexing), it is possible to copy a previously transmitted QM instead of transmitting a new one. Simple downsampling is used when the reference QM is larger. It was finally rejected during HEVC standardization.

These two proposals relate to HEVC and do not address the complexities introduced by VVC.

This application simplifies the quantization matrix signaling and prediction process of VVC Draft 5 (after adoption of JVET-N0847) while enhancing them so that any QM can be predicted from any previously signaled by incorporating one or more of the following Suggest to do:

- unifying the QM index to include both size and type, such that the reference index difference can address any previously transmitted;

- transmitting the quantization matrices in decreasing block size order;

- designating the prediction process as either a copy or a decimation process, if necessary;

- To transmit all QM coefficients for size-64 so that size-64 QMs can be used as predictors.

Furthermore, a QM derivation process comprising upsampling for blocks larger than 8x8 and downsampling for rectangular blocks is described as selecting a QM index according to block parameters and adapting the QM signaled size to the actual block size. .

For ease of notation, consider the process of predicting a quantization matrix from default values or from previously transmitted as a QM prediction process, and a QM derivation process of adapting the transmitted or predicted QM to the size and chroma format of the transform block Consider it as a process. The QM prediction process may be part of a process of parsing scaling list data at the picture level, for example. The derivation process is generally at a lower level, eg the transform block level. Various aspects are provided in greater detail below, followed by draft text examples and performance results.

· Derivation and use of a single matrix index to identify a QM. A single identifier allows referencing any previously transmitted matrix when using prediction (copy), and transmitting larger matrices first avoids interpolation in the prediction process.

· The QM prediction process, which consists of copying or decimating a previously signaled QM (reference QM), may be a transmit, predict or default reference QM.

· Then, for a given transform block, a QM derivation process consisting of selecting a QM index based on block size, color component and prediction mode adapts the size of the selected QM to the size of the block. The resizing process is based on bit shifts of the x and y coordinates in the transform block to index the coefficients of the selected QM.

· Transmission of all coefficients for size-64 QMs

Compared to VVC Draft 5, these aspects simplify the specification (reduce text changes in half compared to JVET-N0847) and result in significant bit savings (bit cost of scaling_list_data can be cut in half).

Unified QM Index

A QM used for quantization/inverse quantization of a transform block is identified by one single parameter matrixId. In one embodiment, the aggregated matrixId (QM index) is a composite of:

- A size identifier related to the CU size (ie, the CU surrounding a square shape since only square size matrices are transmitted) rather than the block size. Note here that for either luma or chroma, the size identifier is controlled by the luma block size, eg max(luma block width, luma block height). When luma and chroma trees are separated, for chroma, “CU size” will refer to the size of the block projected on the luma plane.

- Since luma QMs can be larger than chroma (eg in the case of 4:2:0 chroma format), a matrix type that enumerates the luma QMs first.

According to this embodiment, QM index derivation is illustrated in Tables 4 and 5 and Equation 1.

Table 4: Size identifiers (suggested)

Table 5: Matrix Type Identifiers (Proposed)

The unified matrixId is derived as follows:

matrixId = N * sizeId + matrixTypeId(1)

Here, N is the number of possible type identifiers, for example, N=6.

In another embodiment, if more than 6 QM types are defined, sizeId should be multiplied by the exact number of quantization matrix types. In other embodiments, other parameters may also be different, eg, certain block sizes, signaled matrix sizes (limited here to 8×8), or the presence of a DC coefficient. Note that QMs here are listed by decreasing block size and identified by a single index, as illustrated in Table 6.

Table 6: Unified matrixId (proposed)

QM Prediction Process

Instead of transmitting the QM coefficients, it is possible to predict the QM from default values, or from any previously transmitted value. In one embodiment, when the reference QM is of the same size, the QM is copied, otherwise decimated by a relative ratio, as illustrated in the example of FIG. 6 , a size-4 luma QM predicted from size-8 do.

Decimation is described by the following equation:

ScalingMatrix[ matrixId ][ x ][ y ] = refScalingMatrix[ i ][ j ] (2)

where matrixSize = (matrixId < 20) ? 8 : (matrixId < 26) ? 4: 2 )

x = 0 .. matrixSize - 1, y = 0 .. matrixSize - 1,

i = x << ( log2(refMatrixSize) - log2( matrixSize ) ), and

j = y << ( log2(refMatrixSize) - log2( matrixSize ) ).

where refMatrixSize matches the size of the refScalingMatrix (and thus the range of i and j variables).

In the example shown in Fig. 6, a luma size-4 QM (4x4 array: matrixSize is 4) is predicted from a luma size-8 QM that is an 8x8 array (refMatrixSize is 8); One line of two and one column of two are dropped to create a 4x4 array (ie element (2x, 2y) in the reference QM is copied to element (x, y) in the current QM).

Equation 3 takes the form:

ScalingMatrix[ matrixId ][ x ][ y ] = refScalingMatrix[ i ][ j ] (3)

x = 0 .. 3, y = 0 .. 3, i = x << 1, and j = y << 1.

When the reference QM has a DC value, if the current QM needs a DC value, it is copied as a DC value; Otherwise, it is copied to the top left QM coefficients.

This QM prediction process is part of the QM decoding process in the preferred embodiment, but may be deferred to the QM derivation process in other embodiments where decimation for prediction purposes is merged with the QM resize subprocess.

QM induction process

The proposed derivation process for the quantization matrix first selects the right QM index according to the block parameters as described above (integrated QM index), then decimation for rectangular blocks, blocks larger than the size, e.g. For example, it integrates the processes of iteration for 8x8, and chroma format adaptation into one single process. The proposed process is based on bit shifts of the x and y output coordinates. To select the right line/column of the selected QM, only a left shift followed by a right shift of the x/y output coordinates is required, as illustrated in the equation below.

m[ x ][ y ] = ScalingMatrix[ matrixId ][ i ][ j ] (4)

where i = ( x << log2MatrixSize ) >> log2( blkWidth ), and

j = ( y << log2MatrixSize ) >> log2( blkHeight ).

where log2MatrixSize is log2 of the size of ScalingMatrix[matrixId] (which is a square 2D array), blkWidth and blkHeight are the width and height of the current transform block, respectively, x is in the range of 0 to blkWidth-1, and y is 0 to blkHeight- It is in the range of 1.

In the following, several examples are provided to illustrate the QM derivation process. In the example illustrated in Figure 7, the QM for a luma 16x8 block is derived from the luma size-16 QM, which is actually an 8x8 array plus DC coefficient. For this example, blkWidth equals 16, blkHeight equals 8, and log2MatrixSize equals 3, so Equation 5 takes the form:

m[ x ][ y ] = ScalingMatrix[ matrixId ][ i ][ j ] (5)

where i= (x << 3) >> 4, and j= (y << 3) >> 3, where x= 0..15 and y= 0..7.

Here, x is shifted right by 1, and y is unchanged (ie, column i of the selected QM is copied to columns 2*i and 2*i + 1 of the current QM). Also, since the selected QM has a DC coefficient, it is copied to m[0][0].

In another example as illustrated in FIG. 8 , a QM for a chroma 4×2 block for an 8×4 CU (4:2:0 format) is generated. This matches an 8x4 CU size, and its enclosing square is 8x8. Thus, the selected QM is of size-8, where the chroma QMs are coded as 4x4 arrays. Here, blkWidth equals 4, blkHeight equals 2, and log2MatrixSize equals 2, so Equation 6 takes the form:

m[ x ][ y ] = ScalingMatrix[ matrixId ][ i ][ j ] (6)

where i= (x << 2) >> 2, and j= (y << 2) >> 1 where x= 0..3 and y= 0..1.

Here, x remains unchanged and y is left shifted by one (ie, row 2y in the reference QM is copied to row y in the current QM).

In the examples below, the proposed adaptation for 4:2:2 and 4:4:4 formats is different from VVC Draft 5. Instead of finding a QM that matches the chroma block size (except for 64x64 where the chroma matrix does not exist), the size matching is based on the same (luma) CU size (i.e. the size of the block projected onto the luma plane), The coefficients are repeated if necessary. This makes the QM design independent of the chroma format.

In the example shown in FIG. 9 , a QM for a chroma 8×2 block for an 8×4 CU (4:2:2 format) is generated. The selected QM is the same as the above example illustrated in FIG. 8 , but the 4:2:2 chroma format requires twice as many columns. Here, the columns are repeated, so x is shifted right by 1 and y is still shifted left by 1. In particular, blkWidth equals 8, blkHeight equals 2, and log2MatrixSize equals 2, so Equation 7 takes the form:

m[ x ][ y ] = ScalingMatrix[ matrixId ][ i ][ j ] (7)

where i = ( x << 2 ) >> 3, and j = ( y << 2 ) >> 1, where x = 0..7 and y = 0..1.

In the example shown in FIG. 10 , a QM for a chroma 8x4 block for an 8x4 CU (4:4:4 format) is generated. The selected QM is still the same as in the example shown in FIGS. 8 and 9 , but the 4:4:4 chroma format requires twice as many rows and columns as the 4:2:0 chroma format. Here, the columns have to be repeated, so x is shifted right by 1, but (due to the rectangular shape) the decimation of the rows can be skipped, so y is not shifted. In particular, blkWidth equals 8, blkHeight equals 4, and log2MatrixSize equals 2, so Equation 8 takes the form:

m[ x ][ y ] = ScalingMatrix[ matrixId ][ i ][ j ] (8)

where i = ( x << 2 ) >> 3, and j = ( y << 2 ) >> 2, where x = 0..7 and y = 0..3.

Number of Coefficients Transmitted for Size-64

In one embodiment, to enable prediction of a smaller QM from size-64, all coefficients for size-64 are transmitted in the scaling list syntax, even though the lower-right quadrant is never used by the VVC transform and quantization process. do. In general, it is possible to transmit all coefficients of the largest QM.

However, since the syntax element scaling_list_delta_coef can be set to zero for the lower right quadrant of size-64 QM when not used as a predictor, this increased number of transmitted coefficients and It is worth noting that the overhead involved can be limited to 2x16 bits in the worst case: for the lower right quadrant, 4x4 = 16 delta coefficients that take 1 bit if each is forced to zero (coded exp-Golomb) are signaled, and there are two size-64 QMs (luma intra/inter).

The tests are described in Table 8 and show that this overhead is negligible compared to the gains brought by the prediction improvement.

In another embodiment, the coefficients for the lower-right quadrant of the size-64 QM are not transmitted as part of the size-64 QM signaling, but are transmitted as a supplemental parameter when a smaller QM is first predicted from a given size-64 QM.

Table 7 provides some comparisons of the method described in JVET-N0847 and the proposed method:

Table 7

Hereinafter, some syntax and semantics according to an embodiment are described.

PPS syntax and semantics (local adaptation)

pps_scaling_list_data_present_flag equal to 1 specifies that scaling list data used for pictures referencing the PPS is derived based on the scaling lists specified by the active SPS and the scaling lists specified by the PPS. pps_scaling_list_data_present_flag equal to 0 specifies that the scaling list data used for pictures referencing the PPS is inferred to be the same as those specified by the active SPS. When scaling_list_enabled_flag is equal to 0, the value of pps_scaling_list_data_present_flag will be equal to 0. When scaling_list_enabled_flag is equal to 1, sps_scaling_list_data_present_flag is equal to 0 and pps_scaling_list_data_present_flag is equal to 0, and default scaling matrices are used to derive the Array ScalingMatrix as described in Scaling List Data Semantics as specified in clause 7.4.5.

Note that this syntax/semantics is an example intended to be close to the HEVC standard or VVC draft, and is not restrictive. For example, the scaling_list_data carriage is not limited to SPS or PPS, and may be transmitted by other means.

Scaling List Data Syntax/Semantics (Simplified)

scaling_list_pred_mode_flag[matrixId] equal to 0 specifies that the scaling matrix is derived from the values of the reference scaling matrix. The reference scaling matrix is specified by scaling_list_pred_matrix_id_delta[matrixId]. scaling_list_pred_mode_flag[matrixId] equal to 1 specifies that the values of the scaling list are explicitly signaled.

scaling_list_pred_matrix_id_delta[matrixId] specifies a reference scaling matrix used to derive a scaling matrix as follows. The value of scaling_list_pred_matrix_id_delta[ matrixId ] will be in the range of 0 to matrixId (including the boundary value).

When scaling_list_pred_mode_flag[ matrixId ] is equal to 0:

- The variable refMatrixSize and the array refScalingMatrix are first derived as follows:

o If scaling_list_pred_matrix_id_delta[ matrixId ] is equal to 0, the following applies to set default values:

dot refMatrixSize is set equal to 8,

dot If matrixId is even,

refScalingMatrix = (9)

{

{ 16, 16, 16, 16, 16, 16, 16, 16 } // placeholders for intra default values

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

},

dot Otherwise

refScalingMatrix = (10)

{

{ 16, 16, 16, 16, 16, 16, 16, 16 } // placeholders for inter default values

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

{ 16, 16, 16, 16, 16, 16, 16, 16 }

},

o Otherwise (scaling_list_pred_matrix_id_delta[ matrixId ] exceeds 0), the following applies:

refMatrixId = matrixId - scaling_list_pred_matrix_id_delta[ matrixId ] (11)

refMatrixSize = (refMatrixId < 20) ? 8 : (refMatrixId < 26) ? 4: 2 ) (12)

refScalingMatrix = ScalingMatrix[ refMatrixId ] (13)

- Then the array ScalingMatrix[ matrixId ] is derived as:

ScalingMatrix[ matrixId ][ x ][ y ] = refScalingMatrix[ i ][ j ] (14)

where matrixSize = (matrixId < 20) ? 8 : (matrixId < 26) ? 4: 2 )

x = 0 .. matrixSize - 1, y = 0 .. matrixSize - 1,

i = x << ( log2(refMatrixSize) - log2( matrixSize ) ), and

j = y << ( log2(refMatrixSize) - log2( matrixSize ) )

scaling_list_dc_coef_minus8[matrixId] + 8 specifies the first value of the scaling matrix when relevant, as described in clause xxx. The value of scaling_list_dc_coef_minus8[ matrixId ] must be in the range of -7 to 247 (including the boundary value).

When scaling_list_pred_mode_flag[ matrixId ] is equal to 0, if scaling_list_pred_matrix_id_delta[ matrixId ] is greater than 0 and refMatrixId < 14, the following applies:

- If matrixId<14, scaling_list_dc_coef_minus8[ matrixId ] is inferred to be the same as scaling_list_dc_coef_minus8[ refMatrixId ],

- Otherwise, ScalingMatrix[ matrixId ][ 0 ][ 0 ] is set equal to scaling_list_dc_coef_minus8[ refMatrixId ] + 8

When scaling_list_pred_mode_flag[matrixId] is equal to 0, scaling_list_pred_matrix_id_delta[matrixId] is equal to 0 (indicating default values) and if matrixId<14, scaling_list_dc_coef_minus8[matrixId] is inferred to be equal to 8

scaling_list_delta_coef specifies the difference between the current matrix coefficient ScalingList[matrixId][i] and the previous matrix coefficient ScalingList[matrixId][i-1] when scaling_list_pred_mode_flag[matrixId] is equal to 1. The value of scaling_list_delta_coef must be in the range of -128 to 127 (including the boundary value). The value of ScalingList[ matrixId ][ i ] will be greater than 0.

When present (i.e., when scaling_list_pred_mode_flag[matrixId] is equal to 1), the array ScalingMatrix[matrixId] is derived as follows:

ScalingMatrix[ matrixId ][ i ][ j ] = ScalingList[ matrixId ][ k ] (15)

where k = 0 .. coefNum - 1,

i = diagScanOrder[ log2(coefNum)/2 ][ log2(coefNum)/2 ][ k ][ 0 ], and

j = diagScanOrder[ log2(coefNum)/2 ][ log2(coefNum)/2 ][ k ][ 1 ]

The main simplifications compared to the JVET-N0847 syntax are the elimination of the () loop and the simplification of indexing from [sizeId][matrixId] to [matrixId].

"Article xxx" refers to a section number that has not been determined to be introduced in the VVC specification that is consistent with the scaling matrix derivation process of this document.

Note that this syntax/semantics is an example intended to be close to the HEVC standard or VVC Draft 5, and is not restrictive. For example, the coefficient range is not limited to 1...255, and may be, for example, 1..127 (7 bits), or -64..63. Also, it is not limited to 30 QMs organized in 6 types x 5 sizes (there may be 8 types, and fewer or more sizes. See Table 5, which can be adapted. Then: Simple adaptations to the condition for the presence of DC coefficients and to coefNum will be required.) The type of QM prediction (only copy here) is not restrictive either. For example, a scaling factor or offset may be added, and explicit coding above the prediction may be added as a residual. The same is true for the method used for coefficient transmission (here DPCM), the presence of DC coefficients, and the number of coefficients that are fixed (only a subset can be transmitted).

Regarding default values, it is not limited to the two default QMs associated with MODE_INTRA and MODE_INTER, but with all 16 values until the relevant default values are agreed upon (eg, the same default QMs as HEVC can be selected). It is filled as here.

Also, the number of coefficients to be signaled coefNum can be expressed mathematically instead of a sequence of comparisons, with the same result: coefNum= Min(64, 4096 >> ( (matrixId + 4) / 6) * 2), which is HEVC Alternatively, it is closer to the current VVC draft style, but introduces division that may not be welcome.

Here, the larger matrices are transmitted first, and the prediction reference (indicated by scaling_list_pred_matrix_id_delta) (e.g., when scaling_list_pred_matrix_id_delta is 0) is the block size or type for which it is intended, or any previously transmitted regardless of default values. Note that a single index is used to be a matrix.

11 illustrates a process 1100 for parsing a scaling list data syntax structure, according to one embodiment. In this embodiment, the input is a coded bitstream and the output is an array of ScalingMatrix. Details regarding the DC value are omitted for clarity. In particular, in step 1110, the QM prediction mode is decoded from the bitstream. If QM is predicted ( 1120 ), the decoder further determines whether QM is speculated (predicted) or signaled in the bitstream according to the aforementioned flag. In step 1130, the decoder decodes QM prediction data from the bitstream, which is necessary to infer QM, eg, QM index difference scaling_list_pred_matrix_id_delta when not signaled. Next, the decoder determines whether the QM is predicted from default values (eg, when scaling_list_pred_matrix_id_delta is 0) or from a previously decoded QM ( 1140 ). If the reference QM is the default QM, the decoder selects the default QM as the reference QM ( 1150 ). For example, depending on the parity of matrixId, there may be several default QMs to choose from. Otherwise, the decoder selects the previously decoded QM as the reference QM ( 1155 ). The index of the reference QM is derived from the matrixId and the aforementioned index difference. In step 1160, the decoder predicts a QM from the reference QM. Prediction is made with a simple copy if the reference QM is the same size as the current QM, or with decimation if it is larger than expected. The result is stored in ScalingMatrix[matrixId].

If QM is not predicted ( 1120 ), the decoder determines the number of QM coefficients to be decoded from the bitstream according to matrixId ( 1170 ). For example: 64 if matrixId is less than 20, 16 if matrixId is between 20 and 25, 4 otherwise. In step 1175, the decoder decodes the relevant number of QM coefficients from the bitstream. At step 1180 , the decoder organizes the decoded QM coefficients into a 2D matrix according to a scanning order, eg, a diagonal scan. The result is stored in ScalingMatrix[matrixId]. Using ScalingMatrix[matrixId], the decoder may use a QM derivation process to obtain a quantization matrix m[][] for inverse quantizing the transform block, which may be of a non-square shape and/or a different chroma format.

In step 1190, the decoder checks whether the current QM is the last QM to be parsed. If not, control returns to step 1110; Otherwise, the QM parsing process is aborted when all QMs are parsed from the bitstream.

12 illustrates a process 1200 of encoding a scaling list data syntax structure at the encoder side, according to an embodiment. At the encoder side, QMs are scanned in the order described from larger block sizes to smaller block sizes (eg matrixId from 0 to 30). In step 1210, the encoder searches the prediction preferences to determine if the current QM is a copy (or decimation) of what was previously coded. The QMs can be designed in a way that optimizes the efficiency of QM prediction, eg, if some QMs are initially close enough, or close from the default QMs, they can be forced to be the same (or if the sizes are different, they can be can be mated). Also, the coefficients in the lower right quadrant of size-64 QM can be optimized for better prediction of subsequent QMs, or to reduce QM bit cost if never reused for prediction. Once determined, the QM prediction mode is encoded into the bitstream.

In particular, if the encoder decides to use prediction ( 1220 ), then at step 1230 , the prediction mode is encoded (eg, scaling_list_pred_mode_flag=0). In step 1240, the prediction parameters (eg, QM index difference scaling_list_pred_matrix_id_delta) are encoded: zero index difference to default QM values, or the relevant index difference if a previous QM was selected as the prediction reference. On the other hand, if explicit signaling is determined, in step 1250, a prediction mode is encoded (eg, scaling_list_pred_mode_flag=0). A diagonal scan 1260 is then performed followed by QM coefficient encoding 1270 .

In step 1280, the encoder checks whether the current QM is the last QM to be encoded. If not, control returns to step 1210; Otherwise, the QM encoding process is aborted when all QMs are encoded into the bitstream.

13 illustrates a process 1300 for QM derivation, according to one embodiment. The input may include a ScalingMatrix array, and transform block parameters such as size (width/height), prediction mode (intra/inter/IBC,...), and color component (Y/U/V). The output is a QM with the same size as the transform block. Details regarding the DC value are omitted for clarity. Specifically, in step 1310, the decoder determines the current transform block size (width/height), prediction mode (intra/inter/IBC,...), color components (Y/U/V) as described above (integrated QM index) ) to determine the QM index matrixId. In step 1320, the decoder resizes the selected QM (ScalingMatrix[matrixId]) to match the transform block size, as described above. In a variant, step 1320 may include decimation necessary for prediction.

The QM derivation process is similar on the encoder side. Quantization divides transform coefficients by QM values, whereas inverse quantization multiplies. However, the QM is the same. In particular, the QMs used for reconstruction at the encoder will match those signaled in the bitstream.

Conceptually, the transform coefficients d[　x　][　y　] can be quantized as follows, where qStep is the quantization step size and m[][] is the quantization matrix:

TransCoeffLevel[xTbY][yTbY][cIdx][x][y] = d[x][y] / qStep / m[x][y]

However, for integer arithmetic and to avoid division, it is usually as follows:

TransCoeffLevel[xTbY][yTbY][cIdx][x][y] = ( ( d[x][y] * im[x][y] * ilevelScale[ qP%6 ] >> (qP / 6 ) ) + ( 1 << ( bdShift - 1 ) ) ) >> bdShift )

Here, for example, im[x][y]~= 65536/ m[x][y], ilevelScale[0..5]= 65536/ levelScale[0..5], and bdShift is an appropriate value . In practice, in the case of the software coder im*ilevelScale, it is usually pre-computed and stored in tables.

In the above, the QM prediction process and the QM derivation process are performed separately. In other embodiments, QM prediction may be deferred to the QM derivation process. This embodiment does not change the QM signaling syntax. This embodiment, which may be functionally different due to successive resizing, defers the prediction part (reference QM acquisition + copy/downscale) and diagonal scan to the “QM derivation process” described below.

In one embodiment, then, the output of the scaling list data parsing process 1100 is an array of ScalingList() instead of ScalingMatrix, with prediction flags and valid prediction parameters: index of the defined ScalingList (default or signaled) An array of ScalingMatrixPredId that always contains . This array is built recursively during QM decoding by interpreting scaling_list_pred_matrix_id_delta, so the QM derivation process can use this index directly to get the actual values for building the QM used to inverse quantize the current transform block.

Hereinafter, an example is provided to illustrate scaling list semantics, according to an embodiment.

Scaling Matrix Derivation Process (New: Partially superseding the description in Scaling List Semantics; this is section xxx)

Inputs to this process are the prediction mode predMode, the color component variable cIdx, the block width blkWidth and the block height blkHeight.

The output of this process is the (blkWidth)x(blkHeight) array m[　x　][　y　] (scaling matrix), where x and y are the horizontal and vertical coefficient positions. Note that SubWidthC and SubHeightC depend on the chroma format and indicate the ratio of the number of samples in the luma component and the chroma component.

The variable matrixId is derived as follows:

matrixId = 6 * sizeId + matrixTypeId (xxx-1)

where subWidth = (cIdx > 0) ? SubWidthC: 1,

subHeight = (cIdx > 0) ? SubHeightC: 1,

sizeId = 6 - max( log2( blkWidth * subWidth ), log2( blkHeight * subHeight ) ), and

matrixTypeId = ( 2 * cIdx + ( predMode = = MODE_INTER ? 1: 0 ) )

The variable log2MatrixSize is derived as follows:

log2MatrixSize = (matrixId < 20) ? 3: (matrixId < 26) ? 2:1 (xxx-2)

The output array m[　x　][　y　] is derived by applying the following for x in the range 0 to blkWidth-1 (inclusive) and y in the range 0 to blkHeight-1 (inclusive):

m[ x ][ y ] = ScalingMatrix[ matrixId ][ i ][ j ] (xxx-3)

where i = ( x << log2MatrixSize ) >> log2( blkWidth ), and

j = ( y << log2MatrixSize ) >> log2( blkHeight )

If matrixId is lower than 14, m[ 0 ][ 0 ] is further modified as follows:

m[ 0 ][ 0 ] = scaling_list_dc_coef_minus8[ matrixId ] + 8 (xxx-4)

Note that this is an example and is not restrictive, such as for the scaling_list_data syntax and semantics. For example, it is not limited to the two default QMs associated with MODE_INTRA and MODE_INTER, but with all 16 values until the relevant default values are agreed (for example, default QMs equal to HEVC can be selected) where is filled as There can be a single default QM, or more than two. For example, if there are more or less than 6 types of matrices per block size, the MatrixId calculation may be different. Importantly, the horizontal and vertical downscaling and upscaling to adapt to block sizes different from the selected QM is done in a single process, preferably simply (here left shift followed by right shift).

For rectangular blocks, it is not limited to selecting a QM identifier for the current block surrounding the square: the derivation of sizeId in equation (xxx-1) may follow different rules.

Also, the selected QM size log2MatrixSize can be expressed mathematically instead of a sequence of comparisons, with the same result: log2MatrixSize = min(3, 6-(matrixId+ 4) / 6), but this introduces division which may be unwelcome. do.

Hereinafter, according to an embodiment, an example for explaining the semantics of a scaling process is provided.

Scaling process for transform coefficients (adapted)

[...]

For the derivation of the scaled transform coefficients d[　x　][　y　] with x = 0..nTbW-1, y = 0..nTbH-1, the following applies:

The (nTbW)x(nTbH) intermediate scaling factor array m is derived as follows:

If one or more of the following conditions are true, then m[ x ][ y ] is set equal to 16:

scaling_list_enabled_flag is equal to 0.

transform_skip_flag[　xTbY　][　yTbY　] is equal to 1.

Otherwise, m is the output of the scaling matrix derivation process as specified in clause xxx, called as inputs with the prediction mode CuPredMode[xTbY][yTbY], the color component variable cIdx, the block width nTbW and the block height nTbH.

The scaling factor ls[　x　][　y　] is derived as:

[...]

The major change compared to VVC Draft 5 is the invocation of clause xxx instead of copying part of the array described in scaling_list_data semantics.

As noted above, this is an example intended to minimize changes to the current VVC draft, and is not limiting. For example, the color component input to the scaling matrix derivation process may be different from cIdx. Also, QM is not limited to being used as a scaling factor, and may be used, for example, as a QP-offset.

QM coding performance was tested using the same test set used during HEVC standardization, and QM derived from recommended or default QMs of common standards (JPEG, MPEG2, AVC, HEVC) and QMs found in actual broadcast. reinforced with sets. In all tests, some QMs are copied from one type to another (eg, luma to chroma, or intra to inter), and/or from one size to another.

The table below reports the number of bits required to encode scaling_list_data using three different methods: HEVC, JVET-N0847 and proposed. In particular, HEVC uses the HEVC test set (24 QMs per test), and two others use the derived test set (30 QMs per test, with the following additional sizes: size-2 for chroma and size-64 for luma; size-2 QMs are downsampled from size-4, size-64 copied from size-32, size-32 copied from size-16, sizes 16, 8, 4 are as is maintain).

Table 8

For this test, it can be seen that the proposed method saves a significant amount of bits even compared to HEVC, whereas the proposed method encodes more QMs.

Referring back to the approach in JCTVC-E073, the reference indexing in JCTVC-E073 (triplet: QMPS, size, type) is more complex than that proposed here because the previous QMPS indexing requires the storage of the previous QMPS. Linear interpolation introduces complexity. Downsampling is similar to that proposed here.

Referring back to the approach in JCTVC-H0314, the transmission from larger to smaller is close to the one proposed here, but the fixed prediction tree in JCTVC-H0314 is less flexible than the unified indexing and explicit reference proposed here.

QM for intra block copy mode

In the above, different QMs are specified for the two block prediction modes, intra and inter. However, in addition to intra and inter there is a new prediction mode in VVC: Intra Block Copy (IBC), where a block can be predicted from reconstructed samples of the same picture with an appropriate displacement vector. For QM selection in IBC prediction mode, both JVET-N0847 and the above embodiments use the same QMs as intra mode.

Since IBC mode is closer to inter than intra, in one embodiment, it is proposed to reuse signaled QMs for inter mode (instead of intra). However, IBC, while close to inter prediction, is different: a displacement vector does not match object or camera motion, but is used for texture copying. This can lead to certain artifacts, and certain QMs can help optimize the coding of IBC blocks in a different embodiment. In the following, we propose to change the QM selection for IBC prediction mode:

· A preferred embodiment is to select the same QM as inter mode (instead of intra), since IBC is closer to inter prediction than intra prediction.

· Another option is to have specific QMs for IBC mode.

o These may be explicitly signaled or inferred in the syntax (eg, average of intra and inter QMs).

In a preferred embodiment, the QM selection or derivation process for a particular transform block selects inter QMs if the block has an IBC prediction mode. Referring back to Figure 12, step 1210 of the QM derivation process needs to be coordinated as described below.

In the draft text proposed above, the QM selection is described in Equation (xxx-1), which can be changed as follows.

Note that blocks can be encoded in intra mode (MODE_INTRA), inter mode (MODE_inter) or intra block copy mode (MODE_IBC). When matrixTypeId is set as in Table 6 or as (matrixTypeId= (2* cIdx+ (predMode== MODE_INTER? 1: 0) ) ), the MODE_IBC block selects the matrixTypeId as if it were a MODE_INTRA block. With a change in (xxx-1): if matrixTypeId = ( 2 * cIdx + ( predMode = = MODE_INTRA ? 0 : 1 ) ), the MODE_IBC block selects the matrixTypeId as if it were a MODE_INTER block.

In the draft text proposed by JVET-N0847, the QM selection is described in Tables 7 - 14, which can be changed as follows. In particular, matrixId for MODE_IBC is assigned in the same manner as MODE_INTER, not MODE_INTRA as in JVET-N0847.

Table 9: Changes to JVET-N0847 specification of matrixId according to sizeId, prediction mode and color component

Variant 1: Explicit signaling of QMs for IBC

In this variant, specific QMs are used for IBC blocks (different from intra and inter QMs), and these QMs are explicitly signaled in the bitstream. This makes more QMs, which requires adaptation of the scaling_list_data syntax and matrixId mapping, and has a bit cost impact. According to this variant, the QM selection described in equation (xxx-1) can be changed as follows.

In JVET-N0847, the QM selection table may be changed as follows:

Table 10: Changes to JVET-N0847 adding more QMs for IBC

Variant 2: Inferred QMs for IBC mode

In this variant, specific QMs are used for IBC blocks (different from intra and inter QMs). However, these QMs are not signaled in the bitstream, but are inferred: for example, the average of intra and inter QMs, or certain default values, or certain changes to the inter QM such as scaling and offset.

Variant 3: Explicit IBC QMs for Lumaman

In this variant, additional QMs for IBC are limited to luma only; Chroma QMs for IBC may reuse inter QMs as in variant 1 or infer new ones as in variant 2.

Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of a method, the order and/or use of specific steps and/or actions may be modified or combined. Also, terms such as “first”, “second”, etc. may be used to change an element, component, step, operation, etc. in various embodiments, for example, “first decoding” and “second decoding” "am. Use of these terms does not imply an order for modified operations unless specifically required. Thus, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or within the overlapping period with the second decoding.

Various methods and other aspects described in this application can be applied to modules of video encoder 200 and decoder 300 as shown in FIGS. 2 and 3 , eg, quantization and inverse quantization modules 230 , 240 . , 340) can be used to change Further, the present aspects are not limited to VVC or HEVC, but may apply to, for example, other standards and recommendations, and extensions of any such standards and recommendations. Unless otherwise indicated or technically excluded, aspects described in this application may be used individually or in combination.

Various numerical values are used in this application. The specific values are for illustrative purposes and the described aspects are not limited to these specific values.

Various implementations involve decoding. “Decoding,” as used herein, may include all or part of processes performed, for example, on a received encoded sequence to produce a final output suitable for display. In various embodiments, these processes include processes typically performed by a decoder, eg, one or more of entropy decoding, inverse quantization, inverse transform, and differential decoding. Whether the phrase “decoding process” is specifically intended to refer to a subset of operations or to a broader decoding process in general will be apparent based on the context of the particular descriptions, and is within the skill of the person skilled in the art. is expected to be well understood by

Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding", "encoding" as used herein may include all or part of the processes performed, for example, on an input video sequence, to produce an encoded bitstream. there is.

Note that syntax elements used herein are descriptive terms. As such, they do not preclude the use of other syntax element names. In the above, syntax elements for PPS and scaling list are mainly used to illustrate various embodiments. It should be noted that these syntax elements may be placed in other syntax structures.

Implementations and aspects described herein may be implemented as, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Although discussed only in the context of a single form of implementation (eg, discussed only as a method), implementation of the features discussed may also be implemented in other forms (eg, as an apparatus or program). The apparatus may be implemented in, for example, suitable hardware, software, and firmware. The methods may be implemented in, for example, an apparatus, such as a processor, which refers generally to processing devices, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as computers, cell phones, portable/personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end users.

Reference to “one embodiment” or “an embodiment” or “an implementation” or “implementation”, as well as other variations thereof, refers to a particular feature, structure, characteristic, etc. described in connection with the embodiment is at least one implementation. means included in the example. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” or “in an implementation” or “in an implementation” as well as any other variations appearing in various places throughout this application are necessarily all occurrences of the same embodiment. does not refer to

This application may also refer to "determining" various pieces of information. Determining the information may include, for example, one or more of estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

This application may also refer to “accessing” various information. Accessing information includes, for example, receiving information, retrieving information (eg, from memory), storing information, moving information, copying information, It may include one or more of calculating, determining information, predicting information, or estimating information.

This application may also refer to "receiving" various pieces of information. Receiving, as in “accessing,” is intended to be a broad term. Receiving the information may include, for example, one or more of accessing the information, or retrieving the information (eg, from memory). Also, "receiving" typically includes, in one way or another, during operations, such as storing information, processing information, transmitting information, moving information. , is involved during copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

For example, in the cases of "A/B", "A and/or B" and "at least one of A and B," "/", "and/or", and "at least one of" It is understood that the use of any of these is intended to include the selection of only the first listed option (A), or only the second listed option (B), or the selection of both options (A and B). should be As a further example, in the instances of "A, B, and/or C" and "at least one of A, B, and C," such phrases refer to selection of only the first listed option (A), or the second listed option (A). Selection of only option (B), or selection of only the third listed option (C), or selection of only the first and second listed options (A and B), or selection of the first and third listed options (A and It is intended to include selection of only C), or selection of only the second and third listed options (B and C), or selection of all three options (A and B and C). This can be extended for many of the items listed, as will be apparent to those skilled in the art and related art.

Also, as used herein, the word “signaling” specifically refers to indicating something to a corresponding decoder. For example, in certain embodiments, the encoder signals a quantization matrix for inverse quantization. In this way, in one embodiment, the same parameters are used on both the encoder side and the decoder side. Thus, for example, the encoder may transmit (explicitly signal) certain parameters to the decoder so that the decoder can use the same particular parameters. Conversely, if the decoder already has certain parameters as well as others, then signaling can be used without transmission (implicit signaling) simply to allow the decoder to know and select certain parameters. By avoiding the transmission of any actual functions, bit savings are realized in various embodiments. It will be appreciated that signaling may be accomplished in a variety of ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. Although the foregoing relates to the verb form of the word "signal", the word "signal" may also be used herein as a noun.

As will be apparent to one of ordinary skill in the art, implementations may generate various signals formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry the bitstream of the described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (eg, using a radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. A signal may be transmitted over a variety of different wired or wireless links as is known. The signal may be stored on a processor-readable medium.

Claims (29)

As a method,
obtaining a single identifier for a quantization matrix based on a block size, a color component, and a prediction mode of a block to be decoded within a picture;
decoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix;
obtaining the quantization matrix based on the reference quantization matrix;
dequantizing transform coefficients for the block in response to the quantization matrix; and
and decoding the block in response to the inverse quantized transform coefficients. A device comprising one or more processors, comprising:
The one or more processors,
obtain a single identifier for a quantization matrix based on a block size, a color component, and a prediction mode of a block to be decoded in the picture;
decode a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix;
obtain the quantization matrix based on the reference quantization matrix;
inverse quantize transform coefficients for the block in response to the quantization matrix; And
and decode the block in response to the inverse quantized transform coefficients. The method of claim 1 or the device of claim 2,
wherein the size of the block is different from the size of the block to which the reference quantization matrix is applied for inverse quantization. 4. The method of claim 1 or 3 or the device of claim 2 or 3,
The elements of the quantization matrix are used as scaling factors when inverse quantizing respective transform coefficients of the block. 5. A method according to any one of claims 1 and 2 to 4 or a device according to any one of claims 2 to 4,
The elements of the quantization matrix are used as offsets when inverse quantizing respective transform coefficients of the block. As a method,
accessing a block to be encoded in the picture;
accessing a quantization matrix for the block;
obtaining a single identifier for the quantization matrix based on a block size, a color component, and a prediction mode of the block;
encoding a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix;
quantizing transform coefficients for the block in response to the quantization matrix; and
and entropy encoding the quantized transform coefficients. A device comprising one or more processors, comprising:
The one or more processors,
access a block to be encoded in the picture;
access a quantization matrix for the block;
obtain a single identifier for the quantization matrix based on a block size, a color component, and a prediction mode of the block;
encode a syntax element indicating a reference quantization matrix, wherein the syntax element specifies a difference between an identifier of the reference quantization matrix and the obtained identifier for the quantization matrix;
quantize transform coefficients for the block in response to the quantization matrix; And
and entropy encode the quantized transform coefficients. The method of claim 6 or the device of claim 7,
wherein the size of the block is different from the size of the block to which the reference quantization matrix is to be applied for quantization. The method of claim 6 or 8 or the device of claim 7 or 8,
The elements of the quantization matrix are used as scaling factors when quantizing respective transform coefficients of the block. The method of claim 6 or 8 or the device of claim 7 or 8,
The elements of the quantization matrix are used as offsets when quantizing respective transform coefficients of the block. 11. A method according to any one of claims 1, 3 to 6, and 8 to 10 or an apparatus according to any one of claims 2 to 5 and 7 to 10, comprising:
wherein the reference quantization matrix is previously signaled. 12. The method of any one of claims 1, 3-6, and 8-11 or the device of any one of claims 2-5 and 7-11,
The quantization matrix is obtained from the reference quantization matrix through copying or decimation. 13. The method of claim 12 or the device of claim 12, comprising:
wherein the quantization matrix is obtained from the reference quantization matrix via copy in response to the quantization matrix having the same size as the reference quantization matrix. 13. The method of claim 12 or the device of claim 12, comprising:
wherein the quantization matrix is obtained from the reference quantization matrix through decimation by a corresponding ratio in response to the quantization matrix having a different size than the reference quantization matrix. 15. The method of any one of claims 1, 3-6, and 8-14 or the device of any one of claims 2-5 and 7-14, comprising:
wherein the block size is MxN, and M is the width and N is the height, wherein the identifier for the block is based on the size of max(M,N), where max(M,N) is the greater of M and N. As defined, a method or apparatus. 16. The method of any one of claims 1, 3-6, and 8-15 or the device of any one of claims 2-5 and 7-15,
A method or apparatus, wherein the set of quantization matrices is signaled in order of increasing identifier, and the quantization matrix of the largest block size is signaled first. 17. The method of claim 16 or the device of claim 16,
Quantization matrices for a luma color component are signaled before quantization matrices for a chroma color component when signaling the set of quantization matrices. 18. The method of any one of claims 1, 3-6, and 8-17 or the device of any one of claims 2-5 and 7-17, wherein
Quantization matrices for larger block sizes are signaled before quantization matrices for smaller block sizes when signaling the set of quantization matrices. 19. The method of any one of claims 1, 3-6, and 8-18 or the device of any one of claims 2-5 and 7-18,
The identifier is derived as matrixId = N * sizeId + matrixTypeId, where N is the number of possible type identifiers, sizeID indicates the block size, and matrixTypeId indicates the color component and the prediction mode. . 20. The method of any one of claims 1, 3 to 6, and 8 to 19, wherein
The method further comprising the one or more processors being further configured to perform adapting the reference quantization matrix to the block size, or
20. The device of any one of claims 2-5 and 7-19, comprising:
and the one or more processors are further configured to perform adapting the reference quantization matrix to the block size. 21. The method of any one of claims 1, 3 to 6, and 8 to 20, wherein
The method further comprising: for a chroma format of the block that is different from a default chroma format, the one or more processors are further configured to perform adapting the reference quantization matrix to the chroma format of the block, or
The device of any one of claims 2 to 5 and 7 to 20, comprising:
and the one or more processors are further configured to: for a chroma format of the block that is different from a default chroma format, adapt the reference quantization matrix to the chroma format of the block. 21. The method of claim 20 or the device of claim 20, comprising:
wherein the default chroma format is 4:2:0. 23. The method of any one of claims 20-22 or the device of any one of claims 20-22,
wherein the adaptation is based on bit shifts of x and y coordinates in the quantization matrix to index the coefficients of the reference quantization matrix. 24. The method of any one of claims 1, 3-6, and 8-23 or the device of any one of claims 2-5 and 7-23, comprising:
The method or apparatus, wherein the identifier is obtained based on whether the prediction mode of the block is an intra prediction mode or an inter prediction mode. 25. The method of any one of claims 1, 3-6, and 8-24 or the device of any one of claims 2-5 and 7-24, comprising:
An intra block copy prediction mode is considered as an inter prediction mode when obtaining the identifier. 26. The method of any one of claims 1, 3-6, and 8-25 or the device of any one of claims 2-5 and 7-25, comprising:
wherein the prediction mode is an intra block copy, a quantization matrix is signaled for a luma component of the block, and a quantization matrix is derived for a chroma component by considering the prediction mode as an inter prediction mode. 27. The method of any one of claims 1, 3-6, and 8-26 or the device of any one of claims 2-5 and 7-26, comprising:
the prediction mode is an intra block copy, the reference quantization matrix is obtained by considering the prediction mode as an intra mode, another reference quantization matrix is obtained by considering the prediction mode as an inter mode, and the quantization matrix is the reference quantization matrix A method or apparatus, obtained as an average of a matrix and the other reference quantization matrix. A signal comprising encoded video formed by performing the method of any of claims 6 and 8 to 27 . A computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to the method of any one of claims 1, 3 to 6 and 8 to 27.

Images