JP7492066B2 - Image or video coding based on scaling list data - Google Patents

Description

This technology relates to video or image coding, for example coding techniques based on scaling list data.

In recent years, the demand for high-resolution, high-quality images/videos, such as 4K or 8K or higher UHD (Ultra High Definition) images/videos, has been increasing in various fields. As the image/video data has higher resolution and quality, the amount of information or bits transmitted increases relatively compared to existing image/video data. Therefore, when image data is transmitted using a medium such as an existing wired or wireless broadband line, or when image/video data is stored using an existing storage medium, the transmission and storage costs increase.

In addition, in recent years, interest in and demand for immersive media such as virtual reality (VR) and artificial reality (AR) content and holograms has increased, and the broadcast of images/videos with different image characteristics from real images, such as game images, has increased.

This calls for highly efficient image/video compression technologies to effectively compress and transmit, store, and play back high-resolution, high-quality image/video information with the various characteristics described above.

There is also discussion of adaptive frequency weighting quantization techniques during the scaling process to improve compression efficiency and enhance subjective/objective visual quality. A method for signaling relevant information to efficiently apply such techniques is needed.

The technical problem of this document is to provide a method and apparatus for improving video/image coding efficiency.

Another technical problem of this document is to provide a method and apparatus for improving coding efficiency during the scaling process.

Another technical problem of this document is to provide a method and apparatus for efficiently constructing a scaling list used in the scaling process.

Another technical problem of this document is to provide a method and apparatus for hierarchically signaling scaling list related information used in the scaling process.

Another technical problem of this document is to provide a method and apparatus for efficiently applying a scaling list-based scaling process.

According to one embodiment of this document, scaling list data may be signaled via an adaptation parameter set (APS). In addition, the APS may be identified based on APS ID information included in the APS, and the APS may include scaling list data based on APS type information indicating that the APS is an APS related to the scaling list data included in the APS. In addition, for the APS type information indicating that the APS is an APS related to the scaling list data, the value of the APS ID information may have a value within a specific range.

According to one embodiment of this document, availability flag information indicating the availability of scaling list data may be signaled hierarchically, and availability flag information in a lower level syntax (e.g., picture header/slice header/type group header, etc.) may be signaled based on availability flag information signaled in a higher level syntax (e.g., SPS).

According to one embodiment of this document, the SPS syntax and PPS syntax can be configured to not signal scaling list data syntax directly at the SPS or PPS level. This can improve coding efficiency by parsing/signaling scaling list data individually in the APS based on availability flag information in the SPS.

According to one embodiment of the present document, there is provided a video/image decoding method performed by a decoding device. The video/image decoding method may include the methods disclosed in the embodiments of the present document.

According to one embodiment of the present document, there is provided a decoding device for performing video/image decoding. The decoding device is capable of performing the method disclosed in the embodiment of the present document.

According to one embodiment of the present document, there is provided a video/image encoding method performed by an encoding device. The video/image encoding method may include the methods disclosed in the embodiments of the present document.

According to one embodiment of the present document, there is provided an encoding device for performing video/image encoding. The encoding device is capable of performing the method disclosed in the embodiment of the present document.

According to one embodiment of the present document, there is provided a computer-readable digital storage medium having encoded video/image information generated by the video/image encoding method disclosed in at least one of the embodiments of the present document.

According to one embodiment of the present document, there is provided a computer-readable digital storage medium having encoded information or encoded video/image information stored thereon that causes a decoding device to perform the video/image decoding method disclosed in at least one of the embodiments of the present document.

This document may have various advantages. For example, according to an embodiment of this document, the overall image/video compression efficiency may be increased. According to an embodiment of this document, the coding efficiency may be increased by applying an efficient scaling process, and the subjective/objective visual quality may be improved. According to an embodiment of this document, the scaling list used in the scaling process may be efficiently configured, and the scaling list-related information may be hierarchically signaled therethrough. According to an embodiment of this document, the coding efficiency may be increased by efficiently applying a scaling process based on the scaling list. According to an embodiment of this document, the effect of facilitating implementation and limiting the worst case memory requirement may be obtained by placing restrictions on the scaling list matrix used in the scaling process.

The effects that can be obtained through the specific embodiments of this document are not limited to the effects listed above. For example, there may be various technical effects that a person having ordinary skill in the related art can understand or derive from this document. Thus, the specific effects of this document are not limited to those explicitly described in this document, but may include various effects that can be understood or derived from the technical features of this document.

1 illustrates generally an example of a video/image coding system in which embodiments of the present document may be applied; FIG. 1 is a diagram illustrating a schematic configuration of a video/image encoding device to which embodiments of the present document can be applied. FIG. 1 is a diagram illustrating the configuration of a video/image decoding device to which the embodiments of the present document can be applied. 1 shows an example of a general video/image encoding method to which embodiments of this document can be applied. 1 shows an example of a schematic video/image decoding method to which embodiments of the present document can be applied. 1 shows an exemplary hierarchical structure for coded images/videos. 1 illustrates generally an example of a video/image encoding method and associated components according to embodiments (etc.) of the present document; 1 illustrates generally an example of a video/image encoding method and associated components according to embodiments (etc.) of the present document; 1 illustrates generally an example of a video/image decoding method and associated components according to embodiments (etc.) of the present document; 1 illustrates generally an example of a video/image decoding method and associated components according to embodiments (etc.) of the present document; 1 illustrates an example of a content streaming system in which the embodiments disclosed herein may be applied.

The present disclosure may be modified in various ways and may have various embodiments, and a specific embodiment will be illustrated in the drawings and described in detail. However, this is not intended to limit the present disclosure to a specific embodiment. Terms commonly used in this document are used merely to describe a specific embodiment and are not intended to limit the technical ideas of the present disclosure. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "include" or "have" are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Meanwhile, each configuration in the drawings described in this disclosure is illustrated independently for the convenience of explaining the different characteristic functions, and does not mean that each configuration is realized by separate hardware or software. For example, two or more of the configurations may be combined to form one configuration, and one configuration may be divided into multiple configurations. Embodiments in which each configuration is integrated and/or separated are also included in the scope of the present disclosure as long as they do not deviate from the essence of this document.

In this document, "A or B" can mean "only A", "only B", or "both A and B". Also, in this document, "A or B" can be interpreted as "A and/or B". For example, in this document, "A, B or C" can mean "only A", "only B", "only C", or "any combination of A, B and C".

The slash (/) and comma used in this document can mean "and/or." For example, "A/B" can mean "A and/or B." Thus, "A/B" can mean "only A," "only B," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

In this document, "at least one of A and B" can mean "only A", "only B", or "both A and B". Also, in this document, the expressions "at least one of A or B" and "at least one of A and/or B" can be interpreted as "at least one of A and B".

In this document, "at least one of A, B and C" can mean "only A", "only B", "only C", or "any combination of A, B and C". Also, "at least one of A, B or C" and "at least one of A, B and/or C" can mean "at least one of A, B and C".

Furthermore, parentheses used in this document may mean "for example." Specifically, when "prediction (intra prediction)" is displayed, "intra prediction" is proposed as an example of "prediction." In other words, "prediction" in this document is not limited to "intra prediction," and "intra prediction" is proposed as an example of "prediction." Also, when "prediction (i.e., intra prediction)" is displayed, "intra prediction" is proposed as an example of "prediction."

This document relates to video/image coding. For example, the methods/embodiments disclosed in this document may be applied to methods disclosed in the Versatile Video Coding (VVC) standard. Also, the methods/embodiments disclosed in this document may be applied to methods disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the 2nd generation of audio video coding standard (AVS2) or the next generation video/image coding standard (e.g., H.267 or H.267).

This document presents various embodiments relating to video/image coding, which may be implemented in combination with one another unless otherwise stated.

In this document, video may refer to a collection of a series of images over time. A picture generally refers to a unit that indicates one image at a specific time, and a slice/tile is a unit that constitutes a part of a picture in coding. A slice/tile may include one or more coding tree units (CTUs). A picture may be composed of one or more slices/tiles. A tile is a rectangular region of a particular tile column and a particular tile row in a picture. The tile column is a rectangular region of CTUs having a height equal to the height of the picture and a width specified by syntax elements in the picture parameter set. The tile row is a rectangular region of CTUs having a height specified by syntax elements in the picture parameter set and a width equal to the width of the picture. A tile scan can indicate a specific sequential ordering of CTUs partitioning a picture, where the CTUs are ordered consecutively in a raster scan of the CTUs in a tile, and the tiles in a picture are ordered consecutively in a raster scan of the tiles of the picture. A slice includes an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile of a picture that may be exclusively contained in a single NAL unit.

On the other hand, a picture can be divided into two or more subpictures. A subpicture is a rectangular region of one or more slices within a picture.

A pixel or pel can refer to the smallest unit that constitutes one picture (or image). A term corresponding to a pixel can also be used as a "sample." A sample can generally refer to a pixel or a pixel value, can refer to only a pixel/pixel value of a luma component, or can refer to only a pixel/pixel value of a chroma component. Alternatively, a sample can refer to a pixel value in the spatial domain, or, when such a pixel value is transformed into the frequency domain, can refer to a transform coefficient in the frequency domain.

A unit may refer to a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to the region. A unit may include one luma block and two chroma (e.g., cb, cr) blocks. A unit may be used interchangeably with terms such as block or area, depending on the case. In general, an M×N block may include a sample (or sample array) or a set (or array) of transform coefficients consisting of M columns and N rows.

Also, in this document, at least one of quantization/dequantization and/or transform/inverse transform may be omitted. If quantization/dequantization is omitted, the quantized transform coefficients may be referred to as transform coefficients. If transform/inverse transform is omitted, the transform coefficients may also be referred to as coefficients or residual coefficients, or may still be referred to as transform coefficients for uniformity of representation.

In this document, the quantized transform coefficients and the transform coefficients may be referred to as transform coefficients and scaled transform coefficients, respectively. In this case, the residual information may include information about the transform coefficient(s), and the information about the transform coefficient(s) may be signaled via a residual coding syntax. Based on the residual information (or information about the transform coefficient(s), transform coefficients may be derived, and scaled transform coefficients may be derived via an inverse transform (scaling) on the transform coefficients. Residual samples may be derived based on an inverse transform (transform) on the scaled transform coefficients. This may be similarly applied/expressed in other parts of this document.

In this document, technical features described individually in one drawing may be realized individually or simultaneously.

The preferred embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. Hereinafter, the same reference numerals will be used for the same components in the drawings, and duplicated descriptions of the same components may be omitted.

Figure 1 illustrates a schematic diagram of an example video/image coding system that can be applied to embodiments of this document.

Referring to FIG. 1, a video/image coding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video/image information or data in a file or streaming form to the receiving device via a digital storage medium or a network.

The source device may include a video source, an encoding device, and a sending unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be referred to as a video/image encoding device, and the decoding device may be referred to as a video/image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may also include a display unit, which may be a separate device or an external component.

A video source can acquire video/images through a video/image capture, synthesis, or generation process, etc. A video source can include a video/image capture device and/or a video/image generation device. A video/image capture device can include, for example, one or more cameras, a video/image archive containing previously captured video/images, etc. A video/image generation device can include, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate video/images. For example, a virtual video/image can be generated through a computer, etc., in which case the video/image capture process can be replaced by a process in which the associated data is generated.

An encoding device can encode an input video/image. The encoding device can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video/image information) can be output in the form of a bitstream.

The transmitting unit can transmit the encoded video/image information or data output in the form of a bitstream to a receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitting unit can include elements for generating a media file via a predetermined file format and can include elements for transmission via a broadcasting/communication network. The receiving unit can receive/extract the bitstream and transmit it to a decoding device.

The decoding device can decode the video/image by performing a series of steps such as inverse quantization, inverse transformation, and prediction that correspond to the operations of the encoding device.

The renderer can render the decoded video/image. The rendered video/image can be displayed via the display unit.

Figure 2 is a diagram that illustrates the configuration of a video/image encoding device to which the embodiments of this document can be applied. Hereinafter, the encoding device can include an image encoding device and/or a video encoding device.

Referring to FIG. 2, the encoding device 200 may include an image partitioner 210, a prediction unit 220, a residual processor 230, an entropy encoder 240, an adder 250, a filtering unit 260, and a memory 270. The prediction unit 220 may include an inter prediction unit 221 and an intra prediction unit 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processing unit 230 may further include a subtractor 231. The adder 250 may be called a reconstruction unit or a reconstruction block generator. The image division unit 210, the prediction unit 220, the residual processing unit 230, the entropy encoding unit 240, the addition unit 250, and the filtering unit 260 may be configured by one or more hardware components (e.g., an encoder chip set or a processor) according to an embodiment. In addition, the memory 270 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware components may further include the memory 270 as an internal/external component.

The image division unit 210 may divide an input image (or picture, frame) input to the encoding device 200 into one or more processing units. As an example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) according to a QTBTTT (quad-tree binary-tree ternary-tree) structure. For example, one coding unit may be divided into multiple coding units of deeper depths based on a quad-tree structure, a binary tree structure, and/or a ternary structure. In this case, for example, a quad tree structure may be applied first, and a binary tree structure and/or a ternary structure may be applied thereafter. Alternatively, the binary tree structure may be applied first. The coding procedure according to this document may be performed based on a final coding unit that is not further divided. In this case, based on coding efficiency according to image characteristics, the largest coding unit may be used as the final coding unit, or the coding unit may be recursively divided into coding units of lower depths as necessary, and a coding unit of an optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may each be divided or partitioned from the final coding unit described above. The prediction unit is a unit of sample prediction, and the transform unit is a unit for deriving transform coefficients and/or a unit for deriving a residual signal from the transform coefficients.

The unit may be used interchangeably with terms such as block or area. In the general case, an M×N block may refer to a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally refer to a pixel or pixel value, or may refer to only a pixel/pixel value of a luma component, or may refer to only a pixel/pixel value of a chroma component. A sample may be used as a term corresponding to a pixel or pel of a picture (or image).

The encoding device 200 may generate a residual signal (residual block, residual sample array) by subtracting a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from an input image signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, as shown in the figure, a unit in the encoder 200 that subtracts a prediction signal (predicted block, prediction sample array) from an input image signal (original block, original sample array) may be called a subtraction unit 231. The prediction unit may perform prediction on a block to be processed (hereinafter, referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied on a current block or CU basis. The prediction unit may generate various information related to prediction, such as prediction mode information, as described below in the description of each prediction mode, and transmit the information to the entropy encoding unit 240. The prediction information may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

The intra prediction unit 222 may predict the current block by referring to samples in the current picture. The referenced samples may be located adjacent to the current block or may be located away from the current block depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes may include, for example, a DC mode and a planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the fineness of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the settings. The intra prediction unit 222 may also determine a prediction mode to be applied to the current block using a prediction mode applied to a neighboring block.

The inter prediction unit 221 may derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on the reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring blocks may include a spatial neighboring block present in the current picture and a temporal neighboring block present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. The temporally neighboring blocks may be referred to as collocated reference blocks, collocated CUs, etc., and a reference picture including the temporally neighboring blocks may be referred to as a collocated picture (colPic). For example, the inter prediction unit 221 may generate information indicating which candidate is used to derive a motion vector and/or a reference picture index of the current block by forming a motion information candidate list based on the neighboring blocks. Inter prediction may be performed based on various prediction modes, and for example, in the case of a skip mode and a merge mode, the inter prediction unit 221 may use motion information of neighboring blocks as motion information of the current block. In the case of the skip mode, unlike the merge mode, a residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the neighboring block can be used as a motion vector predictor, and the motion vector of the current block can be indicated by signaling the motion vector difference.

The prediction unit 220 may generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for prediction of one block, and may simultaneously apply intra prediction and inter prediction. This may be called combined inter and intra prediction (CIIP). The prediction unit may also be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image/video coding such as games, for example, screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in deriving a reference block within the current picture. That is, IBC can utilize at least one of the inter prediction techniques described in this document. Palette mode can be considered as an example of intra coding or intra prediction. When palette mode is applied, sample values in a picture can be signaled based on information about a palette table and a palette index.

The prediction signal generated through the prediction unit (including the inter prediction unit 221 and/or the intra prediction unit 222) may be used to generate a restored signal or may be used to generate a residual signal. The transform unit 232 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally non-linear transform (CNT). Here, GBT refers to a transform obtained from a graph when the relationship information between pixels is expressed as a graph. CNT refers to a transformation that is obtained based on a prediction signal generated using all previously reconstructed pixels. The transformation process can be applied to pixel blocks having the same square size, or to non-square blocks of variable size.

The quantization unit 233 quantizes the transform coefficients and transmits them to the entropy encoding unit 240, and the entropy encoding unit 240 may encode the quantized signal (information about the quantized transform coefficients) and output it to a bitstream. The information about the quantized transform coefficients may be called residual information. The quantization unit 233 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and may generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form. The entropy encoding unit 240 may perform various encoding methods, such as, for example, exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). The entropy encoding unit 240 may also encode information required for video/image restoration (e.g., syntax element values, etc.) together with or separately from the quantized transform coefficients. The encoded information (e.g., encoded video/image information) may be transmitted or stored in the form of a bitstream in network abstraction layer (NAL) units. The video/image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/image information may further include general constraint information. In this document, information and/or syntax elements transmitted/signaled from an encoding device to a decoding device may be included in the video/image information. The video/image information may be encoded through the above-mentioned encoding procedure and included in the bitstream. The bitstream may be transmitted via a network or stored in a digital storage medium. Here, the network may include a broadcasting network and/or a communication network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitting unit (not shown) for transmitting the signal output from the entropy encoding unit 240 and/or a storing unit (not shown) for storing the signal may be configured as an internal/external element of the encoding device 200, or the transmitting unit may be included in the entropy encoding unit 240.

The quantized transform coefficients output from the quantizer 233 may be used to generate a prediction signal. For example, a residual signal (residual block or residual sample) may be restored by applying inverse quantization and inverse transform to the quantized transform coefficients via the inverse quantizer 234 and the inverse transformer 235. The adder 155 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit 221 or the intra prediction unit 222. When there is no residual for the current block, such as when a skip mode is applied, the predicted block may be used as the reconstructed block. The adder 250 may be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next current block in the current picture, and may also be used for inter prediction of the next picture after filtering, as described below.

Meanwhile, LMCS (luma mapping with chroma scaling) can also be applied during picture encoding and/or restoration.

The filtering unit 260 may apply filtering to the reconstructed signal to improve subjective/objective image quality. For example, the filtering unit 260 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit 260 may generate various information related to filtering and transmit it to the entropy encoding unit 240, as described below in the description of each filtering method. The filtering information may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

The modified reconstructed picture transmitted to the memory 270 can be used as a reference picture in the inter prediction unit 221. When inter prediction is applied through this, the encoding device can avoid prediction mismatch between the encoding device 100 and the decoding device, and can also improve encoding efficiency.

The DPB of the memory 270 may store the modified reconstructed picture for use as a reference picture in the inter prediction unit 221. The memory 270 may store motion information of a block from which motion information in the current picture is derived (or encoded) and/or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter prediction unit 221 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 270 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 222.

Figure 3 is a diagram that illustrates the configuration of a video/image decoding device to which the embodiments of this document can be applied. Hereinafter, a decoding device can include an image decoding device and/or a video decoding device.

Referring to FIG. 3, the decoding device 300 may be configured to include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filtering unit 350, and a memory 360. The predictor 330 may include an inter prediction unit 331 and an intra prediction unit 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. The entropy decoder 310, the residual processor 320, the predictor 330, the adder 340, and the filtering unit 350 may be configured as one hardware component (e.g., a decoder chipset or processor) according to an embodiment. The memory 360 may also include a decoded picture buffer (DPB) and may be configured with a digital storage medium. The hardware components may further include the memory 360 as an internal/external component.

When a bitstream including video/image information is input, the decoding device 300 can restore an image corresponding to the process in which the video/image information was processed in the encoding device of FIG. 2. For example, the decoding device 300 can derive a unit/block based on block division related information obtained from the bitstream. The decoding device 300 can perform decoding using a processing unit applied in the encoding device. Thus, the processing unit for decoding is, for example, a coding unit, and the coding unit can be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and/or a ternary tree structure. One or more transform units can be derived from the coding unit. Then, the restored image signal decoded and output via the decoding device 300 can be reproduced via a reproduction device.

The decoding device 300 may receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal may be decoded via the entropy decoding unit 310. For example, the entropy decoding unit 310 may derive information (e.g., video/image information) required for image restoration (or picture restoration) by parsing the bitstream. The video/image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/image information may further include general constraint information. The decoding device may decode a picture based on information on the parameter set and/or the general constraint information. Signaling/received information and/or syntax elements described later in this document may be decoded via the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 may decode information in a bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the value of a syntax element required for image restoration and the quantized value of a transform coefficient related to the residual. More specifically, the CABAC entropy decoding method receives a bin corresponding to each syntax element in the bitstream, determines a context model using the syntax element information to be decoded and the decode information of the adjacent and decoded blocks or the symbol/bin information decoded in the previous step, predicts the occurrence probability of the bin according to the determined context model, and performs arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method may update the context model using the decoded symbol/bin information for the context model of the next symbol/bin after determining the context model. Among the information decoded by the entropy decoding unit 310, information related to prediction is provided to a prediction unit (inter prediction unit 332 and intra prediction unit 331), and residual values entropy-decoded by the entropy decoding unit 310, i.e., quantized transform coefficients and related parameter information, may be input to the residual processing unit 320. The residual processing unit 320 may derive a residual signal (residual block, residual sample, residual sample array). Also, among the information decoded by the entropy decoding unit 310, information related to filtering may be provided to the filtering unit 350. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding device may be further configured as an internal/external element of the decoding device 300, or the receiving unit may be a component of the entropy decoding unit 310. Meanwhile, the decoding device according to this document may be called a video/image/picture decoding device, and the decoding device may be divided into an information decoder (video/image/picture information decoder) and a sample decoder (video/image/picture sample decoder). The information decoder may include the entropy decoding unit 310, and the sample decoder may include at least one of the inverse quantization unit 321, the inverse transform unit 322, the addition unit 340, the filtering unit 350, the memory 360, the inter prediction unit 332, and the intra prediction unit 331.

The inverse quantization unit 321 may inverse quantize the quantized transform coefficients to output the transform coefficients. The inverse quantization unit 321 may rearrange the quantized transform coefficients in a two-dimensional block form. In this case, the rearrangement may be performed based on a coefficient scan order performed in the encoding device. The inverse quantization unit 321 may perform inverse quantization on the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

The inverse transform unit 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual sample array).

The prediction unit may perform prediction on the current block and generate a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is applied to the current block based on information about the prediction output from the entropy decoding unit 310, and may determine a specific intra/inter prediction mode.

The prediction unit 320 may generate a prediction signal based on various prediction methods described below. For example, the prediction unit may apply intra prediction or inter prediction for prediction of one block, or may simultaneously apply intra prediction and inter prediction. This may be called combined inter and intra prediction (CIIP). The prediction unit may also be based on an intra block copy (IBC) prediction mode or a palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image/video coding such as games, for example, screen content coding (SCC). IBC basically performs prediction within a current picture, but may be performed similarly to inter prediction in that it derives a reference block within the current picture. That is, IBC may utilize at least one of the inter prediction techniques described in this document. Palette mode may be considered as an example of intra coding or intra prediction. When palette mode is applied, information regarding a palette table and a palette index may be included in the video/image information and signaled.

The intra prediction unit 331 may predict the current block by referring to samples in the current picture. The referenced samples may be located neighbors of the current block or may be located far away depending on the prediction mode. Prediction modes in intra prediction may include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 331 may also determine the prediction mode to be applied to the current block using the prediction modes applied to the neighboring blocks.

The inter prediction unit 332 may derive a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring block may include a spatial neighboring block present in the current picture and a temporal neighboring block present in the reference picture. For example, the inter prediction unit 332 may construct a motion information candidate list based on the neighboring blocks, and derive a motion vector and/or a reference picture index for the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes, and the information regarding the prediction can include information indicating the mode of inter prediction for the current block.

The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to a prediction signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 332 and/or the intra prediction unit 331). When there is no residual for the block to be processed, such as when a skip mode is applied, the predicted block can be used as the reconstructed block.

The adder 340 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of the next block to be processed in the current picture, may be output after filtering as described below, or may be used for inter prediction of the next picture.

Meanwhile, LMCS (luma mapping with chroma scaling) can also be applied during the picture decoding process.

The filtering unit 350 may apply filtering to the reconstructed signal to improve subjective/objective image quality. For example, the filtering unit 350 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may transmit the modified reconstructed picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods may include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc.

The (modified) reconstructed picture stored in the DPB of the memory 360 can be used as a reference picture in the inter prediction unit 332. The memory 360 can store motion information of a block from which motion information in the current picture is derived (or decoded) and/or motion information of a block in an already reconstructed picture. The stored motion information can be transmitted to the inter prediction unit 260 to be used as motion information of a spatial neighboring block or motion information of a temporal neighboring block. The memory 360 can store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra prediction unit 331.

In this document, the embodiments described for the filtering unit 260, inter prediction unit 221, and intra prediction unit 222 of the encoding device 200 can be applied identically or correspondingly to the filtering unit 350, inter prediction unit 332, and intra prediction unit 331 of the decoding device 300, respectively.

As described above, prediction is performed to improve compression efficiency when performing video coding. Through this, a predicted block including predicted samples for a current block, which is a block to be coded, can be generated. Here, the predicted block includes predicted samples in the spatial domain (or pixel domain). The predicted block is derived in the same manner by the encoding device and the decoding device, and the encoding device can improve image coding efficiency by signaling information (residual information) regarding the residual between an original block and a predicted block, which is not the original sample value of the original block itself, to the decoding device. The decoding device can derive a residual block including residual samples based on the residual information, combine the residual block with the predicted block to generate a restored block including restored samples, and generate a restored picture including the restored block.

The residual information may be generated through a transform and quantization procedure. For example, the encoding device may derive a residual block between an original block and a predicted block, perform a transform procedure on the residual samples (residual sample array) included in the residual block to derive transform coefficients, and perform a quantization procedure on the transform coefficients to derive quantized transform coefficients, thereby signaling the related residual information to the decoding device (via a bitstream). Here, the residual information may include information such as value information, position information, transform technique, transform kernel, and quantization parameter of the quantized transform coefficients. The decoding device may derive a residual sample (or a residual block) by performing an inverse quantization/inverse transform procedure based on the residual information. The decoding device may generate a reconstructed picture based on the predicted block and the residual block. In addition, the encoding device may derive a residual block by inverse quantizing/inverse transforming the quantized transform coefficients for reference for inter-prediction of a future picture, and generate a reconstructed picture based on the residual block.

Intra prediction may refer to a prediction that generates a prediction sample for a current block based on a reference sample in a picture to which the current block belongs (hereinafter, referred to as a current picture). When intra prediction is applied to the current block, neighboring reference samples used for intra prediction of the current block may be derived. The neighboring reference samples of the current block may include a total of 2×nH samples adjacent to the left boundary and bottom-left of the current block having a size of nW×nH, a total of 2×nW samples adjacent to the top boundary and top-right of the current block, and one sample adjacent to the top-left of the current block. Alternatively, the neighboring reference samples of the current block may include upper neighboring samples of multiple columns and left neighboring samples of multiple rows. In addition, the neighboring reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block having a size of nW x nH, a total of nW samples adjacent to the bottom boundary of the current block, and one sample adjacent to the bottom-right side of the current block.

However, some of the neighboring reference samples of the current block may not yet be decoded or available. In this case, the decoder may construct neighboring reference samples to be used for prediction by substituting unavailable samples as available samples. Alternatively, the decoder may construct neighboring reference samples to be used for prediction through interpolation of available samples.

When the neighboring reference samples are derived, (i) the prediction sample can be derived based on an average or an interpolation of the neighboring reference samples of the current block, or (ii) the prediction sample can be derived based on a reference sample that exists in a specific (prediction) direction with respect to the prediction sample among the neighboring reference samples of the current block. Case (i) can be called a non-directional mode or a non-angular mode, and case (ii) can be called a directional mode or an angular mode.

In addition, a prediction sample may be generated by interpolating a first adjacent sample located in a prediction direction of an intra prediction mode of the current block and a second adjacent sample located in an opposite direction to the prediction direction based on a prediction sample of the current block among the adjacent reference samples. This may be called linear interpolation intra prediction (LIP). In addition, a chroma prediction sample may be generated based on a luma sample using a linear model (LM). In this case, it may be called an LM mode or a CCLM (chroma component LM) mode.

In addition, a temporary prediction sample of the current block may be derived based on the filtered neighboring reference sample, and a prediction sample of the current block may be derived by taking a weighted sum of at least one reference sample derived according to the intra prediction mode among existing neighboring reference samples, i.e., non-filtered neighboring reference samples, and the temporary prediction sample. The above case may be called position dependent intra prediction (PDPC).

In addition, intra prediction coding can be performed by selecting a reference sample line with the highest prediction accuracy from among the adjacent multiple reference sample lines of the current block, deriving a prediction sample using a reference sample located in the prediction direction on the corresponding line, and signaling the reference sample line used at this time to a decoding device. The above case can be called multi-reference line intra prediction or MRL-based intra prediction.

In addition, the current block may be divided into vertical or horizontal sub-partitions, and intra prediction may be performed based on the same intra prediction mode, and adjacent reference samples may be derived and used on a sub-partition basis. That is, in this case, the intra prediction mode for the current block is equally applied to the sub-partitions, and adjacent reference samples may be derived and used on a sub-partition basis, thereby improving intra prediction performance in some cases. Such a prediction method may be called ISP (intra sub-partitions) based intra prediction.

The intra prediction methods described above may be called intra prediction types, distinguished from intra prediction modes. The intra prediction types may be called by various terms, such as intra prediction techniques or additional intra prediction modes. For example, the intra prediction types (or additional intra prediction modes, etc.) may include at least one of the above-mentioned LIP, PDPC, MRL, and ISP. A general intra prediction method other than the specific intra prediction types, such as the LIP, PDPC, MRL, and ISP, may be called a normal intra prediction type. The normal intra prediction type may be generally applied when the above-mentioned specific intra prediction types are not applied, and prediction may be performed based on the above-mentioned intra prediction modes. Meanwhile, post-processing filtering may be performed on the derived prediction samples, if necessary.

Specifically, the intra prediction procedure may include an intra prediction mode/type determination step, an adjacent reference sample derivation step, and an intra prediction mode/type-based prediction sample derivation step. In addition, if necessary, a post-processing filtering step may be performed on the derived prediction sample.

When intra prediction is applied, the intra prediction mode applied to the current block may be determined using the intra prediction mode of the surrounding block. For example, the decoding device may select one of the MPM candidates in the MPM (most probable mode) list derived based on the intra prediction modes of the surrounding blocks (ex. left and/or upper surrounding blocks) of the current block and additional candidate modes based on the received MPM index, or may select one of the remaining intra prediction modes not included in the MPM candidates (and the planar mode) based on the remaining intra prediction mode information. The MPM list may be configured to include or not include the planar mode as a candidate. For example, if the MPM list includes the planar mode as a candidate, the MPM list may have six candidates, and if the MPM list does not include the planar mode as a candidate, the MPM list may have five candidates. If the MPM list does not include the planar mode as a candidate, a not planar flag (ex. intra_luma_not_planar_flag) indicating that the intra prediction mode of the current block is not a planar mode may be signaled. For example, the MPM flag may be signaled first, and the MPM index and the not planar flag may be signaled if the value of the MPM flag is 1. Also, the MPM index may be signaled if the value of the not planar flag is 1. Here, the reason why the MPM list is configured not to include the planar mode as a candidate is that the planar mode is always considered as an MPM, rather than the planar mode being not an MPM, so the flag (not planar flag) is signaled first to first check whether the mode is a planar mode.

For example, whether the intra prediction mode applied to the current block is among the MPM candidates (and planar mode) or among the remaining mode can be indicated based on the MPM flag (ex.intra_luma_mpm_flag). A value of 1 of the MPM flag can indicate that the intra prediction mode for the current block is among the MPM candidates (and planar mode), and a value of 0 of the MPM flag can indicate that the intra prediction mode for the current block is not among the MPM candidates (and planar mode). A value of 0 of the not planar flag (ex.intra_luma_not_planar_flag) can indicate that the intra prediction mode for the current block is planar mode, and a value of 1 of the not planar flag can indicate that the intra prediction mode for the current block is not planar mode. The MPM index may be signaled in the form of an mpm_idx or intra_luma_mpm_idx syntax element, and the remaining intra prediction mode information may be signaled in the form of a rem_intra_luma_pred_mode or intra_luma_mpm_reminder syntax element. For example, the remaining intra prediction mode information may index the remaining intra prediction modes not included in the MPM candidates (and planar modes) among all intra prediction modes in order of prediction mode numbers, and may indicate one of them. The intra prediction mode may be an intra prediction mode for a luma component (sample). Hereinafter, the intra prediction mode information may include at least one of an MPM flag (ex.intra_luma_mpm_flag), a not planar flag (ex.intra_luma_not_planar_flag), an MPM index (ex.mpm_idx or intra_luma_mpm_idx), and remaining intra prediction mode information (rem_intra_luma_pred_mode or intra_luma_mpm_reminder). In this document, the MPM list may be referred to by various terms such as an MPM candidate list, a candModeList, etc. If matrix-based intra prediction (MIP) is applied to the current block, a separate mpm flag for MIP (ex.intra_mip_mpm_flag), mpm index (ex.intra_mip_mpm_idx), and remaining intra prediction mode information (ex.intra_mip_mpm_reminder) may be signaled, and the not planar flag is not signaled.

In other words, in general, when an image is divided into blocks, the current block to be coded and the neighboring blocks have similar image characteristics. Therefore, the current block and the neighboring blocks are highly likely to have the same or similar intra prediction modes. Therefore, the encoder can use the intra prediction modes of the neighboring blocks to encode the intra prediction mode of the current block.

For example, the encoder/decoder may configure a most probable modes (MPM) list for the current block. The MPM list may also be referred to as an MPM candidate list. Here, MPM may refer to a mode used to improve coding efficiency in consideration of similarity between the current block and a neighboring block during intra prediction mode coding. As described above, the MPM list may be configured to include a planar mode or may be configured to exclude a planar mode. For example, if the MPM list includes a planar mode, the number of candidates in the MPM list may be six. And, if the MPM list does not include a planar mode, the number of candidates in the MPM list may be five. The encoder/decoder may configure an MPM list including five or six MPMs.

To construct the MPM list, three types of modes may be considered: default intra modes, neighborhood intra modes, and derived intra modes. For the neighborhood intra modes, two neighboring blocks, i.e., the left neighboring block and the top neighboring block, may be considered.

As described above, if the MPM list is configured not to include planar mode, planar mode is removed from the list and the number of MPM list candidates can be set to five.

Furthermore, among the intra prediction modes, the non-directional mode (or non-angular mode) may include a DC mode based on the average of neighboring reference samples of the current block or a planar mode based on interpolation.

When inter prediction is applied, the prediction unit of the encoding device/decoding device can perform inter prediction on a block-by-block basis to derive a prediction sample. Inter prediction can refer to a prediction derived in a manner that is dependent on data elements (e.g., sample values or motion information) of picture(s) other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) for the current block may be derived based on a reference block (reference sample array) identified by a motion vector on a reference picture pointed to by a reference picture index. In this case, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information of the current block may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. In addition, the motion information may further include inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.) information. When inter prediction is applied, the neighboring blocks may include a spatial neighboring block present in the current picture and a temporal neighboring block present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different. A temporally adjacent block may be called a collocated reference block, a collocated CU (colCU), etc., and a reference picture including a temporally adjacent block may be called a collocated picture (colPic). For example, a motion information candidate list may be constructed based on the neighboring blocks of the current block, and flag or index information indicating which candidate is selected (used) to derive a motion vector and/or a reference picture index of the current block may be signaled. Inter prediction may be performed based on various prediction modes, for example, in the case of skip mode and merge mode, the motion information of the current block is the same as the motion information of the selected neighboring block. In the case of skip mode, unlike the merge mode, a residual signal is not transmitted. In the case of a motion vector prediction (MVP) mode, the motion vector of the selected neighboring block is used as a motion vector predictor, and a motion vector difference can be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

The motion information may include L0 motion information and/or L1 motion information depending on the inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.). A motion vector in the L0 direction may be referred to as an L0 motion vector or MVL0, and a motion vector in the L1 direction may be referred to as an L1 motion vector or MVL1. A prediction based on an L0 motion vector may be referred to as an L0 prediction, a prediction based on an L1 motion vector may be referred to as an L1 prediction, and a prediction based on both an L0 motion vector and an L1 motion vector may be referred to as a pairwise (Bi) prediction. Here, the L0 motion vector may indicate a motion vector associated with a reference picture list L0 (L0), and the L1 motion vector may indicate a motion vector associated with a reference picture list L1 (L1). The reference picture list L0 may include a previous picture in the output order from the current picture as a reference picture, and the reference picture list L1 may include a subsequent picture in the output order from the current picture. The previous picture may be referred to as a forward (reference) picture, and the subsequent picture may be referred to as a backward (reference) picture. The reference picture list L0 may further include pictures that are later in the output order than the current picture as reference pictures. In this case, the previous picture may be indexed first in the reference picture list L0, and the later picture may be indexed next. The reference picture list L1 may further include pictures that are earlier in the output order than the current picture as reference pictures. In this case, the later picture may be indexed first in the reference picture list L1, and the previous picture may be indexed next. Here, the output order may correspond to a picture order count (POC) order.

Figure 4 shows an example of a schematic video/image encoding method to which embodiments of this document can be applied.

The method disclosed in FIG. 4 may be performed by the encoding device 200 of FIG. 2 described above. Specifically, S400 may be performed by the inter prediction unit 221 or the intra prediction unit 222 of the encoding device 200, and S410, S420, S430, and S440 may be performed by the subtraction unit 231, the transformation unit 232, the quantization unit 233, and the entropy encoding unit 240 of the encoding device 200, respectively.

As shown in FIG. 4, the encoding device may derive a prediction sample through prediction for a current block (S400). The encoding device may determine whether to perform inter prediction or intra prediction for the current block, and may determine a specific inter prediction mode or a specific intra prediction mode based on the RD cost. Depending on the determined mode, the encoding device may derive a prediction sample for the current block.

The encoding device can derive a residual sample by comparing the original sample and the predicted sample for the current block (S410).

The encoding device can derive transform coefficients through a transform procedure on the residual samples (S420) and quantize the derived transform coefficients to derive quantized transform coefficients (S430).

The encoding device may encode image information including prediction information and residual information, and output the encoded image information in the form of a bitstream (S440). The prediction information is information related to a prediction procedure, and may include prediction mode information and information related to motion information (e.g., when inter prediction is applied). The residual information may include information related to quantized transform coefficients. The residual information may be entropy coded.

The output bitstream can be transmitted to a decoding device via a storage medium or a network.

Figure 5 shows an example of a general video/image decoding method to which embodiments of this document can be applied.

The method disclosed in FIG. 5 may be performed by the decoding device 300 of FIG. 3 described above. Specifically, S500 may be performed by the inter prediction unit 332 or the intra prediction unit 331 of the decoding device 300. The procedure of decoding prediction information included in the bitstream in S500 and deriving values of related syntax elements may be performed by the entropy decoding unit 310 of the decoding device 300. S510, S520, S530, and S540 may be performed by the entropy decoding unit 310, the inverse quantization unit 321, the inverse transform unit 322, and the addition unit 340 of the decoding device 300, respectively.

As shown in FIG. 5, the decoding device may perform operations corresponding to those performed by the encoding device. The decoding device may perform inter prediction or intra prediction on the current block based on the received prediction information and derive a prediction sample (S500).

The decoding device may derive quantized transform coefficients for the current block based on the received residual information (S510). The decoding device may derive the quantized transform coefficients from the residual information through entropy decoding.

The decoding device can dequantize the quantized transform coefficients to derive the transform coefficients (S520).

The decoding device derives residual samples through an inverse transform procedure on the transform coefficients (S530).

The decoding device may generate a reconstructed sample for the current block based on the predicted sample and the residual sample, and generate a reconstructed picture based on the reconstructed sample (S540). As described above, an in-loop filtering procedure may then be further applied to the reconstructed picture.

Meanwhile, as described above, the quantization unit of the encoding device can apply quantization to the transform coefficients to derive the quantized transform coefficients, and the inverse quantization unit of the encoding device or the inverse quantization unit of the decoding device can apply inverse quantization to the quantized transform coefficients to derive the transform coefficients.

Generally, in video/image coding, a quantization rate can be changed, and the compression can be adjusted using the changed quantization rate. In terms of implementation, a quantization parameter (QP) can be used instead of directly using the quantization rate in consideration of complexity. For example, an integer value quantization parameter from 0 to 63 can be used, and each quantization parameter value can correspond to an actual quantization rate. A quantization parameter (QP _Y ) for a luma component (luma sample) and a quantization parameter (QP _C ) for a chroma component (chroma sample) can be set differently.

In the quantization process, a transform coefficient (C) is input, and divided into a quantization rate (Q _step ), and a quantized transform coefficient (C') can be obtained based on the quantization rate. In this case, the quantization rate is multiplied by a scale to obtain an integer form in consideration of the computational complexity, and a shift operation can be performed by a value corresponding to the scale value. A quantization scale can be derived based on the multiplication of the quantization rate and the scale value. That is, the quantization scale can be derived according to the QP. A quantization scale can also be applied to the transform coefficient (C), and a quantized transform coefficient (C') can be derived based on the quantization scale.

In the inverse quantization process, the quantized transform coefficient (C') is multiplied by a quantization rate (Q _step ) in the inverse process of the quantization process, and a restored transform coefficient (C'') can be obtained based on the multiplication. In this case, a level scale can be derived according to the quantization parameter, and the level scale can be applied to the quantized transform coefficient (C') to derive a restored transform coefficient (C'') based on the multiplication. The restored transform coefficient (C'') is slightly different from the original transform coefficient (C) due to loss in the transformation and/or quantization process. Therefore, the encoding device also performs inverse quantization in the same way as the decoding device.

In addition, an adaptive frequency weighting quantization technique that adjusts the quantization strength according to frequency may be applied. The adaptive frequency weighting quantization technique is a method of applying quantization strength differently according to frequency. The adaptive frequency weighting quantization may apply quantization strength for each frequency differently using a predefined quantization scaling matrix. That is, the above-mentioned quantization/dequantization process may be further performed based on the quantization scaling matrix. For example, a different quantization scaling matrix may be used depending on the size of the current block and/or whether the prediction mode applied to the current block is inter prediction or intra prediction to generate a residual signal of the current block. The quantization scaling matrix may be referred to as a quantization matrix or a scaling matrix. The quantization scaling matrix may be predefined. In addition, for frequency adaptive scaling, frequency-specific quantization scale information for the quantization scaling matrix may be configured/encoded in the encoding device and signaled to the decoding device. The frequency-specific quantization scale information may be referred to as quantization scaling information. The frequency-specific quantization scale information may include scaling list data (scaling_list_data). A (modified) quantization scaling matrix may be derived based on the scaling list data. In addition, the frequency-specific quantization scale information may include present flag information indicating whether scaling list data is present or absent. Alternatively, when scaling list data is signaled at a higher level (e.g., SPS), information indicating whether the scaling list data is modified at a lower level (e.g., PPS or tile group header, etc.) may further be included.

As mentioned above, scaling list data can be signaled to indicate the scaling matrix used for quantization/dequantization (frequency-based quantization).

Support for signaling default and user-defined scaling matrices exists in the HEVC standard and has now been adopted into the VVC standard. However, for the VVC standard, additional support for signaling the following features has been integrated:

- 3 modes for scaling matrix: OFF, DEFAULT, USER_DEFINED

- Larger size range for blocks (4x4 to 64x64 for luma, 2x2 to 32x32 for chroma)

- Quadrilateral transformation blocks (TBs)

-Dependent quantization

-Multiple Transform Selection (MTS)

- Large transforms with zeroing-out high frequency coefficients

- Intra sub-block partitioning (ISP)

- Intra Block Copy (IBC) (also called current picture referencing (CPR))

- DEFAULT scaling matrix for all TB sizes, default value is 16

It should be noted that the scaling matrix should not be applied to the Transform Skip (TS) and Secondary Transform (ST) for all sizes.

Below, the High Level Syntax (HSL) structure for supporting scaling lists in the VVC standard will be described in detail. First, a flag can be signaled via a Sequence Parameter Set (SPS) to indicate that a scaling list is available for the current coded video sequence (CVS) being decoded. Then, if the flag is available, an additional flag can be parsed in the SPS to indicate whether specific data is present in the scaling list. This can be shown in Table 1.

Table 1 is taken from the SPS to illustrate the scaling list for CVS.

The semantics of the syntax elements included in the SPS syntax in Table 1 can be shown in Table 2 below.

Referring to Tables 1 and 2, scaling_list_enabled_flag can be signaled from the SPS. For example, if the value of scaling_list_enabled_flag is 1, it can indicate that the scaling list is used in the scaling process for the transform coefficients, and if the value of scaling_list_enabled_flag is 0, it can indicate that the scaling list is not used in the scaling process for the transform coefficients. In this case, if the value of scaling_list_enabled_flag is 1, sps_scaling_list_data_present_flag can be further signaled from the SPS. For example, if the value of sps_scaling_list_data_present_flag is 1, it indicates that the scaling_list_data() syntax structure is present in the SPS, and if the value of sps_scaling_list_data_present_flag is 0, it may indicate that the scaling_list_data() syntax structure is not present in the SPS. If sps_scaling_list_data_present_flag is not present, the value of sps_scaling_list_data_present_flag may be inferred to be 0.

Also, a flag (e.g., pps_scaling_list_data_present_flag) can be parsed first in the Picture Parameter Set (PPS). If this flag is available, scaling_list_data() can be parsed in the PPS. If scaling_list_data() is present first in the SPS and parsed later in the PPS, the data in the PPS can take precedence over the data in the SPS. Table 3 below is excerpted from the PPS to explain the scaling list data.

The semantics of the syntax elements included in the PPS syntax in Table 3 can be shown in Table 4 below.

Referring to Tables 3 and 4, pps_scaling_list_data_present_flag can be signaled from the PPS. For example, if the value of pps_scaling_list_data_present_flag is 1, it can indicate that the scaling list data used for the picture referencing the PPS is derived based on the scaling list specified by the active SPS and the scaling list specified by the PPS. If the value of pps_scaling_list_data_present_flag is 0, it can indicate that the scaling list data used for the picture referencing the PPS is inferred to be the same as the scaling list specified by the active SPS. In this case, if the value of scaling_list_enabled_flag is 0, the value of pps_scaling_list_data_present_flag must be 0. If the value of scaling_list_enabled_flag is 1, the value of sps_scaling_list_data_present_flag is 0, and the value of pps_scaling_list_data_present_flag is 0, the default scaling list data can be used to derive the scaling factor array as described in the scaling list data semantics.

Scaling lists can be defined in the HEVC standard for the following quantization matrix sizes, which can be seen in Table 5 below. The supported range for quantization matrices has been extended to include 2x2 and 64x64 in the HEVC standard, with 4x4, 8x8, 16x16 and 32x32.

Table 5 defines sizeId for all quantization matrix sizes used. Using the above combinations, matrixIds can be assigned for different combinations of sizeId, coding unit prediction mode (CuPredMode), and color components. Here, CuPredModes that can be considered are inter, intra, and IBC (Intra Block Copy). Intra mode and IBC mode can be treated the same. Therefore, the same matrixId(s) can be shared for a given color component. Here, color components that can be considered are luma (Luma(Y)) and two color components (Cb and Cr). The assigned matrixIds can be shown as shown in Table 6 below.

Table 6 shows the matrixId by sizeId, prediction mode, and color component.

Table 7 below shows an example of a syntax structure for scaling list data (e.g., scaling_list_data()).

The semantics of the syntax elements included in the syntax of Table 7 can be shown as shown in Table 8 below.

Referring to Tables 7 and 8, to extract the scaling list data (e.g., scaling_list_data()), for all sizeIds from 1 to 6 and matrixIds from 0 to 5, the scaling list data can be applied to the 2x2 chroma components and 64x64 luma components. Then, a flag (e.g., scaling_list_pred_mode_flag) can be parsed to indicate whether the values of the scaling list are the same as the values of the reference scaling list. The reference scaling list is indicated by scaling_list_pred_matrix_id_delta[sizeId][matrixId]. However, if scaling_list_pred_mode_flag[sizeId][matrixId] is 1, the scaling list data can be explicitly signaled. If scaling_list_pred_matrix_id_delta is 0, the DEFAULT mode with the default value may be used as shown in Tables 9 to 12. For other values of scaling_list_pred_matrix_id_delta, the refMatrixId may be determined first as shown in the semantics of Table 8.

In explicit signaling, i.e. in USER_DEFINED mode, the maximum number of coefficients to be signaled can be determined in advance. For quantization block sizes 2x2, 4x4 and 8x8, all coefficients can be signaled. For sizes larger than 8x8, i.e. 16x16, 32x32 and 64x64, only 64 coefficients can be signaled. That is, an 8x8 base matrix is signaled and the remaining coefficients can be upsampled from the base matrix.

Table 9 below is an example showing the default values of ScalingList[1][matrixId][i] (i = 0..3).

Table 10 below is an example showing the default values of ScalingList[2][matrixId][i] (i = 0..15).

The following Table 11 is an example showing the default values of ScalingList[3..5][matrixId][i] (i = 0..63).

The following Table 12 is an example showing the default values of ScalingList[6][matrixId][i] (i = 0..63).

As mentioned above, the default scaling list data can be used to derive a scaling factor.

The five-dimensional array scaling factor ScalingFactor[sizeId][sizeId][matrixId][x][y] (where x, y = 0..(1<<sizeId)-1) can indicate the array of scaling factors using the variable sizeId shown in Table 5 and the variable matrixId shown in Table 6.

Table 13 below shows an example of deriving a scaling factor based on the quantization matrix size based on the default scaling list described above.

For a rectangular sized quantization matrix, the 5-dimensional array of scaling factors ScalingFactor[sizeIdW][sizeIdH][matrixId][x][y] (where x = 0..(1<<sizeIdW)-1, y = 0..(1<<sizeIdH)-1, sizeIdW!=sizeIdH) can indicate the array of scaling factors using the variables sizeIdW and sizeIdH shown in Table 15 below, and can be derived as shown in Table 14 below.

The tile size quantization matrix must be zero for samples that meet the following conditions:

-x>32

-y>32

- The decoded TU is not coded in the default transformation mode, (1<<sizeIdW)==32 and x>16

- The decoded TU is not coded in the default transformation mode, (1<<sizeIdH)==32 and y>16

Table 15 below is an example showing sizeIdW and sizeIdH depending on the quantization matrix size.

Further, as an example, the above-mentioned scaling list data (e.g., scaling_list_data()) can be described based on the syntax structure as shown in Table 16 below and the semantics as shown in Table 17 below. As described above, the scaling list, scaling matrix, scaling factor, etc. can be derived based on the syntax elements included in the scaling list data (e.g., scaling_list_data()) disclosed in Tables 16 and 17, and this process may be the same as or similar to the above-mentioned Tables 5 to 15.

In the following, this document proposes a method for efficiently signaling scaling list data when applying adaptive frequency-specific weighting techniques during the quantization/dequantization process.

Figure 6 shows an example of a hierarchical structure for coded images/video.

As shown in FIG. 6, the coded image/video is divided into a VCL (video coding layer) that handles the image/video decoding process and the image/video itself, a lower system that transmits and stores the coded information, and a NAL (network abstraction layer) that exists between the VCL and the lower system and is responsible for network adaptation functions.

The VCL can generate VCL data including compressed image data (slice data), or generate parameter sets including information such as a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), and a Video Parameter Set (VPS), or a Supplemental Enhancement Information (SEI) message that is additionally required for the image decoding process.

In the NAL, a NAL unit can be generated by adding header information (NAL unit header) to the RBSP (Raw Byte Sequence Payload) generated in the VCL. In this case, the RBSP means slice data, parameter set, SEI message, etc. generated in the VCL. The NAL unit header can include NAL unit type information identified by the RBSP data included in the NAL unit.

In addition, NAL units can be classified into VCL NAL units and non-VCL NAL units according to the RBSP generated by the VCL. A VCL NAL unit can refer to a NAL unit that contains information about an image (slice data), and a non-VCL NAL unit can refer to a NAL unit that contains information required to decode an image (parameter set or SEI message).

VCL NAL units and non-VCL NAL units can be transmitted over a network with header information according to the data standard of the lower system. For example, NAL units can be transformed into a data format of a specific standard such as H.266/VVC file format, RTP (Real-time Transport Protocol), TS (Transport Stream), etc., and transmitted over various networks.

As mentioned above, the NAL unit type of a NAL unit can be identified by the RBSP data structure included in the NAL unit, and information about the NAL unit type can be stored and signaled in the NAL unit header.

For example, NAL units can be broadly classified into VCL NAL unit types and non-VCL NAL unit types depending on whether the NAL unit contains information about an image (slice data). VCL NAL unit types can be classified according to the nature and type of the picture that the VCL NAL unit contains, and non-VCL NAL unit types can be classified according to the type of parameter set.

Below is an example of a NAL unit type identified by the type of parameter set that the Non-VCL NAL unit type contains.

-APS (Adaptation Parameter Set) NAL unit: Type for NAL units that contain APS

-DPS (Decoding Parameter Set) NAL unit: Type for NAL unit containing DPS

-VPS (Video Parameter Set) NAL unit: Type for NAL unit containing VPS

- SPS (Sequence Parameter Set) NAL unit: Type for NAL unit containing SPS

-PPS (Picture Parameter Set) NAL unit: Type for NAL units that contain PPS

-PH (Picture header) NAL unit: Type for NAL unit containing PH

The above-mentioned NAL unit type has syntax information for the NAL unit type, and the syntax information can be stored in the NAL unit header and signaled. For example, the syntax information is nal_unit_type, and the NAL unit type can be specified by the nal_unit_type value.

Meanwhile, as described above, one picture may include multiple slices, and one slice may include a slice header and slice data. In this case, one picture header may be added for multiple slices (slice header and slice data set) in one picture. The picture header (picture header syntax) may include information/parameters that are commonly applicable to pictures. In this document, a tile group may be mixed with or substituted for a slice or a picture. Also, in this document, a tile group header may be mixed with or substituted for a slice header or a picture header.

A slice header (slice header syntax) may include information/parameters commonly applicable to slices. An APS (APS syntax) or a PPS (PPS syntax) may include information/parameters commonly applicable to one or more slices or pictures. An SPS (SPS syntax) may include information/parameters commonly applicable to one or more sequences. A VPS (VPS syntax) may include information/parameters commonly applicable to multiple layers. A DPS (DPS syntax) may include information/parameters commonly applicable to video in general. A DPS may include information/parameters related to concatenation of a coded video sequence (CVS). In this document, high level syntax (HLS) may include at least one of the APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, picture header syntax, and slice header syntax.

In this document, the image/video information encoded from the encoding device to the decoding device and signaled in the form of a bitstream may include not only partitioning-related information within a picture, intra/inter prediction information, residual information, in-loop filtering information, etc., but also information contained in a slice header, information contained in a picture header, information contained in an APS, information contained in a PPS, information contained in an SPS, information contained in a VPS, and/or information contained in a DPS. In addition, the image/video information may further include information of a NAL unit header.

Meanwhile, the Adaptation Parameter Set (APS) is used in the VVC standard to transmit information for the Adaptive Loop Filter (ALF) and Luma Mapping with Chroma Scaling (LMCS) procedures. The APS also has an extensible structure so that it can be used to transmit other data structures (i.e., other syntax structures). Therefore, this document proposes a method of parsing/signaling scaling list data used for frequency-specific weighting via the APS.

As described above, the scaling list data is quantization scale information for frequency-specific weighting that can be applied during the quantization/dequantization process, and is a list that associates scale factors with each frequency index.

As one embodiment, Table 18 below shows an example of an adaptation parameter set (APS) structure used to transmit scaling list data.

The semantics of the syntax elements included in the APS syntax in Table 18 can be shown in Table 19 below.

Referring to Tables 18 and 19, the adaptation_parameter_set_id syntax element can be parsed/signaled in the APS. The adaptation_parameter_set_id provides an identifier for the APS for reference in other syntax elements. That is, the APS can be identified based on the adaptation_parameter_set_id syntax element. The adaptation_parameter_set_id syntax element can be referred to as APS ID information. The APS can be shared between pictures and may be different for different tile groups within a picture.

In addition, the aps_params_type syntax element can be parsed/signaled in the APS. The aps_params_type can indicate the type of APS parameters sent in the APS as shown in Table 18 below. The aps_params_type syntax element can be referred to as APS parameter type information or APS type information.

For example, the following Table 20 is an example showing the types of APS parameters that can be sent via APS, and each APS parameter type can be indicated corresponding to the value of aps_params_type.

Referring to Table 20, aps_params_type is a syntax element for classifying the type of the APS. If the value of aps_params_type is 0, the APS type is ALF_APS, and the APS can carry ALF data, which can include ALF parameters for deriving filters/filter coefficients. If the value of aps_params_type is 1, the APS type is LMCS_APS, and the APS can carry LMCS data, which can include LMCS parameters for deriving LMCS model/bin/mapping indexes. If the value of aps_params_type is 2, the APS type is SCALING_APS, and the APS can carry SCALING list data, which can include scaling list data parameters for deriving frequency-based quantization scaling matrix/scaling factor/scaling list values.

For example, as shown in Table 18, the aps_params_type syntax element can be parsed/signaled in APS, and in this case, if the value of aps_params_type indicates 0 (i.e., if aps_params_type indicates ALF_APS), ALF data (i.e., alf_data()) can be parsed/signaled. Or, if the value of aps_params_type indicates 1 (i.e., if aps_params_type indicates LMCS_APS), LMCS data (i.e., lmcs_data()) can be parsed/signaled. Alternatively, if the value of aps_params_type indicates 2 (i.e., if aps_params_type indicates SCALING_APS), scaling list data (i.e., scaling_list_data()) can be parsed/signaled.

Also, referring to Tables 18 and 19, the aps_extension_flag syntax element can be parsed/signaled in APS. The aps_extension_flag can indicate whether the APS extension data flag (aps_extension_data_flag) syntax element is present. The aps_extension_flag can be used, for example, to provide an extension point for a later version of the VVC standard. The aps_extension_flag syntax element can be referred to as an APS extension flag. For example, if the value of aps_extension_flag is 0, it can indicate that the APS extension data flag (aps_extension_data_flag) is not present in the APS RBSP syntax structure. Alternatively, if the value of aps_extension_flag is 1, it may indicate that the APS extension data flag (aps_extension_data_flag) is present in the APS RBSP syntax structure.

The aps_extension_data_flag syntax element can be parsed/signaled based on the aps_extension_flag syntax element. The aps_extension_data_flag syntax element can be referred to as an APS extension data flag. For example, if the value of aps_extension_flag is 1, aps_extension_data_flag can be parsed/signaled, and in this case, aps_extension_data_flag can have any value.

As described above, according to one embodiment of this document, the scaling list data can be efficiently transported by assigning a data type (e.g., SCALING_APS) to indicate the scaling list data and parsing/signaling a syntax element (e.g., aps_params_type) that indicates the data type. That is, according to one embodiment of this document, an APS structure that integrates the scaling list data can be used.

Meanwhile, in the current VVC standard, the use of scaling list data (i.e., scaling_list_data()) can first be indicated based on the presence of a flag (i.e., sps_scaling_list_enabled_flag) indicating the availability of scaling list data in the SPS (Sequence Parameter Set). If the flag (i.e., sps_scaling_list_enabled_flag) is enabled (i.e., 1 or true indicating that scaling list data is available), another flag (i.e., sps_scaling_list_data_present_flag) can be parsed. In addition, if sps_scaling_list_data_present_flag is enabled (i.e., if the scaling list data is present in the SPS and is 1 or true), the scaling list data (i.e., scaling_list_data()) can be parsed. That is, currently, the VVC standard signals the scaling list data in the SPS. In this case, since the SPS enables session negotiation and is generally transmitted out-of-band, it is not necessary to transmit the scaling list data with information related to determining the scaling factor of the transform block, which can be used during the decoding process. When the decoder transmits the scaling list data in the SPS, the decoder needs to reserve a significant amount of memory to store information obtained from the scaling list data, and also needs to maintain the information until it is used in transform block decoding. Therefore, such a process is not necessary at the SPS level, and it is more effective to parse/signal the scaling list data at a lower level. Therefore, this document proposes a hierarchical structure to effectively parse/signal the scaling list data.

In one embodiment, scaling list data is not parsed/signaled from a higher level syntax, SPS, but is parsed/signaled in a lower level syntax, PPS, tile group header, slice header, and/or other appropriate header.

For example, the SPS syntax can be modified as shown in Table 21 below. Table 19 below shows an example of SPS syntax to describe a scaling list for a CVS.

The semantics of the syntax elements included in the SPS syntax in Table 21 can be shown in Table 22 below.

Referring to Tables 21 and 22, the scaling_list_enabled_flag syntax element can be parsed/signaled in the SPS. The scaling_list_enabled_flag syntax element can indicate whether a scaling list is available based on whether its value is 0 or 1. For example, if the value of scaling_list_enabled_flag is 1, it can indicate that the scaling list is used in the scaling process for the transform coefficients, and if the value of scaling_list_enabled_flag is 0, it can indicate that the scaling list is not used in the scaling process for the transform coefficients.

That is, the scaling_list_enabled_flag syntax element can be referred to as a scaling list availability flag and can be signaled at the SPS (or SPS level). That is, based on the value of scaling_list_enabled_flag signaled at the SPS level, it can be determined that a scaling list is essentially available for pictures in the CVS that reference the SPS. Then, an additional availability flag can be signaled at a level lower than the SPS (e.g., PPS, tile group header, slice header, and/or other appropriate header) to obtain the scaling list.

In addition, the sps_scaling_list_data_present_flag syntax element may not be parsed/signaled in the SPS. That is, by removing the sps_scaling_list_data_present_flag syntax element in the SPS, this flag information may not be parsed/signaled. The sps_scaling_list_data_present_flag syntax element is flag information indicating whether or not the syntax structure of the scaling list data exists in the SPS, and the scaling list data specified by the SPS may be parsed/signaled according to this flag information. However, by removing the sps_scaling_list_data_present_flag syntax element, the scaling list data may not be parsed/signaled at the SPS level.

As described above, according to one embodiment of this document, the SPS level may be configured to explicitly signal only the scaling list availability flag (scaling_list_enabled_flag) without directly signaling the scaling list (scaling_list_data()). Thereafter, the scaling list (scaling_list_data()) may be parsed individually in the lower level syntax based on the availability flag (scaling_list_enabled_flag) in the SPS. Therefore, according to one embodiment of this document, the scaling list data may be parsed/signaled according to a hierarchical structure, thereby improving coding efficiency.

Meanwhile, the existence and use of scaling list data is conditioned on the existence of a tool enabling flag. Here, the tool enabling flag is information indicating whether the corresponding tool is enabled, and may include, for example, a scaling_list_enabled_flag syntax element. That is, the scaling_list_enabled_flag syntax element can be used to indicate whether the scaling list is enabled by indicating whether the scaling list data is available. However, this tool should have a syntactic constraint for the decoder. That is, there should be a constraint flag that informs the decoder that this tool is not currently used for decoding a CVS (coded video sequence). Therefore, this document proposes a method for applying a constraint flag to the scaling list data.

As one embodiment, Table 23 below shows an example of syntax (e.g., general restriction information syntax) for signaling scaling list data using restriction flags.

The semantics of the syntax elements included in the syntax of Table 23 can be shown in Table 22 below.

Referring to Tables 23 and 24, the constraint flag can be parsed/signaled via general_constraint_info(). general_constraint_info() is referred to as the general constraint information field or information on the constraint flag. For example, the no_scaling_list_constraint_flag syntax element can be used as the constraint flag. Here, the constraint flag can be used to specify conformance bitstream properties. For example, if the value of the no_scaling_list_constraint_flag syntax element is 1, it indicates a bitstream conformance requirement where scaling_list_enabled_flag should be set to 0, and if the value of the no_scaling_list_constraint_flag syntax element is 0, it can indicate that there is no restriction.

Meanwhile, as mentioned above, according to one embodiment of this document, the scaling list data can be transmitted via a hierarchical structure. Accordingly, this document proposes a structure of scaling list data that can be parsed/signaled via a slice header. Here, the slice header can also be referred to as a tile group header, or can be mixed with or substituted for the picture header.

As one embodiment, Table 25 below shows an example of slice header syntax for signaling scaling list data.

The semantics of the syntax elements included in the slice header syntax in Table 25 can be shown in Table 26 below.

Referring to Tables 25 and 26, the slice_pic_parameter_set_id syntax element can be parsed/signaled in the slice header. The slice_pic_parameter_set_id syntax element can indicate an identifier for the PPS in use. That is, the slice_pic_parameter_set_id syntax element is information for identifying the PPS referenced in the slice, and can indicate the value of pps_pic_parameter_set_id. The value of slice_pic_parameter_set_id must be within the range of 0 to 63. The slice_pic_parameter_set_id syntax element can be referred to as PPS identification information or PPS ID information referenced in the slice.

In addition, a slice_scaling_list_enabled_flag syntax element may be parsed/signaled in the slice header. The slice_scaling_list_enabled_flag syntax element may indicate whether a scaling list is available in the current slice. For example, if the value of slice_scaling_list_enabled_flag is 1, it may indicate that a scaling list is available in the current slice, and if the value of slice_scaling_list_enabled_flag is 0, it may indicate that a scaling list is not available in the current slice. Alternatively, if slice_scaling_list_enabled_flag is not present in the slice header, its value may be inferred to 0.

In this case, the slice_scaling_list_enabled_flag syntax element can be parsed or not based on the scaling_list_enabled_flag syntax element signaled in the upper level syntax (i.e., SPS). For example, if the value of the scaling_list_enabled_flag signaled in the SPS is 1 (i.e., if it is determined that the scaling list data is available at the upper level), the slice_scaling_list_enabled_flag can be parsed in the slice header to determine whether to perform a scaling process using the scaling list in the slice.

In addition, a slice_scaling_list_aps_id syntax element can be parsed/signaled in the slice header. The slice_scaling_list_aps_id syntax element can indicate an identifier for the APS referenced in the slice. That is, the slice_scaling_list_aps_id syntax element can indicate ID information (adaptation_parameter_set_id) of the APS containing the scaling list data referenced in the slice. Meanwhile, the TemporalId (i.e., TemporalID) of an APS NAL unit (i.e., an APS NAL unit containing scaling list data) having the same APS ID information (adaptation_parameter_set_id) as slice_scaling_list_aps_id must be smaller than or equal to the TemporalId (i.e., TemporalID) of the slice NAL unit being coded.

In addition, whether the slice_scaling_list_aps_id syntax element can be parsed can be determined based on the slice_scaling_list_enabled_flag syntax element. For example, if the value of slice_scaling_list_aps_id is 1 (i.e., if it is determined that the scaling list is available in the slice header), slice_scaling_list_aps_id can be parsed. Thereafter, scaling list data can be obtained from the APS indicated by the parsed slice_scaling_list_aps_id.

In addition, when multiple SCALING DATA APS (multiple APS including scaling list data) with the same APS ID information (adaptation_parameter_set_id) value are referenced by two or more slices in the same picture, the multiple SCALING DATA APS with the same APS ID information (adaptation_parameter_set_id) value must contain the same content.

In addition, if the aforementioned syntax elements are present, the values of the slice header syntax elements slice_pic_parameter_set_id, slice_pic_order_cnt_lsb, and slice_temporal_mvp_enabled_flag must be the same in all slice headers within the picture being coded.

As described above, according to one embodiment of this document, a hierarchical structure can be used to efficiently signal scaling list data. That is, an availability flag (e.g., scaling_list_enabled_flag) indicating whether scaling list data is available at a higher level (SPS syntax) is first signaled, and then an additional availability flag (e.g., slice_scaling_list_enabled_flag) is signaled at a lower level (e.g., slice header, picture header, etc.) to determine whether scaling list data is to be used at each lower level. In addition, APS ID information (e.g., slice_scaling_list_aps_id) referenced by the slice or tile group can be signaled via a lower level (e.g., slice header, picture header, etc.), and the scaling list data can be derived from the APS identified by the APS ID information.

In addition, this document can be applied to signal scaling list data using a hierarchical structure as proposed in Tables 23 and 24 above, or the scaling list data can be transmitted via a slice header structure as shown in Table 25 below.

As one embodiment, Table 27 below shows an example of slice header syntax for signaling scaling list data. Here, the slice header may also be referred to as a tile group header, or may be mixed with or substituted for the picture header.

The semantics of the syntax elements included in the slice header syntax in Table 27 can be shown in Table 28 below.

Referring to Tables 27 and 28, the slice_pic_parameter_set_id syntax element can be parsed/signaled in the slice header. The slice_pic_parameter_set_id syntax element can indicate an identifier for the PPS in use. That is, the slice_pic_parameter_set_id syntax element is information for identifying the PPS referenced in the slice, and can indicate the value of pps_pic_parameter_set_id. The value of slice_pic_parameter_set_id must be within the range of 0 to 63. The slice_pic_parameter_set_id syntax element can be referred to as PPS identification information or PPS ID information referenced in the slice.

In addition, a slice_scaling_list_aps_id syntax element can be parsed/signaled in the slice header. The slice_scaling_list_aps_id syntax element can indicate an identifier for the APS referenced in the slice. That is, the slice_scaling_list_aps_id syntax element can indicate the ID information (adaptation_parameter_set_id) of the APS containing the scaling list data referenced in the slice. As an example, the TemporalId (i.e., TemporalID) of an APS NAL unit (i.e., an APS NAL unit containing scaling list data) having the same APS ID information (adaptation_parameter_set_id) as slice_scaling_list_aps_id must be smaller than or equal to the TemporalId (i.e., TemporalID) of the slice NAL unit being coded.

In this case, the slice_scaling_list_aps_id syntax element can be parsed or not based on the scaling_list_enabled_flag syntax element signaled in the upper level syntax (i.e., SPS). For example, if the value of scaling_list_enabled_flag signaled in the SPS is 1 (i.e., if it is determined that scaling list data is available at the upper level), slice_scaling_list_aps_id can be parsed in the slice header. Thereafter, scaling list data can be obtained from the APS indicated by the parsed slice_scaling_list_aps_id.

That is, according to this embodiment, the APS ID including the scaling list data can be parsed if the corresponding flag (e.g., scaling_list_enabled_flag) in the SPS is enabled, so that, as shown in Table 25 above, the APS ID (e.g., slice_scaling_list_aps_id) information including the scaling list data referenced at the corresponding lower level (e.g., slice header or picture header) can be parsed based on the scaling_list_enabled_flag syntax element signaled in the higher level syntax (i.e., SPS).

This document also proposes a method for using multiple APSs to signal scaling list data. In the following, a method for efficiently signaling multiple APS IDs with scaling list data according to one embodiment of this document is described. This method is useful during bitstream merge.

As one embodiment, Table 29 below shows an example of slice header syntax for signaling scaling list data using multiple APS. Here, the slice header may also be referred to as a tile group header, or may be mixed with or substituted for the picture header.

The semantics of the syntax elements included in the slice header syntax in Table 29 can be shown in Table 30 below.

Referring to Tables 29 and 30, the slice_pic_parameter_set_id syntax element can be parsed/signaled in the slice header. The slice_pic_parameter_set_id syntax element can indicate an identifier for the PPS in use. That is, the slice_pic_parameter_set_id syntax element is information for identifying the PPS referenced in the slice, and can indicate the value of pps_pic_parameter_set_id. The value of slice_pic_parameter_set_id must be within the range of 0 to 63. The slice_pic_parameter_set_id syntax element can be said to be the PPS identification information or PPS ID information referenced by the slice.

In addition, a slice_scaling_list_enabled_flag syntax element may be parsed/signaled in the slice header. The slice_scaling_list_enabled_flag syntax element may indicate whether a scaling list is available in the current slice. For example, if the value of slice_scaling_list_enabled_flag is 1, it may indicate that a scaling list is available in the current slice, and if the value of slice_scaling_list_enabled_flag is 0, it may indicate that a scaling list is not available in the current slice. Alternatively, if slice_scaling_list_enabled_flag is not present in the slice header, its value may be inferred to 0.

In this case, the slice_scaling_list_enabled_flag syntax element can be parsed or not based on the scaling_list_enabled_flag syntax element signaled in the upper level syntax (i.e., SPS). For example, if the value of the scaling_list_enabled_flag signaled in the SPS is 1 (i.e., if it is determined that the scaling list data is available at the upper level), the slice_scaling_list_enabled_flag can be parsed in the slice header to determine whether to perform a scaling process using the scaling list in the slice.

In addition, the num_scaling_list_aps_ids_minus1 syntax element can be parsed/signaled in the slice header. The num_scaling_list_aps_ids_minus1 syntax element is information for indicating the number of APSs including scaling list data referenced by the slice. For example, the value of the num_scaling_list_aps_ids_minus1 syntax element plus 1 is the number of APSs. The value of num_scaling_list_aps_ids_minus1 must be in the range of 0 to 7.

Here, the parsing of the num_scaling_list_aps_ids_minus1 syntax element can be determined based on the slice_scaling_list_enabled_flag syntax element. For example, if the value of slice_scaling_list_enabled_flag is 1 (i.e., it is determined that scaling list data is available in the slice), num_scaling_list_aps_ids_minus1 can be parsed. In this case, the slice_scaling_list_aps_id[i] syntax element can be parsed/signaled based on the value of num_scaling_list_aps_ids_minus1.

That is, slice_scaling_list_aps_id[i] may indicate the identifier (adaptation_parameter_set_id) of the APS (i.e., the i-th SCALING LIST APS) including the i-th scaling list data. That is, APS ID information may be signaled for the number of APSs indicated by the num_scaling_list_aps_ids_minus1 syntax element. Meanwhile, the TemporalId (i.e., TemporalID) of an APS NAL unit (i.e., an APS NAL unit containing scaling list data) having the same APS ID information (adaptation_parameter_set_id) as slice_scaling_list_aps_id[i] must be smaller than or equal to the TemporalId (i.e., TemporalID) of the slice NAL unit being coded.

In addition, when multiple SCALING DATA APS (multiple APS including scaling list data) with the same APS ID information (adaptation_parameter_set_id) value are referenced by two or more slices in the same picture, the multiple SCALING DATA APS with the same APS ID information (adaptation_parameter_set_id) value must contain the same content.

This document also proposes a method for signaling scaling list data in a hierarchical structure to avoid redundant signaling. In one embodiment, signaling of scaling list data in PPS (Picture Parameter Set) can be eliminated. The scaling list data can be fully signaled in SPS, or APS, and/or other suitable header sets.

As one embodiment, Table 31 below shows an example of PPS syntax that does not signal scaling list data in the PPS.

The following Table 32 is an example showing the semantics of syntax elements (e.g., pps_scaling_list_data_present_flag) that can be removed to avoid redundant signaling of scaling list data in the PPS syntax of Table 31.

With reference to Tables 31 and 32, information for signaling scaling list data in the PPS, for example, the pps_scaling_list_data_present_flag syntax element, can be removed. The pps_scaling_list_data_present_flag syntax element can indicate whether scaling list data is signaled in the PPS based on whether its value is 0 or 1. For example, if the value of the pps_scaling_list_data_present_flag syntax element is 1, it can indicate that the scaling list data used for a picture referencing the PPS is derived based on the scaling list specified by the active SPS and the scaling list data specified by the PPS. If the value of the pps_scaling_list_data_present_flag syntax element is 0, it may indicate that the scaling list data used for a picture that references a PPS is inferred to be the same as that specified by the active SPS. That is, the pps_scaling_list_data_present_flag syntax element may be information indicating whether or not scaling list data signaled from the PPS is present.

In other words, in this embodiment, in order to prevent redundant signaling of scaling list data, the pps_scaling_list_data_present_flag syntax element is removed (i.e., the pps_scaling_list_data_present_flag syntax element is not signaled) as shown in Table 31, thereby preventing the scaling list data from being parsed/signaled by the PPS.

This document also proposes a method for signaling the scaling list matrix in the APS. Existing methods use three modes (i.e., OFF, DEFAULT, and USER_DEFINED modes). The OFF mode indicates that the scaling list data is not applied to the transform block. The DEFAULT mode indicates that a fixed value is used to generate the scaling matrix. The USER_DEFINED mode indicates that a scaling matrix is used based on the block size, prediction mode, and color components. Currently, the total number of scaling matrices supported in VVC is 44, which is an increase from 28 in HEVC. Currently, the scaling list data is signaled in the SPS and can be conditionally present in the PPS. By signaling the scaling list data in the APS, redundant signaling of the same data in the SPS and PPS is unnecessary and can be eliminated.

For example, the scaling matrix currently defined in VVC indicates whether it is available (enabled) in the SPS using scaling_list_enabled_flag. If this flag indicates that it is available, the scaling list is used in the scaling process for the transform coefficients, and if this flag indicates that it is not available, the scaling list is not used in the scaling process for the transform coefficients (e.g., OFF mode). In addition, sps_scaling_list_data_present_flag, which indicates whether scaling list data exists in the SPS, can be parsed. In addition to signaling in the SPS, the scaling list data can also exist in the PPS. If pps_scaling_list_data_present_flag indicates that it is available in the PPS, the scaling list data can exist in the PPS. If the scaling list data exists in both the SPS and the PPS, the scaling list data from the PPS can be used for frames that refer to the active PPS. If the scaling_list_enable_flag indicates availability, but the scaling list data exists only in the SPS and not in the PPS, the scaling list data from the SPS can be referenced by the frame. If the scaling_list_enable_flag indicates availability and the scaling list data does not exist in the SPS or the PPS, the DEFAULT mode can be used. The use of the DEFAULT mode can also be signaled within the scaling list data itself. If the scaling list data is explicitly signaled, the USER DEFINED mode can be used. The scaling factor for a given transform block can be determined using the information signaled in the scaling list. It is proposed to signal the scaling list data in the APS.

Currently, VVC adopts and uses scaling list data, and the scaling matrices supported by VVC are wider than those supported by HEVC. The scaling matrices supported by VVC allow block sizes to be selected from 4x4 to 64x64 for luma and from 2x2 to 32x32 for chroma. In addition, rectangular transform block (TB) size, dependent quantization, multiple transform selection, large transform with zeroing out high frequency coefficients for large transform blocks, intra subblock partitioning (ISP), and intra block copy (IBC) can be integrated. Intra block copy (IBC) and intra coding modes can share the same scaling matrix.

Therefore, for USER_DEFINED mode, the number of matrices signaled can be:

・MatrixType: 30 = 2 (2 for intra & IBC/inter) x 3 (Y/Cb/Cr components) x 5 (square TB size: from 4 x 4 to 64 x 64 for luma, from 2 x 2 to 32 x 32 for chroma)

・MatrixType_DC: 14 = 2 (2 for intra & IBC/inter x 1 for Y component) x 3 (TB size: 16 x 16, 32 x 32, 64 x 64) + 4 (2 for intra & IBC/inter x 2 for Cb/Cr components) x 2 (TB size: 16 x 16, 32 x 32)

DC values can be coded separately for scaling matrices of sizes 16x16, 32x32, and 64x64. If the transform block (TB) size is smaller than 8x8, all elements can be signaled in one scaling matrix. If the transform block (TB) size is greater than or equal to 8x8, only 64 elements in one 8x8 scaling matrix can be signaled as the base scaling matrix. To obtain a square matrix larger than 8x8, the base 8x8 matrix can be upsampled to the required size. If the DEFAULT mode is used, the scaling matrix can be set to 16. Thus, VVC supports 44 different matrices, whereas HEVC supports only 28 matrices. Since the number of scaling matrices supported in VVC is more extensive than HEVC, APS can be used as a more practical choice for signaling scaling list data to avoid redundant signaling in SPS and/or PPS. This avoids unnecessary and redundant signaling of scaling list data.

In an attempt to solve the above problems, this document proposes a method of signaling a scaling matrix in the APS. For this purpose, the scaling list data can be signaled only in the APS, without the need to be signaled in the SPS or conditionally present in the PPS. In addition, the APS ID can be signaled in the slice header.

In one embodiment, a flag (e.g., scaling_list_enable_flag) indicating whether scaling list data is available in the SPS is signaled, and a flag (e.g., pps_scaling_list_data_present_flag) indicating whether scaling list data is present in the PPS is removed, so that scaling list data can be signaled in the APS without signaling. Also, the APS ID can be signaled in the slice header. In this case, if the value of the scaling_list_enable_flag signaled in the SPS is 1 (i.e., indicating that the scaling list data is available) and the APS ID is not signaled in the slice header, the DEFAULT scaling matrix may be used. One embodiment of this document can be realized with the syntax and semantics shown in Tables 33 to 41 below.

The following Table 33 shows an example of an APS structure used to signal scaling list data.

The semantics of the syntax elements included in the APS syntax in Table 33 can be expressed as shown in Table 34 below.

Referring to Tables 33 and 34, the adaptation_parameter_set_id syntax element can be parsed/signaled in the APS. The adaptation_parameter_set_id provides an identifier for the APS for reference in other syntax elements. That is, the APS can be identified based on the adaptation_parameter_set_id syntax element. The adaptation_parameter_set_id syntax element can be referred to as APS ID information. The APS can be shared between pictures and can be different in different slices within a picture.

In addition, the aps_params_type syntax element can be parsed/signaled in the APS. The aps_params_type can indicate the type of APS parameters sent from the APS as shown in Table 35 below. The aps_params_type syntax element can be referred to as APS parameter type information or APS type information.

For example, the following Table 35 shows an example of the types of APS parameters that can be sent via APS, and each APS parameter type can be represented corresponding to the value of aps_params_type.

Referring to Table 35, aps_params_type may be a syntax element for classifying the type of the APS. If the value of aps_params_type is 0, the APS type may be ALF_APS, and the APS may carry ALF data, which may include ALF parameters for deriving filters/filter coefficients. If the value of aps_params_type is 1, the APS type may be LMCS_APS, and the APS may carry LMCS data, which may include LMCS parameters for deriving LMCS model/bin/mapping indexes. If the value of aps_params_type is 2, the APS type can be SCALING_APS, and the APS can carry SCALING list data, which can include frequency-based quantization scaling matrix/scaling, scaling list data parameters for deriving factor/scaling list values.

For example, as shown in Table 33, the aps_params_type syntax element can be parsed/signaled in APS, and in this case, if the value of aps_params_type represents 0 (i.e., if aps_params_type represents ALF_APS), ALF data (i.e., alf_data()) can be parsed/signaled. Or, if the value of aps_params_type represents 1 (i.e., if aps_params_type represents LMCS_APS), LMCS data (i.e., lmcs_data()) can be parsed/signaled. Alternatively, if the value of aps_params_type represents 2 (i.e., if aps_params_type represents SCALING_APS), scaling list data (i.e., scaling_list_data()) can be parsed/signaled.

Also, referring to Tables 33 and 34, the aps_extension_flag syntax element can be parsed/signaled in APS. The aps_extension_flag can indicate whether the APS extension data flag (aps_extension_data_flag) syntax element is present or not. The aps_extension_flag can be used, for example, to provide an extension point for a later version of the VVC standard. The aps_extension_flag syntax element can be referred to as an APS extension flag. For example, if the value of aps_extension_flag is 0, it can indicate that the APS extension data flag (aps_extension_data_flag) is not present in the APS RBSP syntax structure. Alternatively, if the value of aps_extension_flag is 1, it may indicate that the APS extension data flag (aps_extension_data_flag) is present in the APS RBSP syntax structure.

The aps_extension_data_flag syntax element can be parsed/signaled based on the aps_extension_flag syntax element. The aps_extension_data_flag syntax element can be referred to as an APS extension data flag. For example, if the value of aps_extension_flag is 1, aps_extension_data_flag can be parsed/signaled, and in this case, aps_extension_data_flag can have any value.

As described above, according to one embodiment of this document, scaling list data can be efficiently transported by assigning a data type (e.g., SCALING_APS) to represent the scaling list data and parsing/signaling a syntax element (e.g., aps_params_type) that represents the data type. In other words, according to one embodiment of this document, an APS structure that integrates scaling list data can be used.

In addition, when signaling scaling list data in the APS, the SPS may signal whether or not the scaling list data is available, and based on this, the APS may parse/signal the scaling list data according to the APS parameter type (e.g., aps_params_type). In addition, in one embodiment of this document, in order to avoid signaling redundant scaling list data in higher level syntax, flag information indicating whether or not the scaling list data syntax structure (e.g., scaling_list_data()) is present in the SPS or PPS may not be parsed/signaled from the SPS or PPS, so that the scaling list data may not be signaled in the SPS or PPS. This may be achieved by the syntax and semantics shown in Tables 36 to 39 below.

For example, the SPS syntax can be modified as shown in Table 36 below. Table 36 below shows an example of an SPS syntax structure that does not signal scaling list data in the SPS.

The semantics of the syntax elements included in the SPS syntax of Table 36 may be modified as shown in Table 37 below. As an example, in Tables 36 and 37, some syntax elements (e.g., sps_scaling_list_data_present_flag) may be removed from the syntax elements included in the SPS to avoid redundant signaling of scaling list data.

Referring to Tables 36 and 37, the scaling_list_enabled_flag syntax element can be parsed/signaled in the SPS. The scaling_list_enabled_flag syntax element can indicate whether a scaling list is available or not based on whether its value is 0 or 1. For example, if the value of scaling_list_enabled_flag is 1, it indicates that the scaling list is used in the scaling process for the transform coefficients, and if the value of scaling_list_enabled_flag is 0, it indicates that the scaling list is not used in the scaling process for the transform coefficients.

That is, the scaling_list_enabled_flag syntax element can be referred to as a scaling list availability flag and can be signaled at the SPS (or SPS level). In other words, based on the value of scaling_list_enabled_flag signaled at the SPS level, it can be determined that a scaling list is essentially available for pictures in the CVS that reference the SPS. Then, an additional availability flag can be signaled at a level lower than the SPS (e.g., PPS, tile group header, slice header, and/or other appropriate header) to obtain the scaling list.

In addition, the sps_scaling_list_data_present_flag syntax element may not be parsed/signaled in the SPS. That is, by removing the sps_scaling_list_data_present_flag syntax element in the SPS, this flag information may not be parsed/signaled. The sps_scaling_list_data_present_flag syntax element is flag information indicating whether or not the syntax structure of the scaling list data exists in the SPS, and the scaling list data specified by the SPS may be parsed/signaled according to this flag information. However, by removing the sps_scaling_list_data_present_flag syntax element, the SPS level can be configured to explicitly signal only the scaling list available flag (scaling_list_enabled_flag) without directly signaling the scaling list data.

Furthermore, the signaling of scaling list data in the PPS can be removed as shown in Table 38 below. For example, Table 38 below shows an example of a PPS syntax structure that does not signal scaling list data in the PPS.

The semantics of the syntax elements included in the PPS syntax of Table 38 can be modified as shown in Table 39 below. As an example, Table 39 below shows the semantics for syntax elements (e.g., pps_scaling_list_data_present_flag) that can be removed to avoid redundant signaling of scaling list data in the PPS syntax.

Referring to Tables 38 and 39, the pps_scaling_list_data_present_flag syntax element may not be parsed/signaled in the PPS. That is, by removing the pps_scaling_list_data_present_flag syntax element in the PPS, this flag information may be configured not to be parsed/signaled. The pps_scaling_list_data_present_flag syntax element is flag information indicating whether or not the syntax structure of the scaling list data exists in the PPS, and the scaling list data specified by the PPS may be parsed/signaled according to this flag information. However, by removing the pps_scaling_list_data_present_flag syntax element, the scaling list data may not be directly signaled at the PPS level.

As described above, by removing the flag information indicating whether scaling list data is present at the SPS or PPS level, the SPS syntax and the PPS syntax can be configured so that the scaling list data syntax is not directly signaled at the SPS or PPS level. Only the scaling list availability flag (scaling_list_enabled_flag) is explicitly signaled in the SPS, and thereafter, the scaling list (scaling_list_data()) can be parsed individually in the lower level syntax (e.g., APS) based on the availability flag (scaling_list_enabled_flag) in the SPS. Therefore, according to one embodiment of this document, the scaling list data can be parsed/signaled according to a hierarchical structure, thereby improving coding efficiency.

In addition, when signaling scaling list data in the APS, the APS ID can be signaled in the slice header, the APS can be identified based on the APS ID obtained from the slice header, and the scaling list data can be parsed/signaled from the identified APS. Here, the slice header is described only as one example, and the slice header can be mixed with or substituted for various headers such as a tile group header or a picture header.

For example, Table 40 below shows an example of slice header syntax including an APS ID syntax element to signal scaling list data in the APS.

The semantics of the syntax elements included in the slice header syntax in Table 40 can be expressed as shown in Table 41 below.

Referring to Tables 40 and 41, the slice_pic_parameter_set_id syntax element can be parsed/signaled in the slice header. The slice_pic_parameter_set_id syntax element can represent an identifier for the PPS in use. That is, the slice_pic_parameter_set_id syntax element is information for identifying the PPS referenced in the slice, and can represent the value of pps_pic_parameter_set_id. The value of slice_pic_parameter_set_id should be in the range of 0 to 63. The slice_pic_parameter_set_id syntax element can be said to be PPS identification information or PPSID information referenced in the slice.

In addition, a slice_scaling_list_present_flag syntax element may be parsed/signaled in the slice header. The slice_scaling_list_present_flag syntax element may be information indicating whether a scaling list matrix exists for the current slice. For example, if the value of slice_scaling_list_present_flag is 1, it may indicate that a scaling list matrix exists for the current slice, and if the value of slice_scaling_list_present_flag is 0, it may indicate that default scaling list data is used to derive a scaling factor (ScalingFactor) array. Alternatively, if slice_scaling_list_present_flag is not present in the slice header, its value may be inferred to 0.

In this case, the slice_scaling_list_present_flag syntax element may be parsed or not based on the scaling_list_enabled_flag syntax element signaled in the higher level syntax (i.e., SPS). For example, if the value of the scaling_list_enabled_flag signaled in the SPS is 1 (i.e., if it is determined that scaling list data is available at the higher level), the slice_scaling_list_present_flag can be parsed in the slice header to determine whether to perform a scaling process using the scaling list in the slice.

In addition, a slice_scaling_list_aps_id syntax element can be parsed/signaled in the slice header. The slice_scaling_list_aps_id syntax element can represent an identifier for an APS referenced in the slice. That is, the slice_scaling_list_aps_id syntax element can represent ID information (adaptation_parameter_set_id) of an APS containing scaling list data referenced in the slice. On the other hand, the TemporalId (i.e., Temporal ID) of an APS NAL unit (i.e., an APS NAL unit containing scaling list data) having the same APS ID information (adaptation_parameter_set_id) as slice_scaling_list_aps_id must be smaller than or equal to the TemporalId (i.e., Temporal ID) of the slice NAL unit being coded.

In addition, whether the slice_scaling_list_aps_id syntax element can be parsed can be determined based on the slice_scaling_list_present_flag syntax element. For example, if the value of slice_scaling_list_present_flag is 1 (i.e., if a scaling list is present in the slice header), slice_scaling_list_aps_id can be parsed. Then, scaling list data can be obtained from the APS indicated by the parsed slice_scaling_list_aps_id.

In addition, when multiple SCALING DATA APS (multiple APS including scaling list data) with the same APS ID information (adaptation_parameter_set_id) value are referenced by two or more slices in the same picture, the multiple SCALING DATA APS with the same APS ID information (adaptation_parameter_set_id) value must contain the same content.

In Tables 40 and 41 above, it has been described that the APS ID is signaled in the slice header, but this is just one example, and in this document, the APS ID can be signaled in the picture header or tile group header, etc.

This document also proposes a general method for repositioning scaling list data in other header sets. As an embodiment, a general structure is proposed that includes scaling list data in a header set. Currently, for VVC, APS is used as the appropriate header set, but it may also be possible to encapsulate scaling list data identified by Nal Unit Type (NUT) in its own header set. This can be realized as shown in Tables 42 and 43 below.

For example, the following Table 42 shows an example of NAL unit types and the corresponding RBSP syntax structures. As described above, the NAL unit type of a NAL unit can be identified by the RBSP data structure included in the NAL unit, and information regarding such a NAL unit type can be stored and signaled in the NAL unit header.

As shown in Table 42, the scaling list data can be defined as one NAL unit type (e.g., SCALING_NUT), and a specific value (e.g., 21, or one of the reserved values not specified in the NAL unit type) can be specified as the value of the NAL unit type for SCALING_NUT. SCALING_NUT can be a type for a NAL unit that includes a scaling list data parameter set (e.g., Scaling_list_data_parameter set).

Additionally, the availability of SCALING_NUT may be determined at a higher level than APS, PPS, and/or other appropriate headers, or may be determined at a lower level than other NAL unit types.

For example, the following Table 43 shows the syntax of a scaling list data parameter set used to signal scaling list data.

The semantics of the syntax elements included in the scaling list data parameter set syntax in Table 43 can be expressed as shown in Table 44 below.

Referring to Tables 43 and 44, the scaling list data parameter set (e.g., scaling_list_data_parameter_set) can be a header set identified by the NAL unit type value for SCALING_NUT (e.g., 21). In the scaling list data parameter set, the scaling_list_data_parameter_set_id syntax element can be parsed/signaled. The scaling_list_data_parameter_set_id syntax element provides an identifier for the scaling list data for reference to other syntax elements. That is, the scaling list parameter set can be identified based on the scaling_list_data_parameter_set_id syntax element. The scaling_list_data_parameter_set_id syntax element may be referred to as scaling list data parameter set ID information. Scaling list data parameter sets may be shared between pictures and may be different for other slices within a picture.

Scaling list data (e.g., scaling_list_data syntax) can be parsed/signaled from the scaling list data parameter set identified by scaling_list_data_parameter_set_id.

In addition, a scaling_list_data_extension_flag syntax element can be parsed/signaled in the scaling list data parameter set. The scaling_list_data_extension_flag syntax element can indicate whether the scaling list data extension flag (scaling_list_data_extension_flag) syntax element is present in the scaling list data RBSP syntax structure. For example, if the value of scaling_list_data_extension_flag is 1, it can indicate that the scaling list data extension flag (scaling_list_data_extension_flag) syntax element is present in the scaling list data RBSP syntax structure. Alternatively, if the value of scaling_list_data_extension_flag is 0, this may indicate that the scaling list data extension flag (scaling_list_data_extension_flag) syntax element is not present in the scaling list data RBSP syntax structure.

The scaling_list_data_extension_data_flag syntax element may be parsed/signaled based on the scaling_list_data_extension_flag syntax element. The scaling_list_data_extension_data_flag syntax element may be referred to as an extension data flag for the scaling list data. For example, if the value of scaling_list_data_extension_flag is 1, scaling_list_data_extension_data_flag may be parsed/signaled, and in this case, scaling_list_data_extension_data_flag may have any value.

As described above, according to one embodiment of this document, one header set structure for scaling list data can be defined and used, and the header set for the scaling list data can be specified for a NAL unit type (e.g., SCALING_NUT). In this case, the header set for the scaling list data can be defined as a scaling list data parameter set (e.g., scaling_list_data_parameter_set), from which the scaling list data can be obtained.

Meanwhile, this document proposes a method for effectively coding the syntax element scaling_lsit_pred_matrix_id_delta included in the scaling list data.

Currently, in the case of VVC, the scaling_lsit_pred_matrix_id_delta syntax element is coded using an unsigned integer 0-th order Exp-Golomb-coded syntax element with the left bit first. However, in order to improve coding efficiency, in one embodiment of this document, a method is proposed in which the scaling_lsit_pred_matrix_id_delta syntax element having a range of 0 to 5 is coded using a fixed length code (e.g., u(3)) as shown in Table 45 below. In this case, it may be sufficient to code using only 3 bits to improve efficiency for the entire range.

For example, the following Table 45 shows an example of a scaling list data syntax structure.

The semantics of the syntax elements included in the scaling list data syntax in Table 45 can be expressed as shown in Table 46 below.

As shown in Tables 45 and 46, the scaling_list_pred_matrix_id_delta syntax element can be parsed/signaled from the scaling list data syntax. The scaling_list_pred_matrix_id_delta syntax element can represent a reference scaling list used to derive the scaling list. In this case, when parsing scaling_list_pred_matrix_id_delta, a fixed length code (e.g., u(3)) can be used for parsing.

On the other hand, when signaling scaling list data in APS, restrictions can be placed on the scaling list matrices. In response to this, this document proposes a method for limiting the number of scaling list matrices. The method proposed in this document has the effect of making implementation easier and limiting worst case memory requirements.

In one embodiment, the number of APSs that contain scaling list data (i.e., APSs that signal scaling list data syntax) can be limited. To this end, the following constraint can be added: This constraint is for placing holders, i.e., different values can be used.

For ease of explanation, an APS that includes scaling list data (i.e., an APS that signals scaling list data syntax) may be referred to as a SCALING LIST APS. In other words, as described above, if the APS parameter type (e.g., aps_params_type) transmitted from the APS is a type that represents scaling list data parameters (e.g., SCALING_APS), the scaling list data transmitted from the APS may be referred to as a SCALING LIST APS.

For example, the total number of APS containing scaling list data (i.e., SCALING LIST APS) may be less than 3. Of course, other suitable values may be used. For example, a suitable value within the range of 0 to 7 may be used. That is, the total number of APS containing scaling list data (i.e., SCALING LIST APS) may be set to a range of 0 to 7.

Also, for example, APS that contain scaling list data (i.e., SCALING LIST APS) are only allowed one SCALING LIST APS per picture.

The following table 47 shows an example of syntax elements and their corresponding semantics that represent constraints for limiting an APS that includes the scaling list data described above.

Referring to Table 47, the number of APSs (i.e., SCALING LIST APSs) that include scaling list data can be limited based on a syntax element (e.g., slice_scaling_list_aps_id) that represents the APS identification information (i.e., APS ID information) of the SCALING LIST APS.

For example, the syntax element slice_scaling_list_aps_id may represent APS identification information (i.e., APS ID information) of the SCALING LIST APS referenced by the slice. In this case, the value of the syntax element slice_scaling_list_aps_id may be limited to a specific value. As an example, the value of the syntax element slice_scaling_list_aps_id may be limited to a range of 0 to 3. This is one example, and the value may be limited to have other values, and as another example, the value of the syntax element slice_scaling_list_aps_id may be limited to a range of 0 to 7.

Furthermore, the TemporalId (i.e., Temporal ID) of a SCALING LIST APS NAL unit having an APS ID (adaptation_parameter_set_id) such as slice_lmcs_aps_id must be less than or equal to the TemporalId (i.e., Temporal ID) of the coded slice NAL unit.

Also, for example, when multiple SCALING LIST APS with the same APS ID (adaptation_parameter_set_id) value are referenced by two or more slices on the same picture, the multiple SCALING LIST APS with the same APS ID (adaptation_parameter_set_id) value must have the same content.

Also, for example, only one SCALING LIST APS with the same value of APS ID (adaptation_parameter_set_id) and the same contents must be referenced by one or more slices on the same picture. In other words, one or more slices in the same picture must reference the same APS that contains scaling list data.

The following drawings have been created to illustrate a specific example of the present document. The names of specific devices and specific terms and names (e.g., names of syntax/syntax elements, etc.) shown in the drawings are presented for illustrative purposes only, and the technical features of the present document are not limited to the specific names used in the following drawings.

Figures 7 and 8 show a schematic diagram of an example video/image encoding method and associated components according to an embodiment (etc.) of this document.

The method disclosed in FIG. 7 may be performed by the encoding device 200 disclosed in FIG. 2. Specifically, step S700 in FIG. 7 may be performed by the subtraction unit 231 disclosed in FIG. 2, step S710 in FIG. 7 may be performed by the conversion unit 232 disclosed in FIG. 2, steps S720 to S730 in FIG. 7 may be performed by the quantization unit 233 disclosed in FIG. 2, and step S740 in FIG. 7 may be performed by the entropy encoding unit 240 disclosed in FIG. 2. In addition, the method disclosed in FIG. 7 may be performed including the embodiments described above in this document. Therefore, in FIG. 7, detailed descriptions of contents that overlap with the above-described embodiments are omitted or simplified.

As shown in FIG. 7, the encoding device can derive a residual sample for the current block (S700).

In one embodiment, the encoding device may first determine a prediction mode for a current block and derive a prediction sample. For example, the encoding device may determine whether to perform inter prediction or intra prediction for the current block, and may determine a specific inter prediction mode or a specific intra prediction mode based on the RD cost. The encoding device may perform prediction according to the determined prediction mode to derive a prediction sample for the current block. In this case, various prediction methods disclosed in this document, such as inter prediction or intra prediction, may be applied. In addition, the encoding device may generate and encode information (e.g., prediction mode information) related to the prediction applied to the current block. Then, the encoding device may derive a residual sample by comparing an original sample and a predicted sample for the current block.

The encoding device can derive transform coefficients based on the residual samples (S710).

In one embodiment, the encoding apparatus may derive transform coefficients through a transform process for the residual sample. In this case, the encoding apparatus may determine whether transform is applied to the current block in consideration of coding efficiency. That is, the encoding apparatus may determine whether transform is applied to the residual sample. For example, if transform is not applied to the residual sample, the encoding apparatus may derive the residual sample as a transform coefficient. Alternatively, if transform is applied to the residual sample, the encoding apparatus may perform transform on the residual sample to derive transform coefficients. In this case, the encoding apparatus may generate and encode transform skip flag information based on whether transform is applied to the current block. The transform skip flag information may be information indicating whether transform is applied to the current block or whether transform is skipped.

The encoding device can derive quantized transform coefficients based on the transform coefficients (S720).

In one embodiment, the encoding device may derive quantized transform coefficients by applying a quantization process to the transform coefficients. In this case, the encoding device may apply frequency-specific weighting, which adjusts the quantization strength according to the frequency. In this case, the quantization process may be further performed based on a frequency-specific quantization scale value. The quantization scale value for the frequency-specific weighting may be derived using a scaling matrix. For example, the encoding device/decoding device may use a predefined scaling matrix, or the encoding device may configure and encode frequency-specific quantization scale information for the scaling matrix and signal the same to the decoding device. The frequency-specific quantization scale information may include scaling list data. A (modified) scaling matrix may be derived based on the scaling list data.

The encoding device may also perform an inverse quantization process, similar to the decoding device. In this case, the encoding device may derive a (modified) scaling matrix based on the scaling list data, and apply inverse quantization to the quantized transform coefficients based on the scaling matrix to derive reconstructed transform coefficients. At this time, the reconstructed transform coefficients may differ from the original transform coefficients due to losses in the transform/quantization process.

Here, the scaling matrix can refer to the frequency-based quantization scaling matrix described above, and can be used interchangeably with or in place of the quantization scaling matrix, quantization matrix, scaling matrix, scaling list, etc. for convenience of explanation, and is not limited to the specific names used in this embodiment.

That is, the encoding device can further apply frequency-specific weighting when performing the quantization process, and at this time, can generate scaling list data as information on the scaling matrix. This process has been described in detail using Tables 5 to 17 as examples, so in this embodiment, duplicate content and detailed description will be omitted.

The encoding device can generate residual information based on the quantized transform coefficients (S730).

Here, the residual information is information generated through a transformation and/or quantization procedure and may be information about quantized transformation coefficients, and may include, for example, value information, position information, transformation technique, transformation kernel, quantization parameters, etc. of the quantized transformation coefficients.

In addition, if frequency-specific weighting is further applied to derive the quantized transform coefficients in the quantization process, scaling list data for the quantized transform coefficients may be generated. The scaling list data may include scaling list parameters used to derive the quantized transform coefficients. In this case, the encoding device may generate information related to the scaling list data, for example, may generate an APS including the scaling list data.

The encoding device may encode image information (or video information) (S740). Here, the image information may include the residual information. The image information may also include information related to the prediction used to derive the prediction sample (e.g., prediction mode information). The image information may also include information related to the scaling list data. That is, the image information may include various information derived during the encoding process, and may be encoded including such various information.

In one embodiment, the image information may include various information related to the embodiments (etc.) described above in this document, and may include information disclosed in at least one of Tables 1 to 47 above.

For example, the image information may include an adaptation parameter set (APS). The APS may include APS ID information (APS identification information) and APS type information (APS parameter type information). That is, the APS may be identified based on the APS ID information, and APS parameters corresponding to the type may be included in the APS based on the APS type information. For example, the APS type information may include an ALF type related to an adaptive loop filter (ALF) parameter, an LMCS type related to a luma mapping with chroma scaling (LMCS) parameter, and a scaling list type related to a scaling list data parameter. It may be represented as in Table 35 above. For example, if the value of the APS type information is 2, the APS type information may indicate that the APS includes a scaling list data parameter.

As an example, the APS may be configured as shown in Table 33 (or Table 18) above. The APS ID information (APS identification information) may be the adaptation_parameter_set_id described in Tables 33 and 34 (or Tables 18 and 19). The APS type information may be the aps_params_type described in Tables 33 to 35 (or Tables 18 to 20). For example, if the type information of the APS parameter (e.g., aps_params_type) is a SCALING_APS type indicating that the APS includes scaling list data (or if the value of the type information of the APS parameter (e.g., aps_params_type) is 2), the APS may include scaling list data (e.g., scaling_list_data()). That is, the encoding device can signal scaling list data (e.g., scaling_list_data()) via the APS based on SCALING_APS type information indicating that the APS includes scaling list data. That is, scaling list data can be included in the APS based on the APS type information (SCALING_APS type information). As described above, the scaling list data can include scaling list parameters for deriving a scaling list/scaling matrix/scale factor used in the quantization/dequantization process. In other words, the scaling list data can include syntax elements used to configure a scaling list.

Also, as an example, the APS ID information may have a value within a specific range. For example, the value of the APS ID information may have a value within a specific range from 0 to 3 or from 0 to 7. However, this is an example, and the value range of the APS ID information may have other values. Also, the value range of the APS ID information may be determined based on the APS type information (e.g., aps_params_type). For example, for APS type information (e.g., SCALING_APS type) that indicates that the APS is related to scaling list data, the value of the APS ID information may be expressed based on a syntax element (e.g., slice_scaling_list_aps_id) as shown in Table 47 above. For example, if the APS type information (e.g., aps_params_type) indicates that the APS includes scaling list data (e.g., SCALING_APS type), the value of the APS ID information may have a value within a range from 0 to 3 or from 0 to 7. Here, slices in one picture can refer to the same APS for the scaling list data. Alternatively, if the APS type information (e.g., aps_params_type) indicates that the APS is related to ALF (e.g., ALF_APS type), the value of the APS ID information can have a value in the range from 0 to 7. Alternatively, if the APS type information (e.g., aps_params_type) indicates that the APS is related to LMCS (e.g., LMCS_APS type), the value of the APS ID information can have a value in the range from 0 to 3.

Also, for example, the image information may include a Sequence Parameter Set (SPS). The SPS may include first availability flag information indicating whether the scaling list data is available. As an example, the SPS may be configured as shown in Table 36 (or Table 21) above, and the first availability flag information may be the scaling_list_enabled_flag described in Tables 36 and 37 (or Tables 21 and 22). Also, the SPS may prevent this flag information from being parsed/signaled by removing the sps_scaling_list_data_present_flag syntax element. The sps_scaling_list_data_present_flag syntax element is flag information indicating whether or not the syntax structure of the scaling list data exists in the SPS, and the scaling list data specified by the SPS can be parsed/signaled according to this flag information. However, by removing the sps_scaling_list_data_present_flag syntax element, the SPS level can be configured to explicitly signal only the scaling list available flag (scaling_list_enabled_flag) without directly signaling the scaling list data.

Also, for example, the image information may include a PPS (Picture Parameter Set). As an example, the PPS may be configured as shown in Table 38 above, and in this case, the PPS may be configured not to include availability flag information indicating whether the scaling list data is available. That is, the PPS may be configured not to parse/signal this flag information by removing the pps_scaling_list_data_present_flag syntax element. The pps_scaling_list_data_present_flag syntax element is flag information indicating whether the syntax structure of the scaling list data is present in the PPS, and the scaling list data specified by the PPS may be parsed/signaled according to this flag information. However, by removing the pps_scaling_list_data_present_flag syntax element, the scaling list data may not be directly signaled at the PPS level.

In this case, the first availability flag information (e.g., scaling_list_enabled_flag) indicating the availability of the scaling list data may be signaled in the SPS and not signaled in the PPS. Therefore, based on the first availability flag information signaled in the SPS (e.g., when the value of the first availability flag information (e.g., scaling_list_enabled_flag) is 1 or true), the scaling list data included in the APS may be obtained.

Also, for example, the image information may include header information. The header information may be header information related to a slice or picture including the current block, and may include, for example, a picture header or a slice header. The header information may include scaling list data related APS identification information. The scaling list data related APS identification information included in the header information may represent APS ID information for an APS including scaling list data. As an example, the scaling list data related APS ID information included in the header information may be slice_scaling_list_aps_id described in Tables 40 to 41 (or Tables 25 to 28), and may be identification information for an APS (including scaling list data; i.e., SCALING LIST APS) referenced by a slice/picture including the current block. That is, an APS including scaling list data may be identified based on the scaling list data related APS ID information (e.g., slice_scaling_list_aps_id) of the header information.

In this case, whether the header information parses/signals the APS ID information for the APS including the scaling list data can be determined based on the first availability flag information (scaling_list_enabled_flag) parsed/signaled in the SPS. For example, based on the first availability flag information indicating that the scaling list in the SPS is available (e.g., when the value of the first availability flag information (e.g., scaling_list_enabled_flag) is 1 or true), the header information can include the APS ID information for the APS including the scaling list data.

For example, the header information may also include second availability flag information indicating whether scaling list data is available in a picture or slice. As an example, the second availability flag information may be slice_scaling_list_present_flag (or slice_scaling_list_enabled_flag) described in Tables 40 and 41 (or Tables 25 to 28).

In this case, whether the header information parses/signals the second availability flag information can be determined based on the first availability flag information (scaling_list_enabled_flag) parsed/signaled in the SPS. For example, based on the first availability flag information indicating that the scaling list in the SPS is available (e.g., when the value of the first availability flag information (e.g., scaling_list_enabled_flag) is 1 or true), the header information can include the second availability flag information. And based on the second availability flag information (e.g., when the value of the second availability flag information (e.g., slice_scaling_list_present_flag) is 1 or true), the header information can include scaling list data related APS ID information.

As an example, the encoding device may signal second availability flag information (e.g., slice_scaling_list_present_flag) via header information based on the first availability flag information (e.g., scaling_list_enabled_flag) signaled in the SPS as shown in Table 40 above, and then signal APS ID information (e.g., slice_scaling_list_aps_id) for the APS including the scaling list data via header information based on the second availability flag information (e.g., slice_scaling_list_present_flag). Then, the encoding device may signal scaling list data from the APS indicated by the signaled APS ID information (e.g., slice_scaling_list_aps_id).

As described above, the SPS syntax and the PPS syntax can be configured so that the scaling list data syntax is not directly signaled at the SPS or PPS level. For example, only the scaling list availability flag (scaling_list_enabled_flag) can be explicitly signaled in the SPS, and the scaling list (scaling_list_data()) can then be parsed individually in a lower level syntax (e.g., APS) based on the availability flag (scaling_list_enabled_flag) in the SPS. Therefore, according to one embodiment of this document, the scaling list data can be parsed/signaled according to a hierarchical structure, thereby improving coding efficiency.

Image information including the various information as described above can be encoded and output in the form of a bitstream. The bitstream can be transmitted to a decoding device via a network or a (digital) storage medium. Here, the network can include a broadcasting network and/or a communication network, and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc.

Figures 9 and 10 show a schematic diagram of an example of a video/image decoding method and associated components according to an embodiment (etc.) of this document.

The method disclosed in FIG. 9 may be performed by the decoding device 300 disclosed in FIG. 3. Specifically, steps S900 to S910 in FIG. 9 may be performed by the entropy decoding unit 310 disclosed in FIG. 3, step S920 in FIG. 9 may be performed by the inverse quantization unit 321 disclosed in FIG. 3, step S930 in FIG. 9 may be performed by the inverse transform unit 321 disclosed in FIG. 3, and step S940 in FIG. 9 may be performed by the addition unit 340 disclosed in FIG. 3. In addition, the method disclosed in FIG. 9 may be performed including the embodiments described above in this document. Therefore, in FIG. 9, detailed descriptions of contents that overlap with the above-described embodiments are omitted or simplified.

As shown in FIG. 9, the decoding device can receive image information (or video information) from a bitstream (S900).

In one embodiment, the decoding device may derive information (e.g., video/image information) required for image restoration (or picture restoration) by parsing the bitstream. In this case, the image information may include residual information, and the residual information may include information such as value information of quantized transform coefficients, position information, transform technique, transform kernel, and quantization parameter. The image information may also include information related to prediction (e.g., prediction mode information). The image information may also include information related to scaling list data. That is, the image information may include various information required in the decoding process, and may be decoded based on a coding method such as exponential-Golomb coding, CAVLC, or CABAC.

In one embodiment, the image information may include various information related to the embodiments (etc.) described above in this document, and may include information disclosed in at least one of Tables 1 to 47 above.

For example, the image information may include an adaptation parameter set (APS). The APS may include APS ID information (APS identification information) and APS type information (APS parameter type information). That is, the APS may be identified based on the APS ID information, and APS parameters corresponding to the type may be included in the APS based on the APS type information. For example, the APS type information may include an ALF type related to an adaptive loop filter (ALF) parameter, an LMCS type related to a luma mapping with chroma scaling (LMCS) parameter, and a scaling list type related to a scaling list data parameter. It may be represented as in Table 35 above. For example, if the value of the APS type information is 2, the APS type information may indicate that the APS includes a scaling list data parameter.

As an example, the APS may be configured as shown in Table 33 (or Table 18) above. The APS ID information (APS identification information) may be the adaptation_parameter_set_id described in Tables 33 and 34 (or Tables 18 and 19). The APS type information may be the aps_params_type described in Tables 33 to 35 (or Tables 18 to 20). For example, if the type information of the APS parameter (e.g., aps_params_type) is a SCALING_APS type indicating that the APS includes scaling list data (or if the value of the type information of the APS parameter (e.g., aps_params_type) is 2), the APS may include scaling list data (e.g., scaling_list_data()). That is, the decoding device can obtain and parse the scaling list data (e.g., scaling_list_data()) through the APS based on the SCALING_APS type information indicating that the APS includes scaling list data. That is, the APS can include scaling list data based on the APS type information (SCALING_APS type information). As described above, the scaling list data can include scaling list parameters for deriving a scaling list/scaling matrix/scale factor used in the quantization/dequantization process. In other words, the scaling list data can include syntax elements used to configure a scaling list.

Also, as an example, the APS ID information may have a value within a specific range. For example, the value of the APS ID information may have a value within a specific range from 0 to 3 or from 0 to 7. However, this is only an example, and the value range of the APS ID information may have other values. Also, the value range of the APS ID information may be determined based on the APS type information (e.g., aps_params_type). For example, for APS type information (e.g., SCALING_APS type) that indicates that the APS is related to scaling list data, the value of the APS ID information may be expressed based on a syntax element (e.g., slice_scaling_list_aps_id) as shown in Table 47 above. For example, if the APS type information (e.g., aps_params_type) indicates that the APS includes scaling list data (e.g., SCALING_APS type), the value of the APS ID information may have a value within a range from 0 to 3 or from 0 to 7. Here, slices in one picture can refer to the same APS for the scaling list data. Alternatively, if the APS type information (e.g., aps_params_type) indicates that the APS is related to ALF (e.g., ALF_APS type), the value of the APS ID information can have a value in the range from 0 to 7. Alternatively, if the APS type information (e.g., aps_params_type) indicates that the APS is related to LMCS (e.g., LMCS_APS type), the value of the APS ID information can have a value in the range from 0 to 3.

Also, for example, the image information may include a Sequence Parameter Set (SPS). The SPS may include first availability flag information indicating whether the scaling list data is available. As an example, the SPS may be configured as shown in Table 36 (or Table 21) above, and the first availability flag information may be the scaling_list_enabled_flag described in Tables 36 and 37 (or Tables 21 and 22). Also, the SPS may prevent this flag information from being parsed/signaled by removing the sps_scaling_list_data_present_flag syntax element. The sps_scaling_list_data_present_flag syntax element is flag information indicating whether or not the syntax structure of the scaling list data exists in the SPS, and the scaling list data specified by the SPS can be parsed/signaled according to this flag information. However, by removing the sps_scaling_list_data_present_flag syntax element, the SPS level can be configured to explicitly signal only the scaling list available flag (scaling_list_enabled_flag) without directly signaling the scaling list data.

Also, for example, the image information may include a PPS (Picture Parameter Set). As an example, the PPS may be configured as shown in Table 38 above, and in this case, the PPS may be configured not to include availability flag information indicating whether the scaling list data is available. That is, by removing the pps_scaling_list_data_present_flag syntax element in the PPS, this flag information may be configured not to be parsed/signaled. The pps_scaling_list_data_present_flag syntax element is flag information indicating whether the syntax structure of the scaling list data is present in the PPS, and the scaling list data specified by the PPS may be parsed/signaled according to this flag information. However, by removing the pps_scaling_list_data_present_flag syntax element, the scaling list data may not be directly signaled at the PPS level.

In this case, the first availability flag information (e.g., scaling_list_enabled_flag) indicating the availability of the scaling list data may be signaled in the SPS and not signaled in the PPS. Therefore, based on the first availability flag information signaled in the SPS (e.g., when the value of the first availability flag information (e.g., scaling_list_enabled_flag) is 1 or true), the scaling list data included in the APS may be obtained.

Also, for example, the image information may include header information. The header information may be header information related to a slice or picture including the current block, and may include, for example, a picture header or a slice header. The header information may include scaling list data related APS identification information. The scaling list data related APS identification information included in the header information may represent APS ID information for an APS including scaling list data. As an example, the scaling list data related APS ID information included in the header information may be slice_scaling_list_aps_id described in Tables 40 to 41 (or Tables 25 to 28), and may be identification information for an APS (including scaling list data; i.e., SCALING LIST APS) referenced by a slice/picture including the current block. That is, an APS including scaling list data may be identified based on the scaling list data related APS ID information (e.g., slice_scaling_list_aps_id) of the header information. That is, the decoding device can identify the APS based on the APS ID information (e.g., slice_scaling_list_aps_id) in the header information and obtain the scaling list data from the APS.

In this case, whether the header information parses/signals the APS ID information for the APS including the scaling list data can be determined based on the first availability flag information (scaling_list_enabled_flag) parsed/signaled in the SPS. For example, based on the first availability flag information indicating that the scaling list in the SPS is available (e.g., when the value of the first availability flag information (e.g., scaling_list_enabled_flag) is 1 or true), the header information can include the APS ID information for the APS including the scaling list data.

For example, the header information may also include second availability flag information indicating whether scaling list data is available in a picture or slice. As an example, the second availability flag information may be slice_scaling_list_present_flag (or slice_scaling_list_enabled_flag) described in Tables 40 and 41 (or Tables 25 to 28).

In this case, whether the header information parses/signals the second availability flag information can be determined based on the first availability flag information (scaling_list_enabled_flag) parsed/signaled in the SPS. For example, based on the first availability flag information indicating that the scaling list in the SPS is available (e.g., when the value of the first availability flag information (e.g., scaling_list_enabled_flag) is 1 or true), the header information can include the second availability flag information. And, based on the second availability flag information (e.g., when the value of the second availability flag information (e.g., slice_scaling_list_present_flag) is 1 or true), the header information can include APS ID information for the APS including the scaling list data.

As an example, the decoding device can acquire second availability flag information (e.g., slice_scaling_list_present_flag) via header information based on the first availability flag information (e.g., scaling_list_enabled_flag) signaled in the SPS, as shown in Table 40 (or Table 25) above, and then acquire APS ID information (e.g., slice_scaling_list_aps_id) for the APS including the scaling list data via header information based on the second availability flag information (e.g., slice_scaling_list_present_flag). Then, the decoding device can acquire scaling list data from the APS indicated by the APS ID information (e.g., slice_scaling_list_aps_id) acquired via the header information.

As described above, the SPS syntax and the PPS syntax can be configured so that the scaling list data syntax is not directly signaled at the SPS or PPS level. For example, only the scaling list availability flag (scaling_list_enabled_flag) can be explicitly signaled in the SPS, and the scaling list (scaling_list_data()) can then be parsed individually in a lower level syntax (e.g., APS) based on the availability flag (scaling_list_enabled_flag) in the SPS. Therefore, according to one embodiment of this document, the scaling list data can be parsed/signaled according to a hierarchical structure, thereby improving coding efficiency.

The decoding device can derive quantized transform coefficients for the current block based on the residual information (S910).

In one embodiment, the decoding device may obtain residual information included in the image information. As described above, the residual information may include information such as value information of the quantized transform coefficients, position information, a transform technique, a transform kernel, and a quantization parameter. The decoding device may derive quantized transform coefficients for the current block based on the quantized transform coefficient information included in the residual information.

The decoding device can derive the transform coefficients based on the quantized transform coefficients (S920).

In one embodiment, the decoding device may derive transform coefficients by applying an inverse quantization process to the quantized transform coefficients. In this case, the decoding device may apply frequency-specific weighting, which adjusts the quantization strength according to the frequency. In this case, the inverse quantization process may be further performed based on the frequency-specific quantization scale value. The quantization scale value for the frequency-specific weighting may be derived using a scaling matrix. For example, the decoding device may use a predefined scaling matrix, or may use frequency-specific quantization scale information for the scaling matrix signaled from the encoding device. The frequency-specific quantization scale information may include scaling list data. A (modified) scaling matrix may be derived based on the scaling list data.

That is, the decoding device can further apply frequency-specific weighting when performing the inverse quantization process. In this case, the decoding device can derive the transform coefficients by applying the inverse quantization process to the quantized transform coefficients based on the scaling list data.

In one embodiment, the decoding device can obtain the APS included in the image information, and can obtain the scaling list data based on the APS type information included in the APS. For example, the decoding device can obtain the scaling list data included in the APS based on SCALING_APS type information indicating that the APS includes scaling list data. In this case, the decoding device can derive a scaling matrix based on the scaling list data, derive a scaling factor based on the scaling matrix, and derive a transform coefficient by applying inverse quantization based on the scaling factor. The process of performing scaling based on such scaling list data has been described in detail using Tables 5 to 17 as examples, so duplicate content and detailed description will be omitted in this embodiment.

The decoding device may also determine whether to apply frequency-specific weighting in the inverse quantization process (i.e., whether to derive transform coefficients using a (frequency-based quantization) scaling list in the inverse quantization process). For example, the decoding device may determine whether to use scaling list data based on a first availability flag obtained from an SPS included in the image information and/or second availability flag information obtained from header information included in the image information. If it is determined to use scaling list data based on the first availability flag and/or second availability flag information, the decoding device may obtain APS ID information for an APS including scaling list data via header information, identify the APS based on the APS ID information, and obtain scaling list data from the identified APS.

The decoding device can derive residual samples based on the transform coefficients (S930).

In one embodiment, the decoding device may perform an inverse transform on transform coefficients for a current block to derive a residual sample for the current block. In this case, the decoding device may obtain information indicating whether to apply an inverse transform to the current block (i.e., transform skip flag information), and may derive a residual sample based on this information (i.e., transform skip flag information).

For example, if an inverse transform is not applied to the transform coefficients (if the value of the transform skip flag information for the current block is 1), the decoding device may derive the transform coefficients from the residual samples of the current block. Alternatively, if an inverse transform is applied to the transform coefficients (if the value of the transform skip flag information for the current block is 0), the decoding device may perform an inverse transform on the transform coefficients to derive the residual samples of the current block.

The decoding device can generate reconstructed samples based on the residual samples (S940).

In one embodiment, the decoding device may determine whether to perform inter prediction or intra prediction for the current block based on prediction information (e.g., prediction mode information) included in the image information, and may perform prediction based on the determination to derive a prediction sample for the current block. The decoding device may then generate a reconstructed sample based on the prediction sample and the residual sample. In this case, the decoding device may directly use the prediction sample as a reconstructed sample depending on the prediction mode, or may generate a reconstructed sample by adding the residual sample to the prediction sample. In addition, the decoding device may derive a reconstructed block or a reconstructed picture based on the reconstructed sample. As described above, the decoding device may then apply an in-loop filtering procedure, such as a deblocking filtering and/or an SAO procedure, to the reconstructed picture to improve subjective/objective image quality as necessary.

In the above-described embodiments, the methods are described with flow charts as a series of steps or blocks, however, the embodiments herein are not limited to the order of steps, and certain steps may occur in a different order or simultaneously with different steps than those described above. Additionally, one of ordinary skill in the art will appreciate that the steps shown in the flow charts are not exclusive, and other steps may be included, or one or more steps of the flow charts may be deleted without affecting the scope of this document.

The method according to the present document described above can be realized in software form, and the encoding device and/or decoding device according to the present document can be included in a device that performs image processing, such as a TV, a computer, a smartphone, a set-top box, or a display device.

When the embodiments in this document are implemented in software, the methods described above can be implemented with modules (processes, functions, etc.) that perform the functions described above. The modules can be stored in a memory and executed by a processor. The memory can be internal or external to the processor and can be coupled to the processor in various well-known ways. The processor can include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and/or data processing devices. The memory can include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. That is, the embodiments described in this document can be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each drawing can be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for implementation can be stored on a digital storage medium.

In addition, the decoding device and encoding device to which this document applies may be included in multimedia broadcast transmitting/receiving devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video interactive devices, real-time communication devices such as video communication, mobile streaming devices, storage media, camcorders, custom video (VoD) service providing devices, over the top video (OTT) devices, Internet streaming service providing devices, three-dimensional (3D) video devices, virtual reality (VR) devices, argumente reality (AR) devices, image telephone video devices, transportation terminals (e.g., vehicles (including autonomous vehicles), airplane terminals, ship terminals, etc.), and medical video devices, and may be used to process video signals or data signals. For example, OTT video (over the top video) devices can include game consoles, Blu-ray players, Internet-connected TVs, home theater systems, smartphones, tablet PCs, DVRs (Digital Video Recorders), and more.

In addition, the processing method to which this document is applied can be produced in the form of a program executed by a computer and can be stored in a computer-readable recording medium. In addition, multimedia data having a data structure according to this document can also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium can include, for example, Blu-ray Disc (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. In addition, the computer-readable recording medium includes media realized in the form of a carrier wave (e.g., transmission via the Internet). In addition, the bit stream generated by the encoding method can be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

Furthermore, the embodiments of this document may be implemented in a computer program product by program code, which may be executed by a computer in accordance with the embodiments of this document. The program code may be stored on a computer readable carrier.

Figure 11 shows an example of a content streaming system to which the embodiments disclosed herein may be applied.

Referring to FIG. 11, the content streaming system applied to the embodiments of this document can broadly include an encoding server, a streaming server, a web server, a media repository, a user device, and a multimedia input device.

The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits it to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, camcorder, etc. generates a bitstream directly, the encoding server can be omitted.

The bitstream may be generated by an encoding method or a bitstream generation method applied to an embodiment of this document, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, and the streaming server transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, and in this case, the control server controls commands/responses between each device in the content streaming system.

The streaming server may receive content from a media repository and/or an encoding server. For example, when content is received from the encoding server, the content may be received in real time. In this case, the streaming server may store the bitstream for a certain period of time to provide a smooth streaming service.

Examples of the user device include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glass, head mounted displays (HMDs), digital TVs, desktop computers, and digital signage).

Each server in the content streaming system can be operated as a distributed server, in which case data received by each server can be processed in a distributed manner.

The claims herein may be combined in various ways. For example, the technical features of the method claims herein may be combined and realized in an apparatus, and the technical features of the apparatus claims herein may be combined and realized in a method. Also, the technical features of the method claims herein and the technical features of the apparatus claims herein may be combined and realized in an apparatus, and the technical features of the method claims herein and the technical features of the apparatus claims herein may be combined and realized in a method.

Claims (3)

An image decoding method performed by a decoding device,
obtaining image information including residual information from a bitstream;
deriving quantized transform coefficients for a current block based on the residual information;
deriving transform coefficients based on the quantized transform coefficients;
deriving residual samples based on the transform coefficients;
generating reconstruction samples based on the residual samples;
Including,
The image information includes an adaptation parameter set (APS) including scaling list data;
The APS includes APS ID information and APS type information,
The APS is identified based on the APS ID information;
the APS includes the scaling list data based on the APS type information;
the APS ID information has a value within a specific range based on the APS type information that identifies the APS as an APS for the scaling list data;
the specific range is predetermined for the APS type information that identifies the APS as an APS for the scaling list data ,
The particular range for the value of the APS ID information includes a range from 0 to 3 . An image encoding method performed by an encoding device, comprising:
deriving a residual sample for a current block;
deriving transform coefficients based on the residual samples;
applying a quantization process to the transform coefficients to derive quantized transform coefficients;
generating residual information including information on the quantized transform coefficients;
encoding image information including the residual information;
Including,
The image information includes an adaptation parameter set (APS) including scaling list data;
The APS includes APS ID information and APS type information,
The APS is identified based on the APS ID information;
the APS includes the scaling list data based on the APS type information;
the APS ID information has a value within a specific range based on the APS type information that identifies the APS as an APS for the scaling list data;
the specific range is predetermined for the APS type information that identifies the APS as an APS for the scaling list data ,
The method of claim 1 , wherein the particular range for the value of the APS ID information comprises a range from 0 to 3 . A method for transmitting data for an image, comprising:
obtaining a bitstream for the image;
transmitting the data including the bitstream;
Including,
the bitstream is generated by deriving residual samples for a current block, deriving transform coefficients based on the residual samples, deriving quantized transform coefficients by applying a quantization process to the transform coefficients, generating residual information including information on the quantized transform coefficients, and encoding image information including the residual information;
The image information includes an adaptation parameter set (APS) including scaling list data;
The APS includes APS ID information and APS type information,
The APS is identified based on the APS ID information;
the APS includes the scaling list data based on the APS type information;
the APS ID information has a value within a specific range based on the APS type information that identifies the APS as an APS for the scaling list data;
the specific range is predetermined for the APS type information that identifies the APS as an APS for the scaling list data ,
The particular range for the value of the APS ID information includes a range from 0 to 3 .

Images