RU2781172C1 - Encoding video or images based on brightness display - Google Patents

Abstract

FIELD: video technology.

SUBSTANCE: invention relates to means for encoding video. A sample of the brightness of the prediction is generated. A displayed prediction brightness sample based on the prediction brightness sample is generated. Information related to the display of brightness with chromaticity scaling (LMCS) is generated, based on the displayed prediction brightness sample. A residual brightness sample based on the displayed prediction brightness sample is generated. The residual information is output based on the residual brightness sample. Image information is encoded, including LMCS-related information and residual information. Additionally, a reconstructed brightness sample is generated based on the predicted brightness sample. The index of the piecewise function is derived based on the LMCS-related information. A modified restored brightness sample is generated based on the restored brightness sample and the piecewise function index.

EFFECT: increase in the coding efficiency.

15 cl, 18 dwg, 37 tbl

Description

disclosure area

[1] This document relates to video or image coding based on luminance mapping.

Related technical field

[2] Recently, the demand for high-definition, high-quality images/videos such as 4K or 8K or higher ultra-high-definition (UHD) images/videos or higher has increased in various fields. Since the image/video data has high resolution and high quality, the amount of information or bits to be transmitted increases with respect to the existing image/video data, and thus the transmission of image data using media such as an existing wired/wireless broadband line or an existing storage media, or storing image/video data using existing storage media increases transmission costs and storage costs.

[3] In addition, the interest and demand for immersive media such as virtual reality (VR) and artificial reality (AR) content or holograms has recently increased, and the broadcast of images/videos having characteristics different from reality images has increased. such as game images.

[4] Accordingly, a highly efficient image/video compression technology is required to efficiently compress, transmit, store, and reproduce high-definition, high-quality image/video information having various characteristics as described above.

[5] In addition, a luminance chrominance scaling (LMCS) mapping process is performed to improve compression efficiency and to improve subjective/objective visual quality, and reduction of computational complexity in the LMCS process is discussed.

Brief description of the essence of the invention

[6] According to an embodiment of the present document, a method and apparatus for improving image encoding efficiency are provided.

[7] According to an embodiment of the present document, a method and apparatus for effectively applying filtering is provided.

[8] In accordance with an embodiment of the present document, a method and apparatus for effectively applying LMCS is provided.

[9] In accordance with an embodiment of the present document, LMCS codewords (or their range) may be limited.

[10] According to an embodiment of the present document, a single residual chrominance scaling factor directly signaled in the LMCS chrominance scaling may be used.

[11] According to an embodiment of the present document, a linear mapping (linear LMCS) may be used.

[12] In accordance with an embodiment of the present document, information about the turning points required for a linear display may be explicitly signaled.

[13] According to an embodiment of the present document, a flexible number of bins can be used to display luminance.

[14] According to an embodiment of the present document, the process of deriving an index for luma inverse mapping and/or residual chroma scaling can be simplified.

[15] According to an embodiment of the present document, a video/image decoding method performed by a decoding apparatus is provided.

[16] According to an embodiment of the present document, a decoding apparatus for performing video/image decoding is provided.

[17] According to an embodiment of the present document, a video/image encoding method performed by an encoding apparatus is provided.

[18] According to an embodiment of the present document, an encoding device for performing video/image encoding is provided.

[19] According to one embodiment of the present document, a computer-readable digital storage medium is provided in which encoded video/image information generated in accordance with the video/image coding method disclosed in at least one embodiment of the present document is stored.

[20] According to an embodiment of the present document, there is provided a computer-readable digital storage medium in which encoded information or encoded video/image information is stored causing a decoding apparatus to perform the video/image decoding method disclosed in at least one embodiment of the present document.

Useful results

[21] According to an embodiment of the present document, the overall image/video compression efficiency can be improved.

[22] According to an embodiment of the present document, subjective/objective visual quality can be improved by efficient filtering.

[23] According to an embodiment of the present document, an LMCS process for image/video coding can be efficiently performed.

[24] According to an embodiment of the present document, it is possible to minimize the resources/costs (software or hardware) required for the LMCS process.

[25] According to an embodiment of the present document, the hardware implementation for the LMCS process can be facilitated.

[26] According to an embodiment of the present document, the division operation required to derive LMCS codewords upon mapping (regeneration) can be eliminated or minimized by limiting the LMCS codewords (or their range).

[27] In accordance with an embodiment of the present document, the delay according to the identification of the chunky index can be eliminated using one residual chrominance scaling factor.

[28] According to an embodiment of the present document, the residual chrominance scaling process can be performed without depending on (recovering) the luminance block using linear mapping in the LMCS, and thus the delay in scaling can be eliminated.

[29] According to an embodiment of the present document, display performance in an LMCS can be improved.

[30] According to an embodiment of the present document, the complexity of the LMCS can be reduced by simplifying the demapping process of the demapping index for luma demapping and/or residual chrominance scaling, and thus video/image coding efficiency can be improved.

Brief description of the drawings

[31] FIG. 1 illustrates an example of a video/image coding system to which embodiments of the present document may be applied.

[32] FIG. 2 is a diagram schematically illustrating the configuration of a video/image coding apparatus to which embodiments of the present document may be applied.

[33] FIG. 3 is a diagram schematically illustrating the configuration of a video/image decoding apparatus to which embodiments of the present document may be applied.

[34] FIG. 4 shows an exemplary hierarchical structure for an encoded image/video.

[35] FIG. 5 illustrates an exemplary hierarchical structure of CVS in accordance with an embodiment of the present document.

[36] FIG. 6 illustrates an exemplary LMCS structure in accordance with an embodiment of the present document.

[37] FIG. 7 illustrates the structure of an LMCS according to another embodiment of the present document.

[38] FIG. 8 shows a graph representing an exemplary forward display.

[39] FIG. 9 is a flowchart illustrating a method for deriving a residual chrominance scaling index according to an embodiment of the present document.

[40] FIG. 10 illustrates a linear approximation of turning points in accordance with an embodiment of the present document.

[41] FIG. 11 illustrates one example of a linear reshaping (or linear reshaping, linear mapping) according to an embodiment of the present document.

[42] FIG. 12 shows an example of linear forward mapping in an embodiment of the present document.

[43] FIG. 13 shows an example of reverse forward mapping in an embodiment of the present document.

[44] FIG. 14 and FIG. 15 schematically show an example of a video/image coding method and related components according to the embodiment(s) of the present document.

[45] FIG. 16 and FIG. 17 schematically show an example of an image/video decoding method and related components according to an embodiment of the present document.

[46] FIG. 18 shows an example of a content streaming system to which the embodiments disclosed herein may be applied.

Description of Exemplary Embodiments

[47] This document may be modified in various forms, and its specific embodiments will be described and shown in the drawings. However, the embodiments are not intended to limit the present document. The terms used in the following description are only used to describe specific embodiments, but are not intended to limit the present document. A singular expression includes a plural expression, unless it is explicitly read otherwise. Terms such as "include" and "have" are intended to indicate that the features, numbers, steps, operations, elements, components, or combinations thereof used in the following description exist, and thus it should be understood that the possibility of the existence or addition of one or more other features, numbers, steps, operations, elements, components, or a combination thereof is not excluded.

[48] Meanwhile, each configuration in the drawings described herein is shown independently for convenience of description regarding different function features, and does not imply that each configuration is implemented as separate hardware or separate software. For example, two or more components from each component may be combined to form one component, or one component may be divided into multiple components. Embodiments where each component is integrated and/or separate are also included within the scope of this disclosure.

[49] Next, examples of the present embodiment will be described in detail with reference to the accompanying drawings. Also, the same reference numerals are used to indicate the same elements throughout the drawings, and the same descriptions of the same elements will be omitted.

[50] FIG. 1 illustrates an example of a video/image coding system to which embodiments of the present document may be applied.

[51] With reference to FIG. 1, the video/image coding system may include a first device (source device) and a second device (receiver device). The source device may transmit the encoded information or video/image data to the receiving device via a digital storage medium or a network as a file or stream.

[52] The source device may include a video source, an encoder, and a transmitter. The receiving device may include a receiver, a decoding device, and a rendering device. The encoding device may be referred to as a video/image encoding device, and the decoding device may be referred to as a video/image decoding device. The transmitter may be included in the encoder. The receiver may be included in the decoding device. The rendering device may include a display, and the display may be configured as a separate device or an external component.

[53] A video source may acquire a video/image through a video/image capture, synthesis, or generation process. The video source may include a video/image capture device and/or a video/image generation device. The video/image capture device may include, for example, one or more cameras, video/image archives including previously captured videos/images, and the like. The video/image generating device may include, for example, computers, tablets, and smartphones, and may (electronically) generate video/images. For example, the virtual video/image may be generated via a computer or the like. In this case, the video/image capture process can be replaced with a related data generation process.

[54] The encoding device may encode the input video/image. The encoder may perform a series of procedures such as prediction, transformation, and quantization for compactness and coding efficiency. Encoded data (encoded video/image information) may be output as a bitstream.

[55] The transmitter may transmit the encoded image/image information or data output as a bitstream to the receiver of the receiving device via a digital storage medium or a network as a file or stream. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, and the like. The transmitter may include an element for generating a media file by a predefined file format and may include an element for transmission over a broadcast/communications network. The receiver may receive/extract the bitstream and transmit the received bitstream to a decoder.

[56] The decoding device can decode the video/image by performing a series of procedures such as dequantization, inverse transformation, and prediction corresponding to the operation of the encoding device.

[57] The rendering device may render the decoded video/image. The rendered video/image can be displayed via the display.

[58] This document relates to video/image coding. For example, the method/embodiment disclosed herein may be applied to the method disclosed in the Versatile Video Coding (VVC) standard, Essential Video Coding (EVC) standard, AOMedia Video 1 (AV1) standard, Audio/Video Coding 2- generation (AVS2) or next generation video/image coding standard (eg H.267, H.268 or the like).

[59] The present document proposes various video/image coding embodiments, and the embodiments above may also be performed in combination with each other unless otherwise specified.

[60] As used herein, video may refer to a sequence of images over time. A picture generally refers to a unit representing a single image in a particular time frame, and a slice/tile refers to a unit constituting a part of a picture in terms of encoding. A slice/tile may include one or more coding tree units (CTUs). One picture may consist of one or more clippings/tiles. One picture can consist of one or more groups of tiles. One group of tiles may include one or more tiles. A "brick" may represent a rectangular area of CTU rows within a tile in a picture. A tile may be broken into multiple bricks, each of which may consist of one or more rows of CTUs within the tile. A tile that does not break into many bricks may also be called a brick. A brick scan may represent a particular sequential order of CTUs breaking a picture, where CTUs may be ordered in a raster scan of CTUs within a brick, and bricks within a tile may be ordered sequentially in a raster scan of tile bricks, and tiles within a picture may be ordered sequentially in raster scanning of the mosaic elements of the picture. A tile is a rectangular area of the CTU within a particular tile column and a particular tile row in the picture. The tile column is a rectangular area of the CTU having a height equal to the height of the picture and a width specified by the syntax elements in the picture parameter set. A tile row is a rectangular area of the CTU having a height specified by the syntax elements in the picture parameter set and a width equal to the width of the picture. A tile scan is a particular sequential ordering of CTUs that tile a picture, in which the CTUs are ordered sequentially in the raster scan CTUs in the tile, and the tiles in the picture are ordered sequentially in the tile raster scan. A slice includes an integer number of picture bricks that can only be contained in one NAL unit. A cut may consist of a number of complete tiles, or only a sequence of complete bricks of a single tile. Herein, the tile group and the cutout may be used interchangeably. For example, herein, a tile group/tile group header may be referred to as a slice/cut header.

[61] Meanwhile, one picture can be divided into two or more sub-pictures. A sub-picture may be a rectangular area of one or more cutouts within a picture.

[62] A pixel or pel can mean the smallest unit that makes up one picture (or image). Also, 'sampling' can be used as a term corresponding to a pixel. The sample may generally represent a pixel or pixel value, and may represent only a luminance component pixel/pixel value, or only a chrominance component pixel/pixel value.

[63] The unit may represent a basic unit of image processing. The unit may include at least one of a particular area of the picture and information related to the area. One unit may include one luminance block and two chrominance blocks (eg, cb, cr). The unit can be used interchangeably with terms such as block or area in some cases. In general, an MxN block may include samples (or arrays of samples) or a set (or array) of transform coefficients M columns and N rows. Alternatively, a sample may mean a pixel value in the spatial domain, and when such a pixel value is converted to the frequency domain, it can mean a transform factor in the frequency domain.

[64] As used herein, "A or B" may mean "only A", "only B", or "both A and B". In other words, "A or B" in this document can be interpreted as "A and/or B". For example, in this document "A, B or C (A, B or C)" means "A only", "B only", "C only", or "any combination of A, B and C".

[65] A slash (/) or a comma used herein can mean "and/or". For example, "A/B" could mean "A and/or B". Accordingly, "A/B" can mean "only A", "only B", or "both A and B". For example, "A, B, C" could mean "A, B, or C".

[66] As used herein, "at least one of A and B" can mean "only A", "only B", or "both A and B". Also, herein, the expression "at least one of A or B" or "at least one of A and/or B" can be interpreted in the same way as "at least one of A and B".

[67] Also, as used herein, "at least one of A, B, and C" means "A only, B only, C only," or "any combination of A, B, and C." Also, "at least one of A, B or C" or "at least one of A, B and/or C" can mean "at least one of A, B and C".

[68] Also, parentheses used herein can mean "for example". Specifically, when "prediction (intra-prediction)" is indicated, "intra-prediction" may be offered as an example of "prediction". In other words, "prediction" herein is not limited to "intra-prediction", and "intra-prediction" may be offered as an example of "prediction". Also, even when "prediction (ie, intra-prediction)" is indicated, "intra-prediction" may be offered as an example of "prediction".

[69] The technical features that are separately described in one drawing herein may be implemented separately or together.

[70] FIG. 2 is a diagram schematically illustrating the configuration of a video/image coding apparatus to which embodiments of the present document may be applied. Further, what is called a video encoding device may include an image encoding device.

[71] With reference to FIG. 2, the encoder 200 includes an image splitter 210, a predictor 220, a residual processor (residual processor) 230, and an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-predictor 221 and an intra- predictor 222. Residual processor 230 may include transform 232, quantizer 233, dequantizer 234, and inverse transform 235. Residual processor 230 may further include subtractor 231. Adder 250 may be referred to as a reconstructor or reconstructed block generator. The image splitter 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be configured by at least one hardware component (eg, encoder chipset or processor) in accordance with an embodiment. In addition, memory 270 may include a decoded picture buffer (DPB) or may be configured with a digital storage medium. The hardware component may further include memory 270 as an internal/external component.

[72] The image splitter 210 may split the input image (or picture or frame) input to the encoder 200 into one or more processors. For example, a processor may be referred to as a coding unit (CU). In this case, a coding unit may be recursively partitioned according to a quadtree-binary-tree-ternary tree (QTBTTT) structure from a coding tree unit (CTU) or a largest coding unit (LCU). For example, one coding unit may be split into multiple coding units of greater depth based on a quadtree structure, a binary tree structure, and/or a ternary tree. In this case, for example, a quadtree structure may be applied first, and a binary tree structure and/or a ternary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. An encoding procedure according to the present disclosure may be performed based on a final coding unit that is no longer split. In this case, the largest coding unit may be used as the final coding unit based on the coding efficiency according to the picture characteristics, or if necessary, the coding unit may be recursively split into coding units of greater depth, and the coding unit having the optimal size may be used as the final coding unit. Here, the encoding procedure may include a prediction, transformation, and recovery procedure to be described later. As an example, the processor may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may be separated or split from the aforementioned final coding unit. The prediction unit may be a sample prediction unit, and the transform unit may be a unit for deriving a transform coefficient and/or a unit for deriving a residual signal from a transform coefficient.

[73] The unit can be used interchangeably with terms such as block or area in some cases. In general, an M×N block may represent a set of samples or transform coefficients consisting of M columns and N rows. The sample may generally represent a pixel or pixel value, may represent only a luminance component pixel/pixel value, or represent only a chrominance component pixel/pixel value. Sampling can be used as a term corresponding to a single picture (or image) for a pixel or pel.

[74] In the encoder 200, the prediction signal (predicted block, prediction sample array) output from the inter predictor 221 or intra predictor 222 is subtracted from the input image signal (original block, original sample array) to generate a residual signal ( residual block, residual sample array), and the generated residual signal is transmitted to the converter 232. In this case, as shown, a module for subtracting a prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) in encoder 200 may be referred to as a subtractor 231. The predictor may perform prediction on a block to be processed (hereinafter referred to as the current block) and generate a predicted block including the prediction samples for the current block. The predictor may determine whether intra-prediction or inter-prediction is applied on the current block or on a CU basis. As described below in the description of each prediction mode, the predictor may generate various prediction-related information such as prediction mode information and pass the generated information to the entropy encoder 240. The prediction information may be encoded in the entropy encoder 240 and output in the form of a bitstream.

[75] Intra-predictor 222 may predict the current block by referring to samples in the current picture. These samples may be adjacent to the current block or may be spaced according to the prediction mode. In intra-prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The non-directional mode may include, for example, a DC mode and a planar mode. The directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes according to the granularity of the prediction direction. However, this is just an example, more or less directional prediction modes may be used depending on the settings. The intra-predictor 222 may determine the prediction mode applied to the current block by using the prediction mode applied to the adjacent block.

[76] The inter predictor 221 may output a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector in the reference picture. Here, in order to reduce the amount of motion information transmitted in the inter-prediction mode, motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between a neighboring block and a current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-prediction, the neighbor block may include a spatial neighbor block represented in the current picture and a temporal neighbor block represented in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighbor block may be the same or different. The temporal neighbor block may be referred to as a co-located reference block, co-located CU (colCU) or the like, and a reference picture including the temporal neighbor block may be referred to as a co-located picture (colPic). For example, the inter predictor 221 may configure a list of motion information candidates based on adjacent blocks and generate information indicating which candidate is used to derive a motion vector and/or a reference picture index of the current block. Inter-prediction may be performed based on various prediction modes. For example, in the case of the skip mode and the combine mode, the inter predictor 221 may use the motion information of the neighboring block as the motion information of the current block. In the skip mode, unlike the combine mode, the residual signal may not be transmitted. In the case of a motion vector prediction (MVP) mode, the motion vector of an adjacent block may be used as a motion vector predictor, and the motion vector of the current block may be indicated by motion vector difference signaling.

[77] Predictor 220 may generate a prediction signal based on various prediction methods described below. For example, the predictor may not only apply intra-prediction or inter-prediction to predict one block, but also apply both intra-prediction and inter-prediction at the same time. This may be referred to as combined inter- and intra-prediction (CIIP). In addition, the predictor may be based on an intra-block copy (IBC) prediction mode or a palette mode for block prediction. An IBC prediction mode or a palette mode may be used for encoding image/video game content or the like, such as screen content coding (SCC). IBC basically performs prediction on the current picture, but can be performed similarly to inter-prediction in that the reference block is output on the current picture. That is, the IBC may use at least one of the inter prediction methods described in this disclosure. The palette mode can be considered as an example of intra-coding or intra-prediction. When a palette mode is applied, the sample value within the picture can be signaled based on information about the palette table and palette index.

[78] A prediction signal generated by a predictor (including inter predictor 221 and/or intra predictor 222) may be used to generate a recovered signal or to generate a residual signal. Transformer 232 may generate transform coefficients by applying a transform method to the residual signal. For example, the transform method may include at least one of discrete cosine transform (DCT), discrete sine transform (DST), Karhunen-Loeve transform (KLT), graph-based transform (GBT), or conditionally non-linear transform (CNT) . Here, GBT means a transformation obtained from a graph when relationship information between pixels is represented by a graph. CNT refers to a transform generated based on a prediction signal generated using all previously reconstructed pixels. Also, the transformation process may be applied to square blocks of pixels having the same size, or may be applied to blocks having a variable size other than square.

[79] The quantizer 233 may quantize the transform coefficients and pass them to the entropy encoder 240, and the entropy encoder 240 may encode the quantized signal (quantized transform coefficient information) and output a bitstream. Information about the quantized transform coefficients may be referred to as residual information. The quantizer 233 may reorder the quantized block-type transform coefficients into a one-dimensional vector form based on the scan order of the coefficients, and generate information about the quantized transform coefficients based on the quantized transform coefficients in the form of a one-dimensional vector. Information about the transform coefficients may be generated. Entropy encoder 240 may perform various coding techniques such as exponential Golomb coding, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), and the like, for example. Entropy encoder 240 may encode information needed for video/image reconstruction other than quantized transform coefficients (eg, syntax element values, etc.) together or separately. Encoded information (eg, encoded video/image information) may be transmitted or stored in NAL (Network Abstraction Layer) units in the form of a bitstream. The video/image information may further include information about various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/image information may further include general restriction information. In the present disclosure, information and/or syntax elements transmitted/signaled from an encoder to a decoder may be included in video/picture information. The video/image information may be encoded by the above-described encoding procedure and included in the bitstream. The bitstream may be transmitted over a network or may be stored in a digital storage medium. The network may include a broadcast network and/or a communications network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, and the like. A transmitter (not shown), a transmitting signal outputted from the entropy encoder 240, and/or a storage module (not shown) storing the signal may be included as an internal/external element of the encoder 200, and alternatively, a transmitter may be included in the entropy encoder. 240.

[80] The quantized transform coefficients output from quantizer 233 may be used to generate a prediction signal. For example, the residual signal (residual block or residual samples) may be recovered by applying dequantization and inverse transform to the quantized transform coefficients by dequantizer 234 and inverse transform 235. Adder 250 adds the recovered residual signal to the prediction signal output from inter-predictor 221 or intra-predictor 222 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). If there is no remainder for the block to be processed, such as in the case where the skip mode is applied, the predicted block may be used as the reconstructed block. Adder 250 may be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal may be used to intra-predict the next block to be processed in the current picture, and may be used to inter-predict the next picture through filtering as described below.

[81] Meanwhile, luma mapping with chrominance scaling (LMCS) may be applied during encoding and/or picture reconstruction.

[82] Filter 260 can improve subjective/objective image quality by applying filtering to the reconstructed signal. For example, filter 260 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture and store the modified reconstructed picture in memory 270, specifically DPB memory 270. Various filtering methods may include, for example, deblocking filtering, adaptive sampling bias, adaptive loop filter, bi-directional filter and the like. The filter 260 may generate various filtering-related information and pass the generated information to the entropy encoder 240, as described later in the description of each filtering method. The filtering related information may be encoded by entropy encoder 240 and output in the form of a bitstream.

[83] The modified reconstructed picture transmitted to the memory 270 can be used as a reference picture in the inter predictor 221. When inter prediction is applied by the encoder, prediction mismatch between the encoder 200 and the decoder 300 can be avoided, and encoding efficiency can be improved. .

[84] The DPB of the DPB memory 270 may store a modified reconstructed picture for use as a reference picture in the inter predictor 221. The memory 270 may store block motion information from which motion information in the current picture is derived (or encoded) and/or motion information blocks in the picture that have already been restored. The stored motion information may be transmitted to the inter predictor 221 and used as spatial neighbor block motion information or temporal neighbor block motion information. Memory 270 may store reconstructed samples of the reconstructed blocks in the current picture and may transfer the reconstructed samples to intra-predictor 222.

[85] FIG. 3 is a schematic diagram illustrating the configuration of a video/image decoding apparatus to which the embodiment(s) of the present disclosure may be applied.

[86] With reference to FIG. 3, decoder 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, a memory 360. Predictor 330 may include an inter predictor 331 and an intra predictor 332. Residual processor 320 may include includes a dequantizer 321 and an inverse transform 321. Entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 may be configured by a hardware component (eg, decoder chipset or processor) according to an embodiment. In addition, memory 360 may include a decoded picture buffer (DPB) or may be configured with a digital storage medium. The hardware component may further include memory 360 as an internal/external component.

[87] When a bitstream including video/image information is input, the decoding apparatus 300 can reconstruct an image corresponding to a process in which the video/image information is processed in the encoding apparatus of FIG. 2. For example, the decoding apparatus 300 may derive ones/blocks based on information related to the block split obtained from the bitstream. The decoder 300 may perform decoding using a processor used in the encoder. Thus, the decoding processor may be a coding unit, for example, and the coding unit may be partitioned according to a quadtree structure, a binary tree structure, and/or a ternary tree structure from the coding tree unit or the largest coding unit. One or more transformation units may be derived from a coding unit. The reconstructed image signal decoded and output by the decoding apparatus 300 can be reproduced by the playback apparatus.

[88] The decoder 300 may receive a signal output from the encoder of FIG. 2 in the form of a bitstream, and the received signal may be decoded by the entropy decoder 310. For example, the entropy decoder 310 may perform parsing of the bitstream to derive information (eg, video/image information) necessary for image reconstruction (or picture restoration). The video/image information may further include information about various parameter sets such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/image information may further include general restriction information. The decoding device may further decode the picture based on the parameter set information and/or general constraint information. The signaled/received information and/or syntax elements described later in this disclosure may be decoded to decode the decoding procedure and obtained from the bitstream. For example, entropy decoder 310 encodes information in a bitstream based on an encoding method such as exponential Golomb coding, CAVLC, or CABAC, and outputs syntax elements required for image reconstruction and quantized transform coefficient values for the remainder. More specifically, the CABAC entropy decoding method can receive a bin corresponding to each syntax element in the bitstream, determine the context model using the decoding information of the target syntax element, the decoding information for decoding the target block, or the character/bin information decoded in the previous step, and perform arithmetic decoding on the bin by predicting the occurrence probability of the bin according to the determined context model, and generating a symbol corresponding to the value of each syntax element. In this case, the CABAC entropy decoding method may update the context model using the decoded symbol/bin information for the context model of the next symbol/bin after determining the context model. Prediction related information, among the information decoded by the entropy decoder 310, may be provided on a predictor (inter predictor 332 and intra predictor 331) and a residual value on which entropy decoding was performed in entropy decoder 310, i.e., quantized coefficients transformations and associated parameter information may be input to the residual processor 320 . Residual processor 320 may output a residual signal (residual block, residual samples, residual array of samples). In addition, filtering information among the information decoded by the entropy decoder 310 may be provided to the filter 350. Meanwhile, a receiver (not shown) for receiving a signal outputted from the encoder may be further configured as an internal/external element of the decoding apparatus 300 , or the receiver may be a component of the entropy decoder 310. Meanwhile, the decoding apparatus according to the present disclosure may be referred to as a video/image/picture decoding apparatus, and the decoding apparatus may be classified into an information decoder (video/image/picture information decoder) and a sample decoder. (video/image/picture sampling decoder). The information decoder may include an entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverse transform 322, an adder 340, a filter 350, a memory 360, an inter predictor 332, and an intra predictor 331.

[89] The dequantizer 321 may dequantize the quantized transform coefficients and output the transform coefficients. The dequantizer 321 may reorder the quantized transform coefficients in the form of a two-dimensional block. In this case, the reordering may be performed based on the coefficient scan order performed in the encoder. The dequantizer 321 may perform dequantization on the quantized transform coefficients using a quantization parameter (eg, quantization step size information) and obtain the transform coefficients.

[90] The inverse transform 322 inversely transforms the transform coefficients to obtain a residual signal (residual block, residual array of samples).

[91] The predictor may perform prediction on the current block and generate a predicted block including the prediction samples for the current block. The predictor may determine whether intra-prediction or inter-prediction is applied to the current block based on the prediction information output from the entropy decoder 310, and may determine a specific intra/inter prediction mode.

[92] Predictor 320 may generate a prediction signal based on various prediction methods described below. For example, the predictor may not only apply intra-prediction or inter-prediction to predict one block, but also apply intra-prediction and inter-prediction at the same time. This may be referred to as combined inter- and intra-prediction (CIIP). In addition, the predictor may be based on an intra-block copy (IBC) prediction mode or a palette mode for block prediction. An IBC prediction mode or a palette mode may be used for encoding image/video game content or the like, such as screen content coding (SCC). IBC basically performs prediction on the current picture, but can be performed similarly to inter-prediction in that the reference block is output on the current picture. That is, the IBC may use at least one of the inter prediction methods described in this disclosure. The palette mode can be considered as an example of intra-coding or intra-prediction. When a palette mode is applied, the sample value within the picture can be signaled based on information about the palette table and palette index.

[93] Intra-predictor 331 may predict the current block by referring to samples in the current picture. These samples may be adjacent to the current block or may be spaced according to the prediction mode. In intra-prediction, the prediction modes may include a plurality of non-directional modes and a plurality of directional modes. The intra predictor 331 may determine the prediction mode applied to the current block using the prediction mode applied to the adjacent block.

[94] The inter predictor 332 may output a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector in the reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the adjacent block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-prediction, the neighbor block may include a spatial neighbor block represented in the current picture and a temporal neighbor block represented in the reference picture. For example, the inter predictor 332 may configure a list of motion information candidates based on adjacent blocks, and derive the current block's motion vector and/or reference picture index based on the received candidate selection information. Inter prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter prediction mode for the current block.

[95] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed array of samples) by adding the obtained residual signal to the prediction signal (predicted block, predicted array of samples) output from the predictor (including the inter-predictor 332 and/ or intra-predictor 331). If there is no remainder for the block to be processed, for example, when the skip mode is applied, the predicted block may be used as the reconstructed block.

[96] The adder 340 may be referred to as a reconstructor or a reconstructed block generator. The generated reconstructed signal may be used to intra-predict the next block to be processed in the current picture, may be output by filtering as described below, or may be used to inter-predict the next picture.

[97] Meanwhile, luminance chrominance scaling (LMCS) mapping may be applied in a picture decoding process.

[98] The filter 350 can improve subjective/objective image quality by applying filtering to the reconstructed signal. For example, filter 350 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture, and store the modified reconstructed picture in memory 360, specifically, DPB memory 360. Various filtering methods may include, for example, deblocking filtering, adaptive sampling bias, adaptive loop filter, bidirectional filter, and the like.

[99] The (modified) reconstructed picture stored in the DPB memory 360 may be used as a reference picture in the inter predictor 332. The memory 360 may store the motion information of the block from which the motion information in the current picture is derived (or decoded), and/or information about the movement of blocks in the picture that have already been restored. The stored motion information may be transmitted to the inter predictor 260 for use as spatial neighbor block motion information or temporal neighbor block motion information. The memory 360 may store the reconstructed samples of the reconstructed blocks in the current picture and transfer the reconstructed samples to the intra-predictor 331.

[100] Herein, the embodiments described in the filter 260, inter-predictor 221, and intra-predictor 222 of the encoder 200 may be the same or respectively applied to correspond to the filter 350, inter-predictor 332, and intra-predictor 331 devices 300 decoding. The same may also apply to module 332 and intra-predictor 331.

[101] As described above, in video coding, prediction is performed to improve compression efficiency. Through this, it is possible to generate a predicted block including the prediction samples for the current block, which is the block to be encoded. Here, the predicted block includes prediction samples in the spatial domain (or pixel domain). The predicted block is output equally from the encoder and the decoder, and the encoder decodes the residual information (residual information) between the original block and the predicted block, rather than the original sample value of the original block itself. By signaling to the device, the image encoding efficiency can be improved. The decoding apparatus can output a residual block including the residual samples based on the residual information, and generate a reconstructed block including the reconstructed samples by summing the residual block and the predicted block, and generate a reconstructed picture including the reconstructed blocks.

[102] Residual information may be generated in the transformation and quantization processes. For example, the coding device may output a residual block between the original block and the predicted block, and perform a transform process on the residual samples (residual array of samples) included in the residual block to derive transform coefficients, and then, by performing a quantization process on the transform coefficients, output the quantized transform coefficients to signal the information related residual to the decoder (via the bitstream). Here, the residual information may include location information, a transform method, a transform kernel and a quantization parameter, value information of quantized transform coefficients, and so on. The decoder may perform a dequantization/inverse transform process based on the residual information and output residual samples (or residual blocks). The decoder may generate a reconstructed picture based on the predicted block and the residual block. The encoder may also dequantize/inversely transform the quantized transform coefficients for a later picture inter-prediction reference to derive a residual block and generate a reconstructed picture based on it.

[103] Herein, at least one of quantization/dequantization and/or transformation/inverse transformation may be omitted. When quantization/dequantization is omitted, the quantized transform coefficient may be referred to as a transform coefficient. When the transform/inverse transform is omitted, the transform coefficients may be referred to as coefficients or residual coefficients, or may still be referred to as transform coefficients for uniformity of expression.

[104] Herein, the quantized transform coefficient and the transform coefficient may be referred to as the transform coefficient and the scaled transform coefficient, respectively. In this case, the residual information may include information on the transform coefficient(s), and the information on the transform coefficient(s) may be signaled through the residual coding syntax. The transform coefficients may be derived based on the residual information (or information on the transform coefficient(s)), and the scaled transform coefficients may be output via an inverse transform (scaling) on the transform coefficients. Residual samples may be derived based on an inverse transform (transform) of the scaled transform coefficients. This may be applied/expressed in other parts of this document as well.

[105] Intra-prediction may refer to a prediction that generates prediction samples for a current block based on reference samples in a picture to which the current block (hereinafter referred to as the current picture) belongs. When intra-prediction is applied to the current block, adjacent reference samples to be used for intra-prediction of the current block may be output. Adjacent reference samples of the current block may include samples adjacent to the left boundary of the current block having size nW×nH and a total of 2×nH samples adjacent to the bottom-left, samples adjacent to the upper boundary of the current block and a total of 2×nW upper-right adjacent samples and one upper-left adjacent sample of the current block. Alternatively, the adjacent reference samples of the current block may include a plurality of upper neighbor samples and a plurality of left neighbor samples. Also, adjacent reference samples of the current block may include a total of nH samples adjacent to the right edge of the current block having size nW×nH, a total of nW samples adjacent to the bottom edge of the current block, and one sample adjacent to the bottom-right of current block.

[106] However, some of the neighboring reference samples of the current block may not yet be decoded or available. In this case, the decoder may configure adjacent reference samples to use for prediction by replacing unavailable samples with available samples. Alternatively, adjacent reference samples to be used for prediction may be configured through interpolation of the available samples.

[107] When neighboring reference samples are derived, (i) a prediction sample may be derived based on an average or an interpolation of neighboring reference samples of the current block, and (ii) a prediction sample may be derived based on a reference sample presented in a particular direction (prediction) for the sample predictions among peripheral reference samples of the current block. Case (i) may be referred to as non-directional mode or non-angular mode, and case (ii) may be referred to as directional mode or angular mode.

[108] Moreover, a prediction sample may also be generated by interpolating between a second neighboring sample and a first neighboring sample located in a direction opposite to the intra-prediction mode prediction direction of the current block based on the prediction sample of the current block among the neighboring reference samples. The case above may be referred to as linear interpolation intra-prediction (LIP). Also, chrominance prediction samples may be generated based on luma samples using a linear model. This case may be referred to as the LM mode.

[109] In addition, the prediction temporal sample of the current block may be derived based on the filtered neighboring reference samples, and at least one reference sample derived in accordance with the intra-prediction mode among the existing neighboring reference samples, that is, the unfiltered neighboring reference samples. , and the temporal prediction sample may be weighted summed to output the prediction sample of the current block. The case above may be referred to as position-dependent intra-prediction (PDPC).

[110] In addition, a reference sample line having the highest prediction accuracy among neighboring lines of the plurality of reference samples of the current block may be selected to output a prediction sample using a reference sample located in the prediction direction on the corresponding line, and then the reference sample line used here may be indicated (signaled) to a decoder, thereby performing intra-prediction encoding. The case above may be referred to as multi-reference line (MRL) intra-prediction or MRL-based intra-prediction.

[111] In addition, intra-prediction can be performed based on the same intra-prediction mode by dividing the current block into vertical or horizontal subdivisions, and adjacent reference samples can be output and used in the subdivision unit. That is, in this case, the intra-prediction mode for the current block is equally applied to the sub-divisions, and the intra-prediction performance can be improved in some cases by deriving and using adjacent reference samples in the subdivision unit. Such a prediction method may be referred to as intra-subdivision (ISP) or ISP-based intra-prediction.

[112] The above-described intra-prediction methods may be referred to as an intra-prediction type apart from an intra-prediction mode. The intra-prediction type may be referred to by various terms such as intra-prediction method or additional intra-prediction mode. For example, the intra prediction type (or additional intra prediction mode) may include at least one of LIP, PDPC, MRL, and ISP described above. The general intra-prediction method except for a special intra-prediction type such as LIP, PDPC, MRL, or ISP may be referred to as normal intra-prediction type. The normal intra prediction type may generally be applied when the special intra prediction type is not applied, and prediction may be performed based on the intra prediction mode described above. Meanwhile, post-filtering can be performed on the inferred predicted sample as needed.

[113] Specifically, the intra prediction procedure may include an intra prediction mode/type determination step, an adjacent reference sample output step, and a prediction sample output step based on the intra prediction mode/type. In addition, the post-filtering step may be performed on the inferred predicted sample, as needed.

[114] When intra prediction is applied, the intra prediction mode applied to the current block may be determined using the neighboring block's intra prediction mode. For example, the decoder may select one of the most likely mode candidates (mpm) from a list of mpm derived based on the neighbor block intra-prediction mode (e.g., left and/or top neighboring blocks) of the current block based on the received mpm index, and select one from other remaining intra-prediction modes not included in the mpm (and planar) candidates, based on the remaining intra-prediction mode information. The mpm list may be configured to include or not include the planar mode as a candidate. For example, if the mpm list includes the planar mode as a candidate, the mpm list can have six candidates. If the mpm list does not include the planar mode as a candidate, the mpm list can have three candidates. When the mpm list does not include a planar mode as a candidate, a non-planar flag (eg, intra_luma_not_planar_flag) indicating whether the intra-prediction mode of the current block is not a planar mode may be signaled. For example, the mpm flag may be signaled first, and the mpm index and the non-planar flag may be signaled when the value of the mpm flag is 1. Also, the mpm index may be signaled when the value of the non-planar flag is 1. Here, the mpm list is configured, to not include a planar mode as a candidate, the non-planar flag must not be signaled first to check if it is a planar mode, since the planar mode is always treated as mpm.

[115] For example, whether the intra prediction mode applied to the current block is in mpm candidates (and planar mode) or remaining mode may be indicated based on an mpm flag (eg, Intra_luma_mpm_flag). The mpm flag value 1 may indicate that the intra-prediction mode for the current block is in mpm candidates (and planar mode), and the mpm flag value 0 may indicate that the intra-prediction mode for the current block is not in mpm candidates (and planar mode). ). A non-planar flag value of 0 (eg, Intra_luma_not_planar_flag) may indicate that the intra-prediction mode for the current block is a planar mode, and a non-planar flag value of 1 may indicate that the intra-prediction mode for the current block is not a planar mode. The index mpm may be signaled in the form of an mpm_idx or intra_luma_mpm_idx syntax element, and the remaining intra-prediction mode information may be signaled in the form of a rem_intra_luma_pred_mode or intra_luma_mpm_remainder syntax element. For example, the remaining intra prediction mode information may index the remaining intra prediction modes not included in the candidates mpm (and planar mode) among all intra prediction modes in order of the number of prediction modes to indicate one of them. The intra-prediction mode may be an intra-prediction mode for the luma component (sample). Further, the intra prediction mode information may include at least one of an mpm flag (eg, Intra_luma_mpm_flag), a non-planar flag (eg, Intra_luma_not_planar_flag), an mpm index (eg, mpm_idx or intra_luma_mpm_idx), and remaining intra prediction mode information. (rem_intra_luma_pred_mode or intra_luma_mpm_remainder). In this document, the MPM list may be referred to by various terms such as MPM candidate list and candModeList. When MIP is applied to the current block, the individual mpm flag (eg, intra_mip_mpm_flag), the mpm index (eg, intra_mip_mpm_idx), and the remaining intra-prediction mode information (eg, intra_mip_mpm_remainder) for the MIP may be signaled, but the non-planar flag is not signalled.

[116] In other words, in general, when block division is performed on an image, the current block and the adjacent block to be encoded have similar image characteristics. Therefore, the current block and the neighboring block are highly likely to have the same or similar intra-prediction mode. Thus, the encoder can use the intra-prediction mode of the neighboring block to encode the intra-prediction mode of the current block.

[117] For example, the encoder/decoder may configure a list of most likely modes (MPM) for the current block. The MPM List may also be referred to as the MPM Candidate List. Here, MPM may refer to a mode used to improve coding efficiency in view of the similarity between a current block and a neighboring block in intra-prediction mode coding. As described above, the MPM list may be configured to include planar mode, or may be configured to exclude planar mode. For example, when the MPM list includes planar mode, the number of candidates in the MPM list may be 6. And, if the MPM list does not include planar mode, the number of candidates in the MPM list may be 5.

[118] The encoder/decoder may configure an MPM list including 5 or 6 MPMs.

[119] To configure the MPM list, three types of modes can be considered: default intra modes, adjacent intra modes, and derived intra modes.

[120] For adjacent intra modes, two adjacent blocks may be considered, i.e. the left neighbor block and the top neighbor block.

[121] As described above, if the MPM list is configured not to include the planar mode, the planar mode is excluded from the list, and the number of MPM list candidates may be set to 5.

[122] In addition, the non-directional mode (or non-angular mode) among the intra-prediction modes may include a DC mode based on the average of neighboring reference samples of the current block or a planar mode based on interpolation.

[123] When inter prediction is applied, the encoder/decoder predictor may output a prediction sample by performing inter prediction in units of blocks. The inter-prediction may be a prediction output in a manner that depends on the data elements (eg, sample values or motion information) of the picture(s) other than the current picture. When inter-prediction is applied to the current block, a predicted block (prediction sample array) for the current block may be derived based on a reference block (reference sample array) specified by a motion vector in a reference picture indicated by the reference picture index. Here, in order to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information of the current block may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between the neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction type information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter-prediction, the neighbor block may include a spatial neighbor block represented in the current picture and a temporal neighbor block represented in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighbor block may be the same or different. The temporal neighbor block may be referred to as a co-located reference block, co-located CU (colCU) or the like, and a reference picture including the temporal neighbor block may be referred to as a co-located picture (colPic). For example, a motion information candidate list may be configured based on neighboring blocks of the current block, and flag or index information indicating which candidate is selected (in use) may be signaled to output a motion vector and/or a reference picture index of the current block. Inter-prediction may be performed based on various prediction modes. For example, in the case of the skip mode and the join mode, the motion information of the current block may be the same as the motion information of the adjacent block. In the skip mode, unlike the combine mode, the residual signal may not be transmitted. In the case of a motion vector prediction (MVP) mode, the motion vector of a selected neighboring block may be used as a motion vector predictor, and the motion vector of the current block may be signaled. In this case, the motion vector of the current block may be output using the sum of the motion vector predictor and the motion vector difference.

[124] The motion information may include L0 motion information and/or L1 motion information according to the type of inter prediction (L0 prediction, L1 prediction, Bi prediction, etc.). The motion vector in the L0 direction may be referred to as the L0 motion vector or MVL0, and the motion vector in the L1 direction may be referred to as the L1 or MVL1 motion vector. Prediction based on motion vector L0 may be referred to as L0 prediction, prediction based on motion vector L1 may be referred to as L1 prediction, and prediction based on motion vector L0 and motion vector L1 may be referred to as bi-prediction. Here, the motion vector L0 may indicate a motion vector associated with the reference picture list L0 (L0), and the motion vector L1 may indicate a motion vector associated with the reference picture list L1 (L1). The reference picture list L0 may include pictures that are earlier in the output order than the current picture as reference pictures, and the reference picture list L1 may include pictures that are later in the output order than the current picture. Previous pictures may be called forward (reference) pictures, and subsequent pictures may be called reverse (reference) pictures. The reference picture list L0 may further include pictures that are later in the output order than the current picture as reference pictures. In this case, previous pictures may be indexed first in the reference picture list L0, and subsequent pictures may be indexed later. The reference picture list L1 may further include pictures preceding the current picture in output order as reference pictures. In this case, subsequent pictures may be indexed first in reference picture list 1, and previous pictures may be indexed later. The output order may correspond to a picture order counting (POC) order.

[125] FIG. 4 shows an exemplary hierarchical structure for an encoded image/video.

[126] With reference to FIG. 4, the encoded image/video is divided into a video coding layer (VCL) that conducts the image/video decoding process, a subsystem that transmits and stores the encoded information, and a NAL (Network Abstraction Layer) responsible for the function and presented between the VCL and the subsystem.

[127] In the VCL, VCL data including compressed image data (crop data) is generated, or a parameter set including a picture parameter set (PSP), a sequence parameter set (SPS), and a video parameter set (VPS) may be generated. or a Supplemental Optimization Information (SEI) message further required for the picture decoding process.

[128] In NAL, a NAL unit may be generated by adding header information (NAL unit header) to a raw byte sequence (RBSP) payload generated in the VCL. In this case, RBSP refers to slice data, parameter set, SEI message, etc. generated in the VCL. The NAL unit header may include NAL unit type information specified according to the RBSP data included in the corresponding NAL unit.

[129] As shown in the drawing, the NAL unit can be classified into a VCL NAL unit and a non-VCL NAL unit according to the RBSP generated in the VCL. A VCL NAL unit may mean a NAL unit that includes image information (cut data) about an image, and a non-VCL NAL unit may mean a NAL unit that includes information (parameter set or SEI message) required to decode a picture. .

[130] The above-described VCL NAL unit and non-VCL NAL unit can be transmitted over the network by attaching header information in accordance with the subsystem data standard. For example, the NAL unit can be converted to a predefined standard data format such as H.266/VVC file format, Real Time Transport Protocol (RTP), Transport Stream (TS), etc., and transmitted through various networks.

[131] As described above, the NAL unit may be specified by the NAL unit type according to the RBSP data structure included in the corresponding NAL unit, and the NAL unit type information may be stored and signaled in the NAL unit header.

[132] For example, the NAL unit may be classified into a VCL NAL unit type and a non-VCL NAL unit type according to whether the NAL unit includes image information (cut data). A VCL NAL unit type may be classified according to the nature and type of pictures included in a VCL NAL unit, and a non-VCL NAL unit type may be classified according to parameter set types.

[133] The following is an example of a NAL unit type specified according to a parameter set type included in a non-VCL NAL unit type.

[134] - APS NAL Unit (Adaptation Parameter Set): A type for a NAL unit including APS

[135] - DPS NAL Unit (Decoding Parameter Set): A type for a NAL unit including a DPS

[136] - VPS NAL Unit (Video Parameter Set): Type for a NAL unit including a VPS

[137] - SPS NAL Unit (Sequence Parameter Set): A type for a NAL unit including an SPS

[138] - NAL Unit PPS (Picture Parameter Set) NAL: Type for a NAL unit including PPS

[139] - NAL unit PH (picture title): Type for NAL unit including PH

[140] The above NAL unit types may have syntax information for NAL unit types, and the syntax information may be stored and signaled in the NAL unit header. For example, the syntax information may be nal_unit_type, and NAL unit types may be specified by the value of nal_unit_type.

[141] Meanwhile, as described above, one picture may include a plurality of slices, and one slice may include a slice header and slice data. In this case, one picture title may be further added to a plurality of slices (a set of slice header and slice data) in one picture. The picture title (picture title syntax) may include information/parameters normally applied to the picture. In this document, the clipping may be mixed or replaced with a group of tiles. Also, herein, the slice header may be mixed or replaced with a tile group header.

[142] The slice header (slice header syntax) may include information/parameters that may normally be applied to the slice. APS (APS syntax) or PPS (PPS syntax) may include information/parameters that may typically be applied to one or more clippings or pictures. The SPS (SPS Syntax) may include information/parameters that may typically be applied to one or more sequences. VPS (VPS Syntax) may include information/parameters that may typically apply to multiple layers. DPS (DPS Syntax) may include information/parameters that may generally be applied to the entire video. The DPS may include information/parameters related to coded video sequence (CVS) concatenation. High-level syntax (HLS) herein may include at least one of APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, and slice header syntax.

[143] Here, the image/image information encoded by an encoder and signaled to a decoder in the form of a bitstream includes not only information related to splitting in a picture, inter/intra prediction information, residual information, in-loop information, filtering, etc., but also information included in the slice header, information included in the APS, information included in the PPS, information included in the SPS, and/or information included in the VPS.

[144] Meanwhile, in order to compensate for the difference between the original image and the reconstructed image due to an error occurring in a compression encoding process such as quantization, an in-loop filtering process may be performed on the reconstructed samples or reconstructed pictures as described above. As described above, in-loop filtering may be performed by an encoder filter and a decoder filter, and a deblocking filter, SAO, and/or an adaptive loop filter (ALF) may be applied. For example, the ALF process may be executed after the deblocking filtering process and/or the SAO process is completed. However, even in this case, the deblocking filtering process and/or the SAO process may be omitted.

[145] Meanwhile, in order to improve coding efficiency, luminance chrominance scaling (LMCS) mapping may be applied as described above. The LMCS may be referred to as a contour former (reshaping). To improve coding efficiency, LMCS control and/or signaling of LMCS related information may be performed hierarchically.

[146] FIG. 5 illustrates an exemplary hierarchical structure of CVS in accordance with an embodiment of the present document. The encoded video sequence (CVS) may include a sequence parameter set (SPS), a picture parameter set (PPS), a tile group header, tile data, and/or a CTU. Here, a tile group header and tile data may be referred to as a slice header and slice data, respectively.

[147] The SPS may include flags for enabling tools to be used in CVS. In addition, the SPS can be accessed using the PPS, which includes information about the parameters that change for each picture. Each of the encoded pictures may include one or more encoded rectangular area tiles. Tiles can be grouped into raster scans forming groups of tiles. Each tile group is encapsulated with header information called a tile group header. Each tile consists of a CTU containing encoded data. Here, the data may include original sample values, prediction sample values, and its luminance and chrominance components (luminance prediction sample values and chrominance prediction sample values).

[148] FIG. 6 illustrates an exemplary LMCS structure in accordance with an embodiment of the present document. LMCS structure 600 in FIG. 6 includes a luma component in-contour mapping portion 610 based on adaptive piecewise linear (adaptive PWL) models and a luminance dependent residual chrominance scaling portion 620 for chrominance components. The dequantizer and inverse transform 611, reconstruction 612, and intra-prediction 613 blocks of the in-loop mapping portion 610 represent processes applicable in the mapped (reformed) region. The loop filter blocks 615, motion compensation or inter-prediction 617 of the in-peak display portion 610 and the reconstruction blocks 622, intra-prediction 623, motion compensation or inter-prediction 624, loop filters 625 of the residual chrominance scaling portion 620 represent processes applicable in the original (not -mapped, non-reformed) area.

[149] As illustrated in FIG. 6, when LMCS is enabled, at least one of a demapping (regeneration) process 614 , a forward mapping (regeneration) process 618 , and a chrominance scaling process 621 may be applied. For example, the inverse mapping process may be applied to the (restored) luminance sample (or luminance samples or array of luminance samples) in the reconstructed picture. The inverse mapping process may be performed based on a piecewise index (inverse) luminance sampling function. The piecewise index of the (inverse) function may identify the fragment to which the brightness sample belongs. The output of the inverse mapping process is a modified (restored) luminance sample (or modified luma samples or an array of modified luma samples). LMCS can be turned on or off at the level of the tile group (or cutout), picture, or higher.

[150] A direct mapping process and/or a chroma scaling process may be applied to generate a reconstructed picture. A picture may contain luma samples and chrominance samples. The reconstructed luma sampled picture may be referred to as a reconstructed luminance picture, and the reconstructed chrominance sampled picture may be referred to as a reconstructed chrominance picture. The combination of the reconstructed luminance picture and the reconstructed chrominance picture may be referred to as the reconstructed picture. The reconstructed luminance picture may be generated based on a direct mapping process. For example, if inter-prediction is applied to the current block, forward mapping is applied to the luminance prediction sample derived based on the (reconstructed) luminance sample in the reference picture. Since the (restored) luminance sample in the reference picture is generated based on the inverse mapping process, forward mapping can be applied to the luminance prediction sample, thus a mapped (reformed) luminance prediction sample can be output. The direct mapping process may be performed based on the piecewise index of the luminance prediction sampling function. The piecewise function index may be derived based on a luminance prediction sample value or a luminance sample value in a reference picture used for inter-prediction. If intra-prediction (or intra-block copy (IBC)) is applied to the current block, forward mapping is not necessary because the inverse mapping process has not yet been applied to the reconstructed samples in the current picture. The (recovered) luminance sample in the reconstructed luminance picture is generated based on the mapped luminance prediction sample and the corresponding residual luminance sample.

[151] The reconstructed chrominance picture may be generated based on the chrominance scaling process. For example, the (restored) chrominance sample in the reconstructed chrominance picture may be derived based on the prediction chroma sample and the residual chrominance sample (c _res ) in the current block. The residual chroma sample (c _res ) is derived based on the (scaled) residual chroma sample (c _resScale ), and the residual chroma scaling factor (cScaleInv may be referred to as varScale) for the current block. The residual chrominance scaling factor may be calculated based on the values of the regenerated luminance prediction sample for the current block. For example, the scaling factor may be computed based on the average luminance value ave(Y' _pred ) of the luminance values Y' _{pred of the} reshaped luminance prediction sample. For reference, in FIG. 6, the (scaled) chrominance residual sample derived based on the inverse transform/dequantization may be referred to as c _resScale , and the chroma residual sample output by performing the (reverse) scaling process into a (scaled) chroma residual sample may be referred to as c _res .

[152] FIG. 7 illustrates the structure of an LMCS according to another embodiment of the present document. Fig. 7 is described with reference to FIG. 6. Here, the difference between the LMCS structure of FIG. 7 and the LMCS structure 600 of FIG. 6. The in-loop display part and the luminance-dependent residual chroma scaling part in FIG. 7 may operate in the same (similar) way as the in-contour display portion 610 and the luminance-dependent residual chroma scaling portion 620 in FIG. 6.

[153] With reference to FIG. 7, the residual chrominance scaling factor may be derived based on the reconstructed luminance samples. In this case, an average luminance value (avgYr) can be obtained (derived) based on neighboring reconstructed luminance samples outside the reconstructed block rather than internal reconstructed luma samples of the reconstructed block, and a residual chrominance scaling factor is derived based on the average luminance value (avgYr). Here, the adjacent luma recovery samples may be adjacent luminance recovery samples of the current block, or may be adjacent luminance recovery samples of Virtual Pipeline Data Units (VPDUs) including the current block. For example, when intra-prediction is applied to a target block, the reconstructed samples may be derived based on the prediction samples that are derived based on the intra-prediction. In another example, when inter-prediction is applied to a target block, forward mapping is applied to the prediction samples that are derived from the inter-prediction, and reconstructed samples are generated (derived) based on the reshaped (or forward-mapped) luma prediction samples.

[154] The video/image information signaled via the bitstream may include LMCS parameters (LMCS information). The LMCS parameters may be configured as high-level syntax (HLS including slice header syntax) or the like. The detailed description and configuration of the LMCS parameters will be described next. As described above, the syntax tables described herein (and subsequent embodiments) may be configured/encoded on the encoder side and signaled to the decoder side via a bitstream. The decoder may parse/decode the LMCS information (in the form of syntax components) in the syntax tables. One or more embodiments to be described below may be combined. The encoder may encode the current picture based on the LMCS information, and the decoder may decode the current picture based on the LMCS information.

[155] Luminance component in-loop mapping can adjust the dynamic range of the input signal by redistributing the codewords over the dynamic range to improve compression efficiency. For luminance mapping, a forward mapping (regeneration) function (FwdMap) and an inverse mapping (regeneration) function (InvMap) corresponding to the forward mapping (FwdMap) function can be used. The FwdMap function may be signaled using piecewise linear models, for example a piecewise linear model may have 16 tiles or bins. Fragments can be of equal length. In one example, the InvMap function should not be signaled and is instead derived from the FwdMap function. That is, the reverse mapping may be a forward mapping function. For example, the back mapping function can be mathematically constructed as a symmetric forward mapping function, as reflected by the line y=x.

[156] In-loop reshaping (luminance) can be used to map input (sample) luminance values to changed values in the reshaped area. The reshaped values may be encoded and then mapped back to the original (non-mapped, non-reshaped) area after restoration. Chroma residual scaling may be applied to compensate for the interaction between the luminance signal and the chrominance signal. In-loop regeneration is performed by specifying a high-level syntax for the shaper model. The shaper model syntax can signal a piecewise linear model (PWL model). For example, the shaper model syntax may signal a PWL model with 16 bins or equal length chunks. The forward lookup table (FwdLUT) and/or the inverse lookup table (InvLUT) may be derived based on a piecewise linear model. For example, the PWL model precomputes forward (FwdLUT) and inverse (InvLUT) lookup tables (LUTs) with 1024 entries. As an example, when the forward lookup table FwdLUT is derived, the reverse lookup table InvLUT may be derived based on the forward lookup table FwdLUT. The FwdLUT forward lookup table may map input luminance values Yi to modified Yr values, and the reverse InvLUT lookup table may map modified Yr values to reconstructed Y'i values. The reconstructed Y'i values may be derived based on the input brightness values Yi.

[157] In one example, the SPS may include the syntax of Table 1 below. The syntax of Table 1 may include sps_reshaper_enabled_flag as a flag enabling tool. Here, sps_reshaper_enabled_flag can be used to specify whether the shaper is used in a coded video sequence (CVS). That is, sps_reshaper_enabled_flag may be a flag to enable reshaper in the SPS. In one example, the syntax of Table 1 may be part of the SPS.

[158] [Table 1]

[159] In one example, the semantics of the sps_seq_parameter_set_id and sps_reshaper_enabled_flag syntax elements may be as shown in Table 2 below.

[160] [Table 2]

[161] In one example, a tile group header or slice header may include the syntax of Table 3 or Table 4 below.

[162] [Table 3]

[163] [Table 4]

[164] The semantics of the syntax elements included in the syntax of Table 3 or Table 4 may include, for example, the content disclosed in the following tables.

[165] [Table 5]

[166] [Table 6]

[167] As one example, when a flag enabling reshaping (i.e., sps_reshaper_enabled_flag) is parsed in the SPS, a tile group header may parse additional data (i.e., information included in Table 5 or 6) that are used to create lookup tables (FwdLUT and/or InvLUT). To do this, the status of the SPS shaper flag (sps_reshaper_enabled_flag) may first be checked in the slice header or tile group header. When sps_reshaper_enabled_flag is true (or 1), an additional flag, i.e., tile_group_reshaper_model_present_flag (or slice_reshaper_model_present_flag) can be parsed. The purpose of tile_group_reshaper_model_present_flag (or slice_reshaper_model_present_flag) may be to indicate the presence of a reshaper model. For example, when tile_group_reshaper_model_present_flag (or slice_reshaper_model_present_flag) is true (or 1), it may indicate that the shaper is present for the current tile group (or current slice). When tile_group_reshaper_model_present_flag (or slice_reshaper_model_present_flag) is false (or 0), it may indicate that no shaper is present for the current tile group (or current slice).

[168] If a shaper is present and the shaper is enabled in the current tile group (or current slice), the shaper model (ie, tile_group_reshaper_model() or slice_reshaper_model()) may be processed. In addition to this, an additional flag, tile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) may also be parsed. tile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) may indicate whether a reshaper model is used for the current tile group (or slice). For example, if tile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) is 0 (or false), it may be indicated that the reshaper model is not used for the current tile group (or current slice). If tile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) is 1 (or true), it may be indicated that a reshaper model is used for the current tile group (or slice).

[169] As one example, tile_group_reshaper_model_present_flag (or slice_reshaper_model_present_flag) may be true (or 1), and tile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) may be false (or 0). This means that the reshaping model is present but not used in the current tile group (or clipping). In this case, the reshaping model can be used in future tile groups (or cutouts). As an example, tile_group_reshaper_enable_flag may be true (or 1) and tile_group_reshaper_model_present_flag may be false (or 0). In such a case, the decoder uses the shaper from the previous initialization.

[170] When the reshaper model (i.e., tile_group_reshaper_model() or slice_reshaper_model()) and tile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) are parsed, it can be determined (judged) whether conditions necessary for chroma scaling are present. The conditions above include condition 1 (the current tile/cut has not been intra-coded) and/or condition 2 (the current tile/cut has not been split into two separate coding quadtree structures for luma and chrominance, i.e. the block structure for the current tile/cut group is not a double tree structure). If condition 1 and/or condition 2 is true and/or tile_group_reshaper_enable_flag (or slice_reshaper_enable_flag) is true (or 1), then tile_group_reshaper_chroma_residual_scale_flag (or slice_reshaper_chroma_residual_scale_flag) may be parsed. When tile_group_reshaper_chroma_residual_scale_flag (or slice_reshaper_chroma_residual_scale_flag) is enabled (if 1 or true), residual chroma scaling may be indicated to be enabled for the current tile group (or slice). When tile_group_reshaper_chroma_residual_scale_flag (or slice_reshaper_chroma_residual_scale_flag) is off (if 0 or false), residual chroma scaling may be indicated to be disabled for the current tile group (or slice).

[171] The purpose of the tile group regeneration model is to parse the data that will be needed to create lookup tables (LUTs). These LUTs are created based on the idea that the distribution of the allowable range of luminance values can be divided into many bins (eg, 16 bins), which can be represented using a set of 16 PWL equations. Therefore, any luminance value that lies within a given bin can be mapped to the modified luminance value.

[172] FIG. 8 shows a graph representing an exemplary forward display. In FIG. 8, five bins are illustrated as an example.

[173] With reference to FIG. 8, the x-axis represents input luminance values, and the y-axis represents modified output luminance values. The x-axis is divided into 5 bins or slices, each bin has a length L. That is, the five bins mapped to changed brightness values have the same length. A forward lookup table (FwdLUT) can be created using data (ie, shaper data) available from a tile group header, and therefore display can be facilitated.

[174] In one embodiment, output pivot points associated with bin indices can be calculated. The output pivot points may set (mark) the minimum and maximum boundaries of the output range of the luminance codeword regeneration. The process of computing the output turning points may be performed by computing a piecewise cumulative distribution function (CDF) of the number of codewords. The output rotation range may be segmented based on the maximum number of bins to be used and the size of the lookup table (FwdLUT or InvLUT). As one example, the output rotation range may be segmented based on the product of the maximum number of bins and the size of the lookup table (LUT size * maximum number of bin indexes). For example, if the product of the maximum number of bins and the sizes of the lookup table is 1024, the output rotation range can be segmented into 1024 inputs. This rotation output range tooth may be performed (applied or achieved) based on (using) the scaling factor. In one example, the scaling factor may be derived based on Equation 1 below.

[175] [Equation 1]

[176] In Equation 1, SF denotes a scaling factor, and y1 and y2 denote output pivot points corresponding to each bin. Also, FP_PREC and c can be predefined constants. The scaling factor determined based on Equation 1 may be referred to as the scaling factor for direct regeneration.

[177] In another embodiment, with respect to reverse reshaping (reverse mapping), for a specific range of bins to be used (i.e., from reshaper_model_min_bin_idx to reshape_model_max_bin_idx), input reshaped pivots that correspond to the mapped pivots of the forward LUT, and the mapped reverse output pivot points (given by the bin index given * the number of original codewords) are output. In another example, the scaling factor SF may be derived based on Equation 2 below.

[178] [Equation 2]

[179] In Equation 2, SF denotes a scaling factor, x1 and x2 denote input pivots, and y1 and y2 denote output pivots corresponding to each bin (inverse mapping output pivots). Here, input pivots may represent pivots mapped based on a forward lookup table (FwdLUT) and output pivots may represent pivots mapped inversely based on an inverse lookup table (InvLUT). Also, FP_PREC may be a predefined constant value. FP_PREC of Equation 2 may be the same or different from FP_PREC of Equation 1. The scaling factor determined based on Equation 2 may be referred to as the scaling factor for inverse regeneration. In reverse reshaping, the splitting of the input pivot points can be performed based on the scaling factor of Equation 2. The scaling factor SF is used to segment the range of input pivot points. Based on the segmented input pivot points, bin indices ranging from 0 to the minimum bin index (reshaper_model_min_bin_idx) and/or from the minimum bin index (reshaper_model_min_bin_idx) to the maximum bin index (reshape_model_max_bin_idx) are assigned rotation values corresponding to the minimum and maximum bin values.

[180] Table 7 below shows the syntax of the builder model according to the embodiment. The shaper model may be referred to as a regeneration model or an LMCS model. Here, a builder model has been described as a tile group builder by way of example, but the present document is not limited to this embodiment. For example, a builder model may be included in the APS, or a tile group builder model may be referred to as a cutout builder model or LMCS data. Also, the reshaper_model or Rsp prefix can be used interchangeably with lmcs. For example, in the following tables and description below, reshaper_model_min_bin_idx, reshaper_model_delta_max_bin_idx, reshaper_model_max_bin_idx, RspCW, RsepDeltaCW can be used interchangeably with lmcs_min_bin_idx, lmcs_delta_cs_bin_idx, lmx, lmcs_delta_csDcselta_idx, lmx, respectively.

[181] [Table 7]

[182] The semantics of the syntax elements included in the syntax of Table 7 may include, for example, the content disclosed in the following table.

[183] [Table 8]

[184]

[185] The demapping process for luminance sampling according to the present document may be described in the form of a standard document as shown in the Table below.

[186] [Table 9]

[187] The process identification of a piecewise function index for luminance sampling according to the present document may be described in the form of a standard document as shown in the Table below. In Table 10, idxYInv may be referred to as a demap index, and the demap index may be derived based on the recovered luma samples (lumaSample).

[188] [Table 10]

[189] Luminance display can be performed based on the above-described embodiments and examples, and the above-described syntax and components included therein can only be exemplary representations, and the embodiments herein are not limited to the above tables or equations. The following describes a method for performing residual chrominance scaling (scaling for chrominance components of residual samples) based on luminance mapping.

[190] The (Luminance Dependent) residual chrominance scaling is designed to compensate for the interaction between the luminance signal and its corresponding chrominance signals. For example, whether residual chroma scaling is enabled or not is also signaled at the tile group level. In one example, if luminance mapping is enabled and if a binary tree split (also known as a separate chroma tree) is not applied to the current tile group, an additional flag is signaled to indicate whether luminance-dependent residual chroma scaling is enabled or not. In another example, when luminance mapping is not used or when binary tree splitting is used in the current tile group, luminance-dependent residual chroma scaling is turned off. In another example, luma-dependent residual chroma scaling is always off for chrominance blocks whose area is less than or equal to 4.

[191] The residual chrominance scaling may be based on the average value of the corresponding luminance prediction block (the luminance component of the prediction block to which the intra-prediction mode and/or the inter-prediction mode is applied). The scaling operations on the encoder side and/or the decoder side may be implemented using fixed-point integer arithmetic based on Equation 3 below.

[192] [Equation 3]

[193] In Equation 3, c denotes the chrominance residual (chroma residual sample, residual sample chrominance component), c' denotes the scaled chrominance residual sample (scaled residual sample chrominance component), s denotes the chrominance residual scaling factor, and CSCALE_FP_PREC denotes (predefined) a constant value to specify the degree of precision. For example, CSCALE_FP_PREC may be 11. The chroma residual scale factor is calculated from ChromaScaleCoef[idxS], where idxS is determined by the corresponding average TU luminance value in the reshaped area.

[194] FIG. 9 is a flowchart illustrating a method for outputting a residual chrominance scaling index according to an embodiment of the present document. The method in FIG. 9 can be performed based on FIG. 6 and the tables, equations, variables, arrays, and functions included in the description relating to FIG. 6.

[195] In step S910, it may be determined whether the prediction mode for the current block is the intra prediction mode or the inter prediction mode based on the prediction mode information. If the prediction mode is an intra-prediction mode, the current block or prediction samples of the current block are considered to be in the reshaped (mapped) area. If the prediction mode is an inter-prediction mode, the current block or prediction samples of the current block are considered to be in the original (non-mapped, non-reformed) region.

[196] In step S920, when the prediction mode is an intra-prediction mode, the average of the current block (or luminance prediction samples of the current block) may be calculated (output). That is, the average of the current block in the already reshaped area is calculated directly. The mean may also be referred to as mean, average, or average.

[197] In step S921, when the prediction mode is an inter prediction mode, direct regeneration (forward mapping) may be performed (applied) on the brightness prediction samples of the current block. Through forward reshaping, luma prediction samples based on the inter prediction mode can be mapped from the original region to the regenerated region. In one example, direct luma prediction resampling may be performed based on the resampling model described with Table 7 above.

[198] In step S922, the average of the forward reshaped (forward mapped) luminance prediction samples may be calculated (output). That is, an averaging process can be performed for the direct reshaped result.

[199] In step S930, a residual chrominance scaling index may be calculated. When the prediction mode is an intra prediction mode, a residual chrominance scaling index may be calculated based on an average of the luminance prediction samples. When the prediction mode is an inter-prediction mode, a residual chrominance scaling index may be calculated based on the average of the direct reshaped luma prediction samples.

[200] In an embodiment, the chroma residual scaling index may be calculated based on the outline syntax. The table below shows an example for a syntax loop for deriving (calculating) the chroma residual scaling index.

[201] [Table 11]

[202] In Table 11, idxS represents the residual chroma scaling index, idxFound represents an index identifying whether a residual chroma scaling index satisfying the condition of the if statement is obtained, S represents a predefined constant value, and MaxBinIdx represents the maximum allowed bin index. ReshapPivot[idxS+1] may be derived based on Table 7 and/or 8 described above.

[203] In an embodiment, the chroma residual scaling factor may be derived based on the chroma residual scaling index. Equation 4 is an example for deriving the residual chrominance scaling factor.

[204] [Equation 4]

[205] In Equation 4, s represents the residual chroma scaling factor, and ChromaScaleCoef may be a variable (or array) derived from Table 7 and/or 8 described above.

[206] As described above, an average luminance value of the reference samples may be obtained, and a residual chrominance scaling factor may be derived based on the average luminance value. As described above, the chrominance residual sample may be scaled based on the chrominance residual scaling factor, and the chrominance component reconstruction sample may be generated based on the scaled chrominance residual sample.

[207] In one embodiment of the present document, a signaling framework is proposed for the efficient use of the above-described LMCS. According to this embodiment of the present document, for example, LMCS data can be included in the HLS (i.e., APS), and through header information (i.e., picture title, slice title), which is a lower level APS, LMCS model (model shaper) can be adaptively derived by signaling the APS ID, which is called header information. The LMCS model can be derived based on the LMCS parameters. Also, for example, a plurality of APS IDs may be signaled via header information, and thereby, different LMCS models may be applied in units of blocks within the same picture/cutout.

[208] In one embodiment, according to the present document, a method is provided for efficiently performing an operation required for an LMCS. In accordance with the semantics described above in Table 8, a division by the fragment length lmcsCW[i] (also referred to as RspCW[i] in this document) is required to derive InvScaleCoeff[i]. The length of the inverse map fragment may not be a power of 2, which means that the division cannot be performed by a bit shift.

[209] For example, the calculation of InvScaleCoeff may require up to 16 divisions per slice. According to Table 8 above, for encoding 10 bits, the range of lmcsCW[i] is from 8 to 511, so in order to implement the division by lmcsCW[i] operation using LUT, the LUT size must be 504. Also, for encoding 12 bits, lmcsCW[i] range is 32 to 2047, so the LUT size must be 2016 in order to implement divide by lmcsCW[i] using LUT. That is, division is costly in hardware implementation, so it is desirable to avoid division whenever possible.

[210] In one aspect of this embodiment, lmcsCW[i] may be limited to a multiple of a fixed number (or a predetermined number). Accordingly, the lookup table (LUT) (capacity or LUT size) for division can be reduced. For example, if lmcsCW[i] becomes a multiple of 2, the size of the LUT to replace the fission process can be halved.

[211] In another aspect of this embodiment, it is proposed that in order to encode with a higher internal bit depth encoding, in addition to the existing restrictions, "the value of lmcsCW[i] must be in the range of (OrgCW>>3) to (OrgCW<< 3-1)", an additional constraint should be that lmcsCW[i] is a multiple of 1<<(BitDepthY-10) if the encoding bit depth is higher than 10. Here, BitDepthY may represent the luma bit depth. Accordingly, the possible number lmcsCW[i] will not change with the bit depth, and the size of the LUT required to calculate InvScaleCoeff does not increase for a higher bit depth. For example, for an internal encoding bit depth of 12 bits, the constraint lmcsCW[i] is a multiple of 4, then the LUT to replace the division process will be the same as the LUT to be used to encode 10 bits. This aspect may be implemented alone, but it may also be implemented in combination with the above aspect.

[212] In another aspect of this embodiment, lmcsCW[i] may be limited to a narrower range. For example, lmcsCW[i] can be limited to the range (OrgCW>>1) to (OrgCW<<1)-1. Then for encoding 10 bits, the range of lmcsCW[i] can be [32, 127], and therefore only a LUT having a size of 96 is needed to calculate InvScaleCoeff.

[213] In another aspect of the present embodiment, lmcsCW[i] is approximated by nearest numbers that are powers of 2 and used in the design of the shaper. Accordingly, the division in the inverse mapping can be performed (and replaced) by a bit shift.

[214] In one embodiment, according to the present document, a range limitation of LMCS codewords is proposed. As shown in Table 8 above, the LMCS codeword values range from (OrgCW>>3) to (OrgCW<<3)-1. This codeword range is too wide. This can lead to a visual artifact problem when there are large differences between RspCW[i] and OrgCW.

[215] In accordance with one embodiment according to the present document, it is proposed to limit the LMCS PWL mapping codeword to a narrower range. For example, lmcsCW[i] can range from (OrgCW>>1) to (OrgCW<<1)-1.

[216] In one embodiment according to the present document, the use of a single chroma residual scaling factor is proposed for chroma residual scaling in an LMCS. The existing method for outputting the residual chrominance scaling factor uses the average value of the corresponding luma block and outputs the slope of each luminance inverse map tile as the corresponding scaling factor. In addition, the process for identifying a chunked index requires the availability of the corresponding luma block, which leads to a latency problem. This is undesirable for hardware implementation. In accordance with this embodiment of the present document, the scaling in the chrominance block may not depend on the value of the luminance block, and it may not be necessary to identify the chunk index. Therefore, the residual chroma scaling process in the LMCS can be performed without a delay problem.

[217] In one embodiment according to the present document, one chrominance scaling factor can be derived from both the encoder and the decoder based on the luminance information of the LMCS. When the LMCS luma model is received, the residual chrominance scaling factor may be updated. For example, when the LMCS model is updated, one residual chroma scaling factor may be updated.

[218] The table below shows an example for obtaining one chrominance scaling factor in accordance with the present embodiment.

[219] [Table 12]

[220] With reference to Table 12, one chroma scaling factor (eg, ChromaScaleCoeff or ChromaScaleCoeffSingle) can be obtained by averaging the luma inverse mapping slopes of all tiles within lmcs_min_bin_idx and lmcs_max_bin_idx.

[221] FIG. 10 illustrates a linear approximation of turning points in accordance with an embodiment of the present document. In FIG. 10, turning points P1, Ps and P2 are shown. The following embodiments or examples thereof will be described with reference to FIGS. ten.

[222] In an example of this embodiment, one chroma scaling factor can be obtained based on a linear approximation of the PWL luminance display between the pivot points lmcs_min_bin_idx and lmcs_max_bin_idx+1. That is, the inverse slope of the linear display can be used as the residual chrominance scaling factor. For example, linear line 1 in FIG. 10 may be a straight line connecting the pivot points P1 and P2. With reference to FIG. 10, at P1, the input value is x1 and the displayed value is 0, and at P2, the input value is x2 and the displayed value is y2. The inverse slope (inverse scale) of the linear line 1 is (x2-x1)/y2, and one chroma scaling factor ChromaScaleCoeffSingle may be calculated based on the input values and the displayed values of the pivot points P1, P2 and the following equation.

[223] [Equation 5]

[224] In Equation 5, CSCALE_FP_PREC represents a shift factor, for example, CSCALE_FP_PREC may be a predetermined constant value. In one example, CSCALE_FP_PREC may be 11.

[225] In another example according to this embodiment, with reference to FIG. 10, the input value at the pivot point Ps is min_bin_idx+1, and the displayed value at the pivot point Ps is ys. Accordingly, the inverse slope (inverse scale) of the linear line 1 may be calculated as (xs-x1)/ys, and one ChromaScaleCoeffSingle chroma scaling factor may be calculated based on the input values and the displayed values of the turning points P1, Ps and the following equation.

[226] [Equation 6]

[227] In Equation 6, CSCALE_FP_PREC represents a shift factor (factor for bit shift), for example, CSCALE_FP_PREC may be a predetermined constant value. In one example, CSCALE_FP_PREC may be 11 and the bit shift for inverse scaling may be performed based on CSCALE_FP_PREC.

[228] In another example, according to this embodiment, one residual chrominance scaling factor may be derived based on a linear approximation line. An example for deriving a linear approximation line may include a linear connection of the turning points (ie, lmcs_min_bin_idx, lmcs_max_bin_idx+1). For example, the result of the linear approximation may be represented by PWL mapping codewords. The displayed value of y2 in P2 can be the sum of the codewords of all bins (fragments), and the difference between the input value in P2 and the input value in P1 (x2-x1) is OrgCW*(lmcs_max_bin_idx-lmcs_min_bin_idx+1) (for OrgCW, see Table 8 above). The table below shows an example of obtaining one chrominance scaling factor according to the above-described embodiment.

[229] [Table 13]

[230] With reference to Table 13, one chroma scaling factor (eg, ChromaScaleCoeffSingle) can be obtained from two pivot points (ie, lmcs_min_bin_idx, lmcs_max_bin_idx). For example, the inverse slope of a linear display may be used as a chrominance scaling factor.

[231] In another example of this embodiment, one chroma scaling factor can be obtained by linear approximation of the turning points to minimize the error (or rms error) between the linear approximation and the existing PWL mapping. This example can be more accurate than just concatenating the two pivots in lmcs_min_bin_idx and lmcs_max_bin_idx. There are many ways to find the optimal line mapping, and such an example is described below.

[232] In one example, the parameters b1 and b0 of the linear approximation equation y=b1*x+b0 for minimizing the sum of the least squared error can be calculated based on Equations 7 and/or 8 below.

[233] [Equation 7]

[234] [Equation 8]

и

являются средним x и y, и x_i и y_i представляют значения i-ых точек поворота.[235] In Equations 7 and 8, x is the original luminance values and y is the reshaped luminance values, and

and

are the average of x and y, and x _i and y _i represent the values of the i-th pivot points.

[236] With reference to FIG. 10, another simple approximation for identifying a line mapping is given as follows:

[237] - Obtain linear line 1 by joining the PWL mapping pivot points at lmcs_min_bin_idx and lmcs_max_bin_idx+1 computed by lmcs_pivots_linear[i] of this linear line with input multiples of OrgW.

[238] - Sum the differences between the displayed values of the turning points using Linear 1 and using the PWL display.

[239] - Get the average difference avgDiff.

[240] - Adjust the last turning point of the linear line according to the mean difference, eg 2*avgDiff.

[241] - Use the reverse slope of the adjusted linear line as the chroma residual scale.

[242] According to the linear approximation described above, the chrominance scaling factor (i.e., forward display inverse slope) can be derived (obtained) based on Equation 9 or 10 below.

[243] [Equation 9]

[244] [Equation 10]

[245] In the above equations, lmcs_pivots_lienar[i] may be mapped values of the linear map. In a linear mapping, all PWL mapping tiles between the minimum and maximum bin indexes may have the same LMCS codeword (lmcsCW). That is, lmcs_pivots_linear[lmcs_min_bin_idx+1] can be the same as lmcsCW[lmcs_min_bin_idx].

[246] Also, in Equations 9 and 10, CSCALE_FP_PREC represents a shift factor (factor for bit shift), for example, CSCALE_FP_PREC may be a predetermined constant value. In one example, CSCALE_FP_PREC may be 11.

[247] With one chroma residual scaling factor (ChromaScaleCoeffSingle), it is no longer necessary to calculate the average of the corresponding luminance block, find the index in the PWL linear mapping, to obtain the chroma residual scaling factor. Accordingly, coding efficiency using residual chrominance scaling can be improved. This not only eliminates the dependency on the corresponding luminance block, solves the delay problem, but also reduces the complexity.

[248] In another embodiment of the present document, an encoder may determine parameters related to one chrominance scaling factor and signal the parameters to a decoder. When signaling, the encoder may use other information available at the encoder to derive a chroma scaling factor. This embodiment aims to eliminate the residual chrominance scaling delay problem.

[249] For example, another example for identifying a linear display to be used to determine the residual chrominance scaling factor is given as follows:

[250] - By connecting the PWL mapping pivots at lcs_min_bin_idx and lmcs_max_bin_idx+1 computed by lmcs_pivots_linear[i] from this linear line at input multiples of OrgW

[251] - Get the weighted sum of the differences between the displayed pivot point values using Linear 1 and the PWL luminance display values. The weight may be based on an encoder statistic such as a bin histogram.

[252] - Get the weighted average difference avgDiff.

[253] - Adjust the last turning point of linear line 1 according to the weighted average difference, eg 2*avgDiff.

[254] - Use the inverse slope of the adjusted linear line to calculate the chroma residual scale.

[255] The table below shows examples of syntaxes for signaling the value of y to output the chrominance scaling factor.

[256] [Table 14]

[257] In Table 14, the lmcs_chroma_scale syntax element may specify one chroma (residual) scaling factor used for LMCS residual chroma scaling (ChromaScaleCoeffSingle=lmcs_chroma_scale). That is, the chroma residual scaling factor information can be directly signaled, and the signaled information can be output as the chroma residual scaling factor. In other words, the value of the signaled chroma residual scaling factor information can be (directly) output as the value of one chroma residual scaling factor. Here, the lmcs_chroma_scale syntax element may be signaled along with other LMCS data (ie, a syntax element relating to the absolute value and sign of the codeword, etc.).

[258] Alternatively, the encoder may signal only the necessary parameters to output the residual chrominance scaling factor at the decoder. To output the residual chrominance scaling factor at the decoder, an input x value and a mapped y value are required. Since the value of x is the length of the bin and is a known number, it does not need to be signaled. In essence, only the value of y needs to be signaled to output the residual chroma scaling factor. Here, the value of y may be the mapped value of any pivot point in the linear display (ie, the mapped values of P2 or Ps in FIG. 10).

[259] The following tables show examples of signaling mapped values for deriving the residual chrominance scaling factor.

[260] [Table 15]

[261] [Table 16]

[262] One of the syntaxes of Tables 15 and 16 described above may be used to signal the value of y at any linear turning points specified by the encoder and decoder. That is, the encoder and decoder can output the value of y using the same syntax.

[263] First, an embodiment is described in accordance with Table 15. In Table 15, lmcs_cw_linear may denote a mapped value at Ps or P2. That is, in the embodiment according to Table 15, the fixed number may be signaled via lmcs_cw_linear.

[264] In the example according to this embodiment, if lmcs_cw_linear denotes a mapped value of one bin (i.e., lmcs_pivots_linear[lmcs_min_bin_idx+1] in Ps according to Fig. 10), the chrominance scaling factor may be derived based on the following equation.

[265] [Equation 11]

[266] In another example according to this embodiment, if lmcs_cw_linear denotes lmcs_max_bin_idx+1 (i.e., lmcs_pivots_linear[lmcs_max_bin_idx+1] in P2 of FIG. 10), the chroma scaling factor may be derived based on the following equation.

[267] [Equation 12]

[268] In the above equations, CSCALE_FP_PREC represents a shift factor (coefficient for bit shift), for example, CSCALE_FP_PREC may be a predetermined constant value. In one example, CSCALE_FP_PREC may be 11.

[269] Next, an embodiment is described in accordance with Table 16. In this embodiment, lmcs_cw_linear may be signaled as a delta value about a fixed number (i.e., lmcs_delta_abs_cw_linear, lmcs_delta_sign_cw_linear_flag). In the example of this embodiment, when lmcs_cw_linear represents the mapped value in lmcs_pivots_linear[lmcs_min_bin_idx+1] (i.e. Ps according to FIG. 10), lmcs_cw_linear_delta and lmcs_cw_linear can be derived based on the following equations.

[270] [Equation 13]

[271] [Equation 14]

[272] In another example of this embodiment, when lmcs_cw_linear represents the mapped value in lmcs_pivots_linear[lmcs_max_bin_idx+1] (i.e., P2 of FIG. 10), lmcs_cw_linear_delta and lmcs_cw_linear may be derived based on the following equations.

[273] [Equation 15]

[274] [Equation 16]

[275] In the above equations, OrgCW may be a value derived from Table 8 described above.

[276] FIG. 11 illustrates one example of a linear reshaping (or linear reshaping, linear mapping) according to an embodiment of the present document. That is, in this embodiment, the use of a line driver in the LMCS is proposed. For example, the example in FIG. 11 may refer to direct linear regeneration (display).

[277] In the present example, the LMCS may use a piecewise linear mapping with fixed 16 tiles. This complicates the shaper circuit, since a sharp transition between turning points can cause an unwanted artifact. Also, in order to inversely display the brightness of the shaper, it is necessary to identify the piecewise index of the function. The process of identifying a piecewise function index is an iterative process with many comparisons. Moreover, for residual chrominance scaling using the corresponding average luminance block, such a piecewise luminance index identification process is also needed. Not only does this have a complexity issue, it also causes a delay in residual chroma scaling depending on restoring the entire luma block. To solve this problem, the use of a line driver is proposed for the LMCS.

[278] With reference to FIG. 11, the line driver may include two pivot points, i.e., P1 and P2. P1 and P2 may represent input and mapped values, for example, P1 may be (min_input, 0) and P2 may be (max_input, max_mapped). Here, min_input represents the minimum input value and max_input represents the maximum input value. Any input values less than or equal to min_input are mapped to 0, any input values greater than max_input are mapped to max_mapped. Any brightness input values within min_input and max_input map linearly to other values. Fig. 11 shows a display example. The pivot points P1, P2 may be determined in the encoder and a linear approximation may be used to approximate a piecewise linear mapping.

[279] There are many ways to signal a line driver. In an example of a line driver signaling method, the original luminance range may be divided into an equal number of bins. That is, the luminance mapping between the minimum and maximum bins can be evenly distributed. For example, all bins may have the same LMCS codeword (lmcsCW). For this reason, the minimum and maximum bin indexes can be signaled. Therefore, only one set of reshape_model_bin_delta_abs_CW (or reshaper_model_delta_abs_CW, lmcs_delta_abs_CW) and reshaper_model_bin_delta_sign_CW_flag (or reshaper_model_delta_sign_CW_flag_sign_CW_flag) should be signaled.

[280] The following tables show exemplary syntax and signaling semantics of a linear driver in accordance with this example.

[281] [Table 17]

[282] [Table 18]

[283] With reference to Tables 17 and 18, the syntax element log2_lmcs_num_bins_minus4 may represent information about the number of bins. Based on this information, the number of bins can be signaled to allow better control of the minimum and maximum turning points. In another existing example, the encoder and/or decoder may output (specify) the (fixed) number of bins without signaling, for example, the number of bins is output as 16 or 32. However, in accordance with the examples of Tables 17 and 18, log2_lmcs_num_bins_minus4 plus 4 may represent a binary logarithm of the number of bins. The number of bins derived from the syntax element can be in the range from 4 to the luma bit depth value BitDepthY.

[284] In Table 18, ScaleCoeffSingle may be referred to as one forward luminance scaling factor, and InvScaleCoeffSingle may be referred to as one inverse luminance scaling factor. (Forward) mapping on the luminance sample prediction may be performed based on a single forward luminance scaling factor, and (inverse) mapping on the reconstructed luma samples may be performed based on a single inverse luminance scaling factor. ChromaScaleCoeffSingle, as described above, may be referred to as one residual chroma scaling factor. ScaleCoeffSingle, InvScaleCoeffSingle, and ChromaScaleCoeffSingle can be used for luminance forward mapping, luminance inverse mapping, and chroma residual scaling, respectively. ScaleCoeffSingle, InvScaleCoeffSingle and ChromaScaleCoeffSingle can be equally applied to all bins (16 bin PWL mappings) as a single coefficient.

[285] With reference to Table 18, FP_PREC and CSCALE_FP_PREC may be constant values for the bit shift. FP_PREC and CSCALE_FP_PREC may or may not be the same. For example, FP_PREC may be greater than or equal to CSCALE_FP_PREC. In one example, FP_PREC and CSCALE_FP_PREC may both be 11. In another example, FP_PREC may be 15 and CSCALE_FP_PREC may be 11.

[286] In another example of a linear generator signaling method, an LMCS codeword (lmcsCWlinearALL) may be derived based on the following equations in Table 19. Also in this example, the linear regeneration syntax elements in Table 17 may be used. The following table shows an example of the semantics described according to this example.

[287] [Table 19]

[288] With reference to Table 19, FP_PREC and CSCALE_FP_PREC may be constant values for the bit shift. FP_PREC and CSCALE_FP_PREC may or may not be the same. For example, FP_PREC may be greater than or equal to CSCALE_FP_PREC. In one example, FP_PREC and CSCALE_FP_PREC may both be 11. In another example, FP_PREC may be 15 and CSCALE_FP_PREC may be 11.

[289] In Table 19, lmcs_max_bin_idx can be used interchangeably with LmcsMaxBinIdx. lmcs_max_bin_idx and LmcsMaxBinIdx may be derived based on the lmcs_delta_max_bin_idx syntax of Tables 17 and 18 described above. For example, Table 19 may be interpreted with reference to Table 18.

[290] In another embodiment according to the present document, another exemplary method for signaling a line driver may be provided. The turning points P1, P2 of the linear generator model can be signaled explicitly. The following tables show an example of the syntax and semantics for explicitly signaling a linear conditioner model according to this example.

[291] [Table 20]

[292] [Table 21]

[293] With reference to Tables 20 and 21, the first pivot point input value may be derived based on the lmcs_min_input syntax element, and the second pivot point input value may be derived based on the lmcs_max_input syntax element. The mapped value of the first pivot point may be a predefined value (a value known to the encoder and decoder), for example, the mapped value of the first pivot point is 0. The mapped value of the second pivot point may be inferred based on the lmcs_max_mapped syntax element. That is, the line driver model can be explicitly (directly) signaled based on the information signaled in the syntax of Table 20.

[294] Alternatively, lmcs_max_input and lmcs_max_mapped may be signaled as delta values. The following tables show an example of the syntax and semantics for signaling a linear shaper model as delta values.

[295] [Table 22]

[296] [Table 23]

[297] With reference to Table 23, the input value of the first pivot point may be derived based on the lmcs_min_input syntax element. For example, lmcs_min_input may have a mapped value of 0. lmcs_max_input_delta may specify the difference between the second pivot input value and the maximum brightness value (ie, (1<<bitdepthY)-1). lmcs_max_mapped_delta may specify the difference between the mapped value of the second pivot point and the maximum brightness value (ie, (1<<bitdepthY)-1).

[298] According to an embodiment of the present document, forward mapping for luminance prediction samples, inverse mapping for luminance restoration samples, and residual chroma scaling can be performed based on the above-described examples of the linear generator. In one example, only one inverse scaling factor may be required for inverse scaling for (reconstructed) luma samples (pixels) in a linearizer-based inverse mapping. This is also true for direct mapping and residual chroma scaling. That is, the steps for determining ScaleCoeff[i], InvScaleCoeff[i], and ChromaScaleCoeff[i], where i is a bin index, can be replaced with only one coefficient. Here, one coefficient is a fixed-point representation of the (forward) slope or inverse slope of the linear display. In one example, the luminance inverse mapping scaling factor (the luminance inverse mapping inverse scaling factor for luminance restoration samples) may be derived based on at least one of the following equations.

[299] [Equation 17]

[300] [Equation 18]

[301] [Equation 19]

[302] lmcsCWLinear from Equation 17 can be derived from Tables 17 and 18 described above. lmcsCWLinearALL from Equations 18 and 19 may be derived from at least one of Tables 19-23 described above. In Equation 17 or 18, OrgCW may be derived from Table 8 or 18.

[303] The following tables describe equations and syntax (conditional statements) indicating a direct mapping process for luminance samples (ie, luminance prediction samples) in picture reconstruction. In the following tables and equations, FP_PREC is a constant value for bit shift and may be a predefined value. For example, FP_PREC could be 11 or 15.

[304] [Table 24]

[305] [Table 25]

[306] Table 24 may be for outputting direct mapped luminance samples in a luminance mapping process based on Tables 7 and 8 described above. That is, Table 24 may be described in conjunction with Tables 7 and 8. In Table 24, direct mapped luma samples (predictions) PredMAPSamples[i][j] as an output may be derived from luma samples (predictions) predSamples[i][j ] as input. idxY of Table 24 may be referred to as a (forward) display index, and the display index may be derived based on prediction luma samples.

[307] Table 25 may be designed to output direct mapped luminance samples in a luminance mapping based on a linear shaper. For example, lmcs_min_input, lmcs_max_input, lmcs_max_mapped, and ScaleCoeffSingle from Table 25 may be derived from at least one of Tables 20-23. In Table 25, in the case of 'lmcs_min_input < predSamples[i][j] < lmcs_max_input', direct mapped luma samples (predictions) PredMAPSamples[i][j] can be output as output from input, luma samples (predictions) predSamples[i] [j]. In comparison between Table 24 and Table 25, the change from the existing LMCS according to the application of the line driver can be seen in terms of direct mapping.

[308] The following equations describe the inverse mapping process for luminance samples (ie, luminance recovery samples). In the following equations, 'lumaSample' as an input can be a pre-backmap luminance restoration sample (before modification). 'invSample' as an output may be an inverse mapped (modified) brightness restoration sample. In other cases, the truncated invSample may be referred to as a modified brightness restoration sample.

[309] [Equation 20]

[310] [Equation 21]

[311] With reference to Equation 20, the idxInv index may be derived based on Table 10 described above. That is, Equation 20 may be designed to output inverse mapped luminance samples in a luminance mapping process based on Tables 7 and 8 described above. Equation 20 may be described in conjunction with Table 9 described above.

[312] Equation 21 may be designed to output inversely mapped luminance samples in a luminance mapping based on a linear shaper. For example, lmcs_min_input from Equation 21 may be output according to at least one of Tables 20-23. In comparison between Equation 20 and Equation 21, the change from the existing LMCS according to the application of the line driver can be seen in terms of direct mapping.

[313] Based on the examples of the linear generator described above, the piecewise index identification process may be omitted. That is, in the present examples, since there is only one fragment having actual reshaped luminance pixels, the chunk index identification process used for luma inverse mapping and residual chroma scaling can be eliminated. Accordingly, the complexity of luminance inverse mapping can be reduced. Also, latency problems caused by dependency on the identification of the chunked luma index during residual chroma scaling can be eliminated.

[314] According to the line generator embodiment described above, the following advantages can be provided for LMCS: i) the encoder generator circuit can be simplified, preventing possible artifact caused by abrupt changes between piecewise line tiles, ii) the decoder demapping process, in which the piecewise index identification process can be eliminated can be simplified by eliminating the piecewise index identification process, iii) by eliminating the piecewise index identification process, the problem of delay in residual chroma scaling caused by dependence on the respective luma blocks can be eliminated, iv) overlap can be reduced signaling and make frequent updating of the shaper more rational, v) for the many places that were required for the 16 fragment loop, the loop can be eliminated. For example, to output InvScaleCoeff[i], the number of divisions by lmcsCW[i] can be reduced to 1.

[315] In another embodiment, in accordance with the present document, the proposed LMCS based on flexible bins. Here, flexible bins may refer to the number of bins not fixed to a predetermined (predetermined, specific) number. In the current embodiment, the number of bins in the LMCS is fixed at 16, and 16 bins are evenly distributed for input sample values. In this embodiment, a flexible number of bins is provided, and fragments (bins) may not be equally distributed in terms of original pixel values.

[316] The following tables show exemplary syntax for LMCS data (data fields) in accordance with this embodiment and the semantics for the syntax elements included therein.

[317] [Table 26]

[318] [Table 27]

[319] With reference to Table 26, information about the number of bins lmcs_num_bins_minus1 may be signaled. With reference to Table 27, lmcs_num_bins_minus1+1 may be equal to the number of bins, of which the number of bins may range from 1 to (1<<BitDepthY)-1. For example, lmcs_num_bins_minus1 or lmcs_num_bins_minus1+1 can be a multiple of a power of 2.

[320] In the embodiment described in conjunction with Tables 26 and 27, the number of pivot points may be derived based on lmcs_num_bins_minus1 (bin number information), whether the shaper is linear or not (lmcs_num_bins_minus1 signaling), and the input values and displayed pivot point values (LmcsPivot_input[i], LmcsPivot_mapped[i]) may be derived based on summation of signaled codeword values (lmcs_delta_input_cw[i], lmcs_delta_mapped_cw[i]) (here, original input value LmcsPivot_input[0] and original output value LmcsPivot_mapped[ 0] are equal to 0).

[321] FIG. 12 shows an example of linear forward mapping in an embodiment of the present document. Fig. 13 shows an example of reverse forward mapping in an embodiment of the present document. In the embodiment according to FIG. 12 and FIG. 13, a method is proposed to support both regular LMCS and linear LMCS. In an example according to this embodiment, regular LMCS and/or linear LMCS may be indicated based on the lmcs_is_linear syntax element. At the encoder, after the linear LMCS line is determined, the mapped value (i.e., the mapped value in pL in FIG. 12 and FIG. 13) may be divided into equal chunks (i.e., LmcsMaxBinIdx-lmcs_min_bin_idx+1). The codeword in the LmcsMaxBinIdx bin may be signaled using the syntaxes for lmcs data or shaper mode described above.

[322] The following tables show exemplary syntax for LMCS data (data fields) and semantics for the syntax elements included therein, in accordance with an example of this embodiment.

[323] [Table 28]

[324] [Table 29]

[325]

[326] The following tables roughly show the syntax for LMCS data (data fields) and the semantics for the syntax elements included therein, in accordance with another example of this embodiment.

[327] [Table 30]

[328] [Table 31]

[329]

[330] With reference to Tables 28-31, when lmcs_is_linear_flag is true, all lmcsDeltaCW[i] between lmcs_min_bin_idx and LmcsMaxBinIdx may have the same value. That is, lmcsCW[i] of all fragments between lmcs_min_bin_idx and LmcsMaxBinIdx can have the same value. Scaling and inverse scaling and chrominance scaling of all tiles between lmcs_min_bin_idx and lmcsMaxBinIdx can be the same. Then, if the linearizer is true, then there is no need to output the fragment index, one can use scaling, inverse scaling from only one of the fragments.

[331] The following table shows an exemplary process for identifying a piece index in accordance with the present embodiment.

[332] [Table 32]

[333] In accordance with another embodiment of the present document, the use of regular 16-piece PWL LMCS and linear LMCS may depend on the syntax of a higher level (ie, sequence level).

[334] The following tables show the syntax for the SPS according to the present embodiment and the semantics for the syntax elements included therein.

[335] [Table 33]

[336] [Table 34]

[337] With reference to Tables 33 and 34, the inclusion of regular LMCS and/or linear LMCS may be determined (signaled) by a syntax element included in the SPS. With reference to Table 34, based on the sps_linear_lmcs_enabled_flag syntax element, one of a regular LMCS or a linear LMCS may be used in sequence units.

[338] In addition, whether only linear LMCS or regular LMCS or both should be included may also depend on the profile level. In one example, for a particular profile (i.e., SDR profile), only linear LMCS may be allowed, for another profile (i.e., HDR profile), only regular LMCS may be allowed, and for another profile, regular LMCS and/or linear LMCS be allowed.

[339] In accordance with another embodiment of the present document, the LMCS chunk index identification process can be used in luma inverse mapping and chrominance residual scaling. In this embodiment, the chunked index identification process can be used for blocks with residual chroma scaling enabled and also invoked for all luminance samples in the reshaped (mapped) region. The present embodiment aims to keep complexity low.

[340] The following table shows the identification process (derivation process) of an existing piecewise function index.

[341] [Table 35]

[342] In an example, in the process of identifying a piecewise index, the input samples may be classified into at least two categories. For example, the input samples may be classified into three categories, first, second, and third categories. For example, the first category may represent samples (their values) less than LmcsPivot[lmcs_min_bin_idx+1], and the second category may represent samples (their values) greater than or equal to (their values) LmcsPivot[LmcsMaxBinIdx], the third category may indicate samples ( their values) between LmcsPivot[lmcs_min_bin_idx+1] and LmcsPivot[LmcsMaxBinIdx].

[343] In this embodiment, it is proposed to optimize the identification process by eliminating classifications into categories. That is, since the input to the chunked index identification process is the luminance values in the reshaped (mapped) region, there should be no values other than the mapped values at the pivot points lmcs_min_bin_idx and LmcsMaxBinIdx+1. Accordingly, a conditional process for classifying samples into categories is not needed in the existing chunked index identification process. With regard to details, specific examples will be described below using tables.

[344] In the example according to this embodiment, the identification process included in Table 35 may be replaced by the process of Tables 36 or 37 below. With reference to Tables 36 and 37, the first two categories of Table 35 can be removed, and for the last category, the limit value (second limit value or end point) in the loop iteration is changed from LmcsMaxBinIdx to LmcsMaxBinIdx+1. That is, the identification process can be simplified and the complexity for deriving the piecewise index can be reduced. Accordingly, LMCS-related coding can be efficiently performed according to the present embodiment.

[345] [Table 36]

[346] [Table 37]

[347] With reference to Table 36, the comparison process corresponding to the condition of the if statement (the equation corresponding to the condition of the if statement) may be repeatedly performed on all bin indices from the minimum bin index to the maximum bin index. The index of the bin, in the case where the equation corresponding to the condition of the if statement is true, can be output as a backmap index for luma backmap (or a backscaling index for residual chroma scaling). Based on the inverse mapping index, modified reconstructed luminance samples (or scaled chrominance residual samples) may be output.

[348] The following drawings explain specific examples of this specification. Since the names of specific devices shown in the drawings or the names of specific signals/messages/fields are given by way of example, the technical features of this specification are not limited to the specific names used in the following drawings.

[349] FIG. 14 and FIG. 15 schematically show an example of a video/image coding method and related components according to the embodiment(s) of the present document. The method disclosed in FIG. 14 may be performed by the encoder disclosed in FIG. 2. Specifically, for example, S1400 in FIG. 14 may be performed by residual processor 230 or encoder predictor 220, S1410 may be performed by predictor 220 or encoder adder 250. S1420 may be performed by encoder residual processor 230 or encoder adder 250, predictor 220 or encoder residual processor 230, S1430 and/or S1440 may be performed by encoder residual processor 230, and S1450 may be performed by entropy encoder 240 of the encoder. The method disclosed in FIG. 14 may include the embodiments described above herein.

[350] With reference to FIG. 14, the encoder may generate prediction luma samples (S1410). With reference to the prediction luma samples, the encoding device may output the prediction luma samples of the current block based on the prediction mode. In this case, various prediction methods disclosed herein, such as inter-prediction or intra-prediction, may be applied.

[351] An encoder may output prediction chrominance samples. The encoder may derive residual chroma samples based on the original chroma samples and the prediction chroma samples of the current block. For example, the encoder may derive residual chroma samples based on a difference between the prediction chroma samples and the original chroma samples.

[352] The encoding device may generate mapped prediction luma samples (S1410). For example, the encoder may output input values and mapped values (output values) of pivot points for displaying luminance, and may generate mapped prediction luminance samples based on the input values and mapped values. In one example, the encoder may derive a display index (idxY) based on the first luminance prediction sample, and may generate a first mapped luminance prediction sample based on the input value and the mapped pivot value corresponding to the display index. In another example, a linear mapping (linear regeneration, linear LMCS) may be used, and the mapped prediction luma samples may be generated based on the direct mapping scaling factor derived from the two pivot points in the linear mapping, thus the index derivation process may be omitted due to the linear display.

[353] An encoder may generate scaled chrominance residual samples. Specifically, the encoding device may output a chroma residual scaling factor and generate scaled chrominance residual samples based on the chrominance residual scaling factor. Here, the residual chrominance scaling of the encoding stage may be referred to as direct residual chroma scaling. Accordingly, the chrominance residual scaling factor output by the encoder may be referred to as the direct chrominance residual scaling factor, and direct scaled chrominance residual samples may be generated.

[354] The encoder may derive LMCS related information based on the mapped prediction luma samples (S1420). In addition, the encoder may generate LMCS related information based on the mapped prediction luma samples and/or the scaled chrominance residual samples. The encoder may generate LMCS related information for the recovered samples. The encoder may output LMCS related parameters that can be used to filter the reconstructed samples, and can generate LMCS related information based on the LMCS related parameters. For example, the LMCS-related information may include luma mapping information (i.e., forward mapping, inverse mapping, linear mapping), chroma residual scaling information, and/or indices (i.e., maximum bin index, minimum bin index , display index) associated with the LMCS (or reshaping, reshaper).

[355] The encoding device may generate residual luminance samples based on the mapped prediction luminance samples (S1430). For example, the encoder may derive residual luminance samples based on a difference between the mapped prediction luminance samples and the original luminance samples.

[356] The encoder may output residual information (S1440). The encoder may derive residual information based on the scaled chroma residuals and/or luminance residuals. The encoder may derive transform coefficients based on the scaled chroma residuals and/or a transform process for the luma residuals. For example, the conversion process may include at least one of DCT, DST, GBT, or CNT. The encoder may derive quantized transform coefficients based on a quantization process for the transform coefficients. The quantized transform coefficients may be in the form of a one-dimensional vector based on the scan order of the coefficients. The encoder may generate residual information specifying the quantized transform coefficients. The residual information may be generated by various coding methods such as exponential Golomb coding, CAVLC, CABAC, and the like.

[357] An encoding device may encode image/video information (S1450). The image information may include LMCS related information and/or residual information. For example, the LMCS related information may include linear LMCS information. In one example, at least one LMCS codeword may be derived based on linear LMCS information. The encoded video/image information may be output in the form of a bitstream. The bit stream may be transmitted to the decoding device via a network or a storage medium.

[358] The image/video information may include various information according to an embodiment of the present document. For example, the image/video information may include information disclosed in at least one of Tables 1-37 described above.

[359] In one embodiment, a minimum bin index (ie, lmcs_min_bin_idx) and/or a maximum bin index (ie, LmcsMaxBinIdx) may be derived based on LMCS related information. Information about the minimum bin index (ie, lmcs_min_bin_idx) and/or the maximum bin index (ie, LmcsMaxBinIdx) may be included in the LMCS related information. The first mapped value (LmcsPivot[lmcs_min_bin_idx]) may be inferred based on the minimum bin index. The second mapped value (LmcsPivot[LmcsMaxBinIdx] or LmcsPivot[LmcsMaxBinIdx+1]) may be derived based on the maximum bin index. The values of the reconstructed luma samples (ie, lumaSample of Tables 36 or 37) may range from the first mapped value to the second mapped value. In one example, the values of all recovered luminance samples may range from the first displayed value to the second displayed value. In another example, the values of some of the reconstructed brightness samples may range from the first displayed value to the second displayed value.

[360] In one embodiment, the image information may include a sequence parameter set (SPS). The SPS may include a linear LMCS enabled flag specifying whether linear LMCS is enabled.

[361] In one embodiment, the encoder may generate (output) a chunked index for residual chrominance scaling. The encoder may derive a residual chrominance scaling factor based on the chunked index. The encoder may generate scaled chroma residual samples based on the chrominance residual samples and the chrominance residual scaling factor.

[362] In one embodiment, the chroma residual scaling factor may be one chroma residual scaling factor.

[363] In one embodiment, the LMCS related information may include an LMCS data field and linear LMCS information. The linear LMCS information may be referred to as linear mapping information. The LMCS data field may include a linear LMCS flag indicating whether linear LMCS is applied. When the linear LMCS flag value is 1, mapped prediction luma samples may be generated based on the linear LMCS information.

[364] In one embodiment, the linear LMCS information includes first pivot point information (ie, P1 in FIG. 11) and second pivot point information (ie, P2 in FIG. 11). For example, the input value and the displayed value of the first pivot point are the minimum input value and the minimum displayed value, respectively. The input value and the displayed value of the second pivot point are the maximum input value and the maximum displayed value, respectively. Input values between the minimum input value and the maximum input value are linearly mapped.

[365] In one embodiment, the image information includes information about the maximum input value and information about the maximum displayed value. The maximum input value is equal to the maximum input value information value (ie, lmcs_max_input in Table 20). The maximum mapped value is equal to the maximum mapped value information value (ie, lmcs_max_mapped in Table 20).

[366] In one embodiment, the linear mapping information includes information about the input delta value of the second pivot (i.e., lmcs_max_input_delta in Table 22) and information about the mapped delta value of the second pivot (i.e., lmcs_max_mapped_delta in Table 22). The maximum input value may be output based on the input delta value of the second pivot point, and the maximum displayed value may be output based on the displayed delta value of the second pivot point.

[367] In one embodiment, the maximum input value and the maximum displayed value may be derived based on at least one equation included in Table 23 described above.

[368] In one embodiment, generating mapped prediction luminance samples comprises deriving a direct mapping scaling factor (i.e., ScaleCoeffSingle) for the prediction luminance samples and generating mapped prediction luminance samples based on the direct mapping scaling factor. The forward display scaling factor may be one factor for prediction luma samples.

[369] In one embodiment, the direct display scaling factor may be derived based on at least one equation included in Tables 21 and/or 23 described above.

[370] In one embodiment, the mapped prediction brightness samples may be derived based on at least one equation included in Table 25 described above.

[371] In one embodiment, the encoder may derive an inverse mapping scaling factor (ie, InvScaleCoeffSingle) for the reconstructed luma samples (ie, lumaSample). Also, the encoder may generate modified reconstructed luma samples (ie, invSample) based on the reconstructed luma samples and the inverse mapping scaling factor. The inverse display scaling factor may be one factor for the reconstructed luminance samples.

[372] In one embodiment, the inverse display scaling factor may be derived using a piecewise index derived from the recovered luminance samples.

[373] In one embodiment, the piece index may be derived based on Table 36 described above. That is, the comparison process included in Table 36 (lumaSample<LmcsPivot[idxYInv+1]) can be iteratively performed from a piecewise index that is a minimum bin index to a piecewise index that is a maximum bin index.

[374] In one embodiment, modified reconstructed luminance samples may be generated based on mapped values based on bin indices from a minimum bin index to a maximum bin index.

[375] In one embodiment, the bin indices include a first bin index indicating the first mapped value. The first bin index may be inferred based on the mapped values based on the bin indices from the minimum bin index to the maximum bin index and a comparison between the values of the reconstructed luminance samples. At least one of the modified recovered luminance samples may be derived based on the recovered luminance samples and the first displayed value.

[374] In one embodiment, the bin indices include a first bin index indicating the first mapped value, and the first bin index may be derived based on the comparison equation included in Table 36 (lumaSample<LmcsPivot[idxYInv+1]). For example, in the equation above, lumaSample represents the value of the target luminance sample among the reconstructed luminance samples, idxYInv represents one of the bin indices, and LmcsPivot[idxYinv+1] represents one of the mapped values based on the bin indices from the minimum bin index to the maximum bin. At least one of the recovered luminance samples may be inferred based on the recovered luminance samples and the first displayed value.

[377] In one embodiment, the comparison process (lumaSample<LmcsPivot[idxYInv+1]) included in Table 36 may be performed on all of the bin indices from the minimum bin index to the maximum bin index.

[378] In one embodiment, the inverse display scaling factor may be derived based on at least one of the equations included in Tables 18, 19, 21, and 23, or equations 11 or 12 described above.

[379] In one embodiment, the modified reconstructed luminance samples may be derived based on Equation 21 described above.

[380] In one embodiment, the LMCS related information may include information about the number of bins for outputting mapped prediction luma samples (ie, lmcs_num_bins_minus1 in Table 26). For example, the number of pivot points for displaying brightness may be set to the number of bins. In one example, the encoder may generate input delta values and mapped pivot delta values by number of bins, respectively. In one example, input values and mapped pivot point values are derived based on input delta values (i.e., lmcs_delta_input_cw[i] in Table 26) and mapped delta values (i.e., lmcs_delta_mapped_cw[i] in Table 26), and mapped prediction luma samples may be generated based on input values (i.e., LmcsPivot_input[i] of Table 27 or InputPivot[i] of Table 8) and mapped values (i.e., LmcsPivot_mapped[i] of Table 27 or LmcsPivot[i ] Tables 8).

[381] In one embodiment, an encoder may derive a delta LMCS codeword based on at least one LMCS codeword and an original codeword (OrgCW) included in the LMCS related information, and mapped luminance prediction samples may be derived based on at least one LMCS codeword and the original codeword. In one example, the linear mapping information may include delta LMCS codeword information.

[382] In one embodiment, at least one LMCS codeword may be derived based on the summation of the LMCS delta codeword and OrgCW, for example, OrgCW is (1<<BitDepthY)/16, where BitDepthY represents the luma bit depth. This embodiment may be based on Equation 12.

[383] In one embodiment, at least one LMCS codeword may be derived based on the summation of the LMCS delta codeword and OrgCW*(lmcs_max_bin_idx-lmcs_min_bin_idx+1), for example, lmcs_max_bin_idx and lmcs_min_bin_idx are the maximum bin index and the minimum bin index, respectively , and OrgCW can be (1<<BitDepthY)/16. This embodiment may be based on Equations 15 and 16.

[384] In one embodiment, at least one LMCS codeword may be a multiple of two.

[385] In one embodiment, when the luminance bit depth (BitDepthY) of the reconstructed luminance samples is greater than 10, at least one LMCS codeword may be a multiple of 1<<(BitDepthY-10).

[386] In one embodiment, at least one LMCS codeword may range from (OrgCW>>1) to (OrgCW<<1)-1.

[387] FIG. 16 and FIG. 17 schematically show an example of an image/video decoding method and related components according to an embodiment of the present document. The method disclosed in FIG. 16 may be performed by the decoding apparatus illustrated in FIG. 3. Specifically, for example, S1600 in FIG. 16 may be performed by decoder entropy decoder 310, S1610 may be performed by decoder predictor 330, S1620 may be performed by residual processor 320, predictor 330, and/or decoder adder 340, and S1630 may be performed by decoder adder 340. The method disclosed in FIG. 16 may include the embodiments described above herein.

[388] With reference to FIG. 16, the decoding apparatus can receive/acquire video/image information (S1600). The video/image information may include LMCS related information. For example, LMCS-related information may include luma mapping information (i.e., forward mapping, inverse mapping, linear mapping), chroma residual scaling information, and/or indices (i.e., maximum bin index, minimum bin index, mapping index) related to the LMCS (or reshaping, shaper). The decoding device can receive/receive image/video information via the bitstream.

[389] The image/video information may include various information according to an embodiment of the present document. For example, the image/video information may include information disclosed in at least one of Tables 1-37 described above.

[390] The decoding apparatus may generate prediction brightness samples (S1610). The decoding apparatus may output prediction luminance samples of the current block based on the prediction mode. In this case, various prediction methods disclosed herein, such as inter-prediction or intra-prediction, may be applied.

[391] The image information may include residual information. The decoder may generate residual chrominance samples and/or residual luminance samples based on the residual information. Specifically, the decoding apparatus may derive quantized transform coefficients based on the residual information. The quantized transform coefficients may be in the form of a one-dimensional vector based on the scan order of the coefficients. The decoding apparatus may derive transform coefficients based on a dequantization process for the quantized transform coefficients. The decoder may output chrominance residual samples and/or luminance residual samples based on the transform coefficients.

[392] The decoding apparatus may generate mapped prediction luma samples (S1620). For example, the decoding apparatus may output input values and mapped values (output values) of pivot points for displaying luminance, and may generate mapped prediction luminance samples based on the input values and mapped values. In one example, the decoding device may derive a (forward) display index (idxY) based on the first prediction luminance sample, and the first mapped prediction luminance sample may be generated based on the input value and the mapped pivot value corresponding to the display index. In another example, a linear mapping (linear regeneration, linear LMCS) may be used, and the mapped prediction luma samples may be generated based on the direct mapping scaling factor derived from the two pivot points in the linear mapping, thus the index derivation process may be omitted due to the linear display.

[393] The decoder may generate recovered luminance samples (S1630). The decoder may generate reconstructed luminance samples based on the mapped prediction luminance samples. Specifically, the decoding apparatus may add the residual luminance samples with the mapped prediction luminance samples, and may generate reconstructed luminance samples based on the result of the summation.

[394] The decoder may generate scaled chrominance residual samples. Specifically, the decoding apparatus may derive a chrominance residual scaling factor and generate scaled chrominance residual samples based on the chrominance residual scaling factor. Here, the residual chrominance scaling of the decoding side may be referred to as inverse residual chrominance scaling, as opposed to the encoding side. Accordingly, the chrominance residual scaling factor output by the decoding apparatus may be referred to as an inverse chroma residual scaling factor, and inversely scaled chrominance residual samples may be generated.

[395] The decoder may generate reconstructed chrominance samples. The decoder may generate reconstructed chrominance samples based on the scaled residual chrominance samples. Specifically, the decoding apparatus may perform a prediction process for the chrominance component, and may generate prediction chrominance samples. The decoder may generate reconstructed chroma samples based on the sum of the prediction chroma samples and the scaled residual chroma samples.

[396] In one embodiment, a minimum bin index (ie, lmcs_min_bin_idx) and/or a maximum bin index (ie, LmcsMaxBinIdx) may be derived based on LMCS related information. The first mapped value (LmcsPivot[lmcs_min_bin_idx]) may be inferred based on the minimum bin index. The second mapped value (LmcsPivot[LmcsMaxBinIdx] or LmcsPivot[LmcsMaxBinIdx+1]) may be derived based on the maximum bin index. The values of the reconstructed luma samples (ie, lumaSample of Tables 36 or 37) may range from the first mapped value to the second mapped value. In one example, the values of all recovered luminance samples may range from the first displayed value to the second displayed value. In another example, the values of some of the reconstructed brightness samples may range from the first displayed value to the second displayed value.

[397] In one embodiment, the image information may include a sequence parameter set (SPS). The SPS may include a linear LMCS enabled flag specifying whether linear LMCS is enabled.

[398] In one embodiment, a piece index (ie, idxYInv in Tables 35, 36 or 37) can be identified based on LMCS related information. The decoding device may derive a residual chrominance scaling factor based on the piecewise index. The decoder may generate scaled chrominance residual samples based on the chroma residual samples and the chrominance residual scaling factor.

[399] In one embodiment, the chroma residual scaling factor may be one chroma residual scaling factor.

[400] In one embodiment, the associated information may include an LMCS data field and linear LMCS information. The linear LMCS information may be referred to as linear mapping information. The LMCS data field may include a linear LMCS flag indicating whether linear LMCS is applied. When the linear LMCS flag value is 1, mapped prediction luma samples may be generated based on the linear LMCS information.

[401] In one embodiment, the linear LMCS information includes first pivot information (i.e., P1 in FIG. 11) and second pivot information (i.e., P2 in FIG. 11) . For example, the input value and the displayed value of the first pivot point are the minimum input value and the minimum displayed value, respectively. The input value and the displayed value of the second pivot point are the maximum input value and the maximum displayed value, respectively. Input values between the minimum input value and the maximum input value are linearly mapped.

[402] In one embodiment, the image information includes information about the maximum input value and information about the maximum displayed value. The maximum input value is equal to the maximum input value information value (ie, lmcs_max_input in Table 20). The maximum mapped value is equal to the maximum mapped value information value (ie, lmcs_max_mapped in Table 20).

[403] In one embodiment, the linear mapping information includes information about the input delta value of the second pivot (i.e., lmcs_max_input_delta in Table 22) and information about the mapped delta value of the second pivot (i.e., lmcs_max_mapped_delta in Table 22). The maximum input value may be output based on the input delta value of the second pivot point, and the maximum displayed value may be output based on the displayed delta value of the second pivot point.

[404] In one embodiment, the maximum input value and the maximum displayed value may be derived based on at least one equation included in Table 23 described above.

[405] In one embodiment, generating mapped prediction luminance samples comprises deriving a direct mapping scaling factor (i.e., ScaleCoeffSingle) for the prediction luminance samples and generating mapped prediction luminance samples based on the direct mapping scaling factor. The forward display scaling factor may be one factor for prediction luma samples.

[406] In one embodiment, the inverse display scaling factor may be derived using a piecewise index derived from the recovered luminance samples.

[407] In one embodiment, the piece index may be derived based on Table 36 described above. That is, the comparison process included in Table 36 (lumaSample<LmcsPivot[idxYInv+1]) can be iteratively performed from a piecewise index that is a minimum bin index to a piecewise index that is a maximum bin index.

[408] In one embodiment, modified reconstructed luminance samples may be generated based on mapped values based on bin indices from a minimum bin index to a maximum bin index.

[409] In one embodiment, the bin indices include a first bin index indicating the first mapped value. The first bin index may be inferred based on the mapped values based on the bin indices from the minimum bin index to the maximum bin index and a comparison between the values of the reconstructed luminance samples. At least one of the modified recovered luminance samples may be derived based on the recovered luminance samples and the first displayed value.

[410] In one embodiment, the bin indices include a first bin index indicating the first mapped value, and the first bin index may be derived based on the comparison equation included in Table 36 (lumaSample<LmcsPivot[idxYInv+1]). For example, in the equation above, lumaSample represents the value of the target luminance sample among the reconstructed luminance samples, idxYInv represents one of the bin indices, and LmcsPivot[idxYinv+1] represents one of the mapped values based on the bin indices from the minimum bin index to the maximum bin. At least one of the recovered luminance samples may be inferred based on the recovered luminance samples and the first displayed value.

[411] In one embodiment, the comparison process (lumaSample<LmcsPivot[idxYInv+1]) included in Table 36 may be performed on all of the bin indices from the minimum bin index to the maximum bin index.

[412] In one embodiment, the direct display scaling factor may be derived based on at least one equation included in Tables 21 and/or 23 described above.

[413] In one embodiment, the mapped prediction brightness samples may be derived based on at least one equation included in Table 25 described above.

[414] In one embodiment, the decoder may derive an inverse mapping scaling factor (ie, InvScaleCoeffSingle) for the reconstructed luma samples (ie, lumaSample). Also, the decoder may generate modified reconstructed luminance samples (ie, invSample) based on the reconstructed luminance samples and the inverse mapping scaling factor. The inverse display scaling factor may be one factor for the reconstructed luminance samples.

[415] In one embodiment, the inverse display scaling factor may be derived based on at least one of the equations included in Tables 18, 19, 20, and 23, or Equations 11 or 12 described above.

[416] In one embodiment, the modified reconstructed luminance samples may be derived based on Equation 21 described above.

[417] In one embodiment, the LMCS related information may include information about the number of bins for outputting mapped prediction luma samples (ie, lmcs_num_bins_minus1 in Table 26). For example, the number of pivot points for displaying brightness may be set to the number of bins. In one example, the decoder may generate input delta values and mapped pivot delta values by number of bins, respectively. In one example, input values and mapped pivot point values are derived based on input delta values (i.e., lmcs_delta_input_cw[i] in Table 26) and mapped delta values (i.e., lmcs_delta_mapped_cw[i] in Table 26), and mapped prediction luma samples may be generated based on input values (i.e., LmcsPivot_input[i] of Table 27 or InputPivot[i] of Table 8) and mapped values (i.e., LmcsPivot_mapped[i] of Table 27 or LmcsPivot[i ] Tables 8).

[418] In one embodiment, a decoder may derive a delta LMCS codeword based on at least one LMCS codeword and an original codeword (OrgCW) included in the LMCS related information, and mapped luminance prediction samples may be derived based on at least one LMCS codeword and the original codeword. In one example, the linear mapping information may include delta LMCS codeword information.

[419] In one embodiment, at least one LMCS codeword may be derived based on the summation of the delta LMCS codeword and OrgCW, for example, OrgCW is (1<<BitDepthY)/16, where BitDepthY represents the luma bit depth. This embodiment may be based on Equation 12.

[420] In one embodiment, at least one LMCS codeword may be derived based on the summation of the LMCS delta codeword and OrgCW*(lmcs_max_bin_idx-lmcs_min_bin_idx+1), for example, lmcs_max_bin_idx and lmcs_min_bin_idx are the maximum bin index and the minimum bin index, respectively , and OrgCW can be (1<<BitDepthY)/16. This embodiment may be based on Equations 15 and 16.

[421] In one embodiment, at least one LMCS codeword may be a multiple of two.

[422] In one embodiment, when the luminance bit depth (BitDepthY) of the reconstructed luminance samples is greater than 10, at least one LMCS codeword may be a multiple of 1<<(BitDepthY-10).

[423] In one embodiment, at least one LMCS codeword may range from (OrgCW>>1) to (OrgCW<<1)-1.

[424] In the above embodiment, the methods are described based on a flowchart having a sequence of steps or blocks. The present disclosure is not limited to the order of steps or blocks above. Some steps or blocks may be performed simultaneously or in a different order from other steps or blocks, as described above. Additionally, those skilled in the art will appreciate that the steps shown in the flowchart above are not exclusive, that additional steps may be included, or that one or more steps in the flowchart may be deleted without affecting scope. of this disclosure.

[425] The method according to the above-described embodiments of the present document may be implemented in the form of software, and the encoder and/or decoder according to the present document, for example, may be included in an apparatus that performs image processing of a TV, computer, smartphone, set-top box, display device, etc.

[426] When the embodiments herein are implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. A module may be stored in memory and executed by a processor. The memory may be internal or external to the processor and may be coupled to the processor by various well known means. The processor may include an application specific integrated circuit (ASIC), other chipsets, logic circuits, and/or data processing devices. The memory may include Read Only Memory (ROM), Random Access Memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. That is, the embodiments described herein may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional blocks shown in each figure may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, the instruction information or the algorithm for implementation may be stored in the digital storage medium.

[427] In addition, the decoding device and the encoding device to which the present disclosure applies can be included in a multimedia broadcast transmission/reception device, a mobile communication terminal, a home theater device, a digital cinema device, a surveillance camera, a video talking device , real-time communication device such as video communication, mobile streaming device, storage media, recorder camera, VoD service providing device, direct-to-content (OTT) video delivery device, Internet streaming service providing device, 3D device video, video teleconferencing device, vehicle user equipment (i.e., ground vehicle user equipment, aircraft user equipment, ship user equipment, etc.) and medical video device and can be used to process video signals and data signals. For example, an OTT video delivery device may include a game console, a blue-ray player, an Internet access TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR), and the like.

[428] Moreover, the processing method to which this document applies may be in the form of a program to be executed by a computer and may be stored in a computer-readable recording medium. Multimedia data having a data structure in accordance with the present disclosure may also be stored in computer-readable recording media. Computer-readable recording media includes all types of storage devices that store data that is readable by a computer system. Computer-readable recording media may include, for example, BD, Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical storage. Moreover, computer-readable recording media includes media implemented in the form of a carrier wave (ie, transmission over the Internet). In addition, the bitstream generated by the encoding method may be stored in a computer-readable recording medium, or may be transmitted over wired/wireless communication networks.

[429] In addition, embodiments of the present document may be implemented by a computer program product in accordance with program codes, and the program codes may be executed on a computer by the embodiments of this document. The program codes may be stored on a medium that can be read by a computer.

[430] FIG. 18 shows an example of a content streaming system to which the embodiments disclosed herein may be applied.

[431] With reference to FIG. 18, the content streaming system to which the embodiment(s) of the present document is applied may broadly include an encoding server, a streaming server, a web server, a media storage, a user device, and a media input device.

[432] The encoding server compresses content input from multimedia input devices such as a smartphone, a camera, a camera recorder, and so on. into digital data to generate a bitstream and transmit the bitstream to a streaming server. As an example, when media input devices such as smartphones, cameras, camera recorders, etc. directly generate a bitstream, the encoding server may be omitted.

[433] The bitstream may be generated by the encoding method or the bitstream generation method to which the embodiment(s) of the present disclosure applies, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[434] The streaming server transmits media data to the user's device based on the user's request through the web server, and the web server serves as a medium for informing the user about the service. When the user requests a desired service from the web server, the web server delivers it to the streaming server, and the streaming server transmits the media data to the user. In this case, the content streaming system may include a separate control server. In this case, the control server serves to manage the command/response between devices in the content streaming system.

[435] A streaming server may receive content from a media storage and/or encoding server. For example, when content is received from an encoding server, the content may be received in real time. In this case, in order to provide a smooth streaming service, the streaming server may store the bitstream for a predetermined time.

[436] Examples of a user device may include a mobile phone, smartphone, laptop, digital broadcast terminal, personal digital assistant (PDA), portable media player (PMP), navigator, thin PC, tablet PCs, ultrabooks, wearable devices (e.g., smart watches, smart glasses, head-mounted displays), digital TVs, desktop computer, digital signage, and the like. Each server in the content streaming system may be used as a distributed server, in which case, data received from each server may be distributed.

[437] Each server in the content streaming system may be applied as a distributed server, in which case, data received from each server may be distributed and processed.

[438] The claims described herein may be combined in various ways. For example, the technical features of the claims on the method according to this document can be combined and implemented as a device, and the technical features of the claims on the device according to this document can be combined and implemented as a method. In addition, the technical features of the claims on the method according to this document and the technical features of the claims on the device can be combined to be implemented as a device, and the technical features of the claims on the method according to this document and the technical features of the claims on the device can be combined and implemented as a method. .

Claims (91)

1. An image decoding method performed by a decoding device, the method comprising: obtaining image information including information related to prediction and information related to luminance display with chrominance scaling (LMCS) from the bitstream; generating a prediction brightness sample based on the prediction-related information; generating a reconstructed luminance sample based on the prediction luminance sample; deriving a piecewise function index based on the LMCS related information; and generating a modified reconstructed luminance sample based on the reconstructed luminance sample and the piecewise function index; wherein the minimum bin index and the maximum bin index are derived based on the LMCS related information, moreover, the output of the index of the piecewise function contains: determining whether the first displayed value, in the case where the index of the piecewise function is equal to the minimum index of the bin, is greater than the value of the recovered luminance sample; and determining whether the second mapped value, in the case where the index of the piecewise function is equal to the maximum bin index - 1, is greater than the value of the reconstructed luminance sample; and determining whether the third mapped value, in the case where the piecewise function index is equal to the maximum bin index, is larger than the recovered luminance sample value, based on determining that the second mapped value, in the case where the piecework index is equal to the maximum bin index -1, is not greater than the value of the reconstructed brightness sample. 2. The method of claim 1, wherein the image information includes a sequence parameter set (SPS), and wherein the SPS includes a linear LMCS enabled flag specifying whether linear LMCS is enabled. 3. The method of claim 1, wherein the image information includes residual information, wherein the residual chrominance samples are generated based on the residual information, wherein the residual chrominance scaling factor is derived based on the index of the piecewise function, and wherein the scaled chrominance residual samples are generated based on the chrominance residual samples and the chrominance residual scaling factor. 4. The method of claim 1, wherein the LMCS related information includes an LMCS data field and linear LMCS information, wherein the LMCS data field includes a linear LMCS flag indicating whether linear LMCS is applied, and wherein, when the value of the linear LMCS flag is 1, a mapped prediction luminance sample is generated based on the linear LMCS information. 5. The method of claim 4, wherein the linear LMCS information includes first turning point information and second turning point information, wherein the input value and the displayed value of the first pivot point are the minimum input value and the minimum displayed value, respectively, wherein the input value and the displayed value of the second pivot point are the maximum input value and the maximum displayed value, respectively, and wherein the input values between the minimum input value and the maximum input value are linearly mapped. 6. The method of claim 4, wherein the image information includes maximum input value information and maximum displayed value information, wherein the maximum input value is equal to the maximum input value information value, and wherein the maximum displayed value is equal to the information value of the maximum displayed value. 7. The method of claim 4, wherein the linear mapping information includes input delta value information of the second pivot point and information of the reflected delta value of the second pivot point, wherein the input value delta information of the second pivot point is the difference between the input value of the second pivot point and the maximum input value, wherein the information about the displayed delta value of the second pivot point is the difference between the displayed value of the second pivot point and the maximum displayed value, wherein the maximum input value is derived based on the delta input value of the second pivot point, and wherein the maximum displayed value is derived based on the displayed delta value of the second pivot point. 8. The method of claim 1, wherein the method further comprises generating a mapped prediction luma sample, and wherein the generation of the mapped prediction brightness sample comprises: deriving the direct mapping scaling factor for the prediction brightness sample: generating a mapped prediction luminance sample based on the direct mapping scaling factor, and wherein the direct display scaling factor is one factor for sampling the prediction luminance. 9. The method of claim 1, wherein the second mapped value, in the case where the index of the piecewise function is equal to the maximum bin index -1, is the mapped value in the bin indicated by the maximum bin index, and wherein the third mapped value, in the case where the index of the piecewise function is equal to the maximum bin index, is the mapped value in the bin indicated by the maximum bin index +1. 10. The method of claim 1, wherein deriving the index of the piecewise function comprises executing the operation of the following syntax

where the syntax idxYInv represents the index of the piecewise function, lmcs_min_bin_idx represents the minimum bin index, LmcsMaxBinIdx represents the maximum bin index, lumaSample represents the reconstructed luma sample value, and LmcsPivot[idxYInv+1] represents the mapped value in the idxYInv+1st bin. 11. The method according to claim 1, and the minimum bin index is not equal to 0. 12. The method of claim 1, wherein the LMCS related information includes information about the number of bins for outputting prediction luminance mapped samples, wherein the number of pivot points for displaying brightness is set to the number of bins, wherein the input delta pivot values and the displayed delta pivot values are signaled in an amount equal to the number of bins, wherein the pivot point input values and the pivot point mapped values are derived based on the pivot point delta input values and the pivot point delta mapped values, and wherein the mapped prediction luminance samples are generated based on the input pivot values and the mapped pivot values, wherein the input delta pivot values are the input delta pivot codeword values and the mapped delta pivot values are the mapped delta pivot codeword values. 13. An image encoding method performed by an encoding device, the method comprising: generating a prediction brightness sample; generating a mapped prediction luminance sample based on the prediction luma sample; generating information related to luminance chrominance scaling (LMCS) display based on the mapped prediction luminance sample; generating a residual luminance sample based on the mapped prediction luminance sample; outputting residual information based on the luminance residual sample; and encoding image information including LMCS related information and residual information, wherein the image encoding method further comprises: generating a reconstructed luminance sample based on the predicted luminance sample; deriving a piecewise function index based on the LMCS related information; and generating a modified reconstructed luminance sample based on the reconstructed luminance sample and the piecewise function index, wherein the minimum bin index and the maximum bin index are derived based on the LMCS related information, and the output of the index of the piecewise function contains: determining whether the first mapped value, in the case where the index of the piecewise function is equal to the minimum index of the bin, is greater than the value of the reconstructed luminance sample; and determining whether the second mapped value, in the case where the index of the piecewise function is equal to the maximum bin index -1, is larger than the recovered luminance sample value; and determining whether the third mapped value, in the case where the piecewise function index is equal to the maximum bin index, is larger than the recovered luminance sample value, based on determining that the second mapped value, in the case where the piecework index is equal to the maximum bin index -1, is not greater than the value of the reconstructed brightness sample. 14. A non-transitory computer-readable digital storage medium storing a bitstream generated by a method, the method comprising: generating a prediction brightness sample; generating a mapped prediction luminance sample based on the prediction luma sample; generating information related to luminance chrominance scaling (LMCS) display based on the mapped prediction luminance sample; generating a residual luminance sample based on the mapped prediction luminance sample; outputting residual information based on the luminance residual sample; and encoding image information including LMCS related information and residual information, wherein the method further comprises: generating a reconstructed luminance sample based on the predicted luminance sample; deriving a piecewise function index based on the LMCS related information; and generating a modified reconstructed luminance sample based on the reconstructed luminance sample and the piecewise function index, wherein the minimum bin index and the maximum bin index are derived based on the LMCS related information, and the output of the index of the piecewise function contains: determining whether the first mapped value, in the case where the index of the piecewise function is equal to the minimum index of the bin, is greater than the value of the reconstructed luminance sample; and determining whether the second mapped value, in the case where the index of the piecewise function is equal to the maximum bin index -1, is larger than the recovered luminance sample value; and determining whether the third mapped value, in the case where the piecewise function index is equal to the maximum bin index, is larger than the recovered luminance sample value, based on determining that the second mapped value, in the case where the piecework index is equal to the maximum bin index -1, is not greater than the value of the reconstructed brightness sample. 15. A data transmission method for image information, comprising: obtaining a bitstream of image information including information related to luminance mapping with chrominance scaling (LMCS) and residual information, wherein the LMCS related information is generated by generating a prediction luminance sample, generating a mapped prediction luminance sample based on the prediction luminance sample, and outputting LMCS-related information based on the mapped prediction luminance sample, wherein the residual information is generated by generating a residual luminance sample based on the mapped prediction luma sample and outputting residual information based on the residual luma sample; and transmitting data containing a bitstream of image information including LMCS related information and residual information, wherein the reconstructed luminance sample is generated based on the predicted luminance sample; wherein the index of the piecewise function is derived based on the LMCS related information; and wherein the modified reconstructed luminance sample is generated based on the reconstructed luminance sample and the index of the piecewise function; wherein the minimum bin index and the maximum bin index are derived based on the LMCS related information, moreover, to display the index of a piecewise function, it is determined whether the first mapped value, in the case where the index of the piecewise function is equal to the minimum index of the bin, is greater than the value of the reconstructed luminance sample; and it is determined whether the second mapped value, in the case where the index of the piecewise function is equal to the maximum bin index -1, is greater than the value of the reconstructed luminance sample; and it is determined whether the third mapped value, in the case where the piecewise function index is equal to the maximum bin index, is greater than the recovered luminance sample value, based on determining that the second mapped value, in the case where the piecework index is equal to the maximum bin index -1, is not greater than the value of the reconstructed brightness sample.

Images