RU2810200C2 - Method and device for image coding based on sub-image - Google Patents

Abstract

FIELD: image encoding and decoding.

SUBSTANCE: residual samples for the current block are generated. Residual information is generated based on the residual samples for the current block. Information related to in-loop filtering is generated for the restored samples of the current picture. Image information including residual information and information related to in-loop filtering is encoded. Moreover, the image information includes a sequence parameter set (SPS) and a picture header. When the SPS includes information related to the sub-picture, information about the number of virtual borders and information about the positions of virtual borders are not included in the picture header, but are included in the SPS.

EFFECT: increased efficiency of image encoding.

8 cl, 14 dwg, 12 tbl

Description

Disclosure area

[1] This document relates to an apparatus and method for encoding an image based on a sub-picture.

Related technical field

[2] Recently, the need for high-definition, high-quality images/videos, such as ultra-high-definition (UHD) 4K or 8K or higher images/videos, has increased in various fields. Since the image/video data is of high resolution and high quality, the amount of information or bits to be transmitted is increased relative to the existing image/video data, and thus transmitting the image data using a medium such as an existing wired/wireless broadband line or existing storage medium, or storing image/video data using an existing storage medium increases transmission costs and storage costs.

[3] In addition, interest and demand for immersive media such as virtual reality (VR) and artificial reality (AR) content or holograms have increased recently, and the broadcast of images/videos having characteristics different from reality images has increased. , such as game images.

[4] Accordingly, a highly efficient image/video compression technology is required to efficiently compress, transmit, store and reproduce high-resolution, high-quality image/video information having various characteristics as described above.

[5] In particular, to improve the subjective/objective visual quality, a picture is composed of sub-pictures, and schemes to improve the efficiency of signaling information related to a sub-picture are discussed.

Brief description of the invention

[6] According to an embodiment of the present document, a method and apparatus for improving image encoding efficiency are provided.

[7] According to an embodiment of the present document, a method and apparatus for applying effective filtration are provided.

[8] In accordance with an embodiment of the present document, a method and apparatus for effectively applying deblocking, adaptive sample offset (SAO), and adaptive loop filtering (ALF) are provided.

[9] According to an embodiment of this document, in-loop filtering may be performed based on virtual boundaries.

[10] According to an embodiment of the present document, signaling of information related to a virtual boundary can be performed based on information related to a sub-picture.

[11] According to an embodiment of the present document, an encoding apparatus for performing video/image encoding is provided.

[12] According to one embodiment of the present document, a computer-readable digital storage medium is provided that stores encoded video/image information generated in accordance with a video/image encoding method disclosed in at least one embodiment of the present document.

[13] According to an embodiment of the present document, a computer-readable digital storage medium is provided in which encoded information or encoded video/image information causing execution of a video/image decoding method disclosed in at least one embodiment of the present document is stored. through a decoding device.

[14] According to an embodiment of the present document, the overall efficiency of image/video compression can be improved.

[15] According to an embodiment of the present document, the subjective/objective visual quality can be improved through effective filtering.

[16] According to an embodiment of the present document, efficient coding can be realized by omitting the bit rewriting process through signaling based on virtual sub-picture boundary signaling.

Brief description of drawings

[17] FIG. 1 schematically shows an example of a video/image coding system to which embodiments of the present document may be applied.

[18] FIG. 2 is a view schematically illustrating the configuration of a video/image encoding apparatus to which embodiments of the present document may be applied.

[19] FIG. 3 is a view schematically illustrating the configuration of a video/image decoding apparatus to which embodiments of the present document may be applied.

[20] FIG. 4 shows an example of a hierarchical architecture for an encoded video/image.

[21] FIG. 5 illustrates a picture in accordance with an embodiment of the present document.

[22] FIG. 6 illustrates a subpicture/slice/tile based encoding method according to an embodiment of the present document.

[23] FIG. 7 illustrates a subpicture/slice/tile based decoding method according to an embodiment of the present document.

[24] FIG. 8 is a flowchart for explaining a filter-based encoding method in an encoding apparatus.

[25] FIG. 9 is a flowchart for explaining a filtering-based decoding method in a decoding apparatus.

[26] FIG. 10 and fig. 11 schematically shows an example of a video/image encoding method and related components in accordance with embodiment(s) of the present document.

[27] FIG. 12 and fig. 13 schematically shows an example of a video/image decoding method and related components in accordance with embodiment(s) of the present document.

[28] FIG. 14 shows an example of a content streaming system in which the embodiments disclosed herein may be applied.

Description of Exemplary Embodiments

[29] Each configuration in the drawings described herein is an independent illustration for explaining functions as features that are different from each other, and does not mean that each configuration is implemented as mutually different hardware or different software. For example, two or more of the configurations may be combined to form one configuration, or one configuration may be divided into multiple configurations. Without departing from the spirit of this document, embodiments in which configurations are combined and/or separated are also included within the scope of the claims.

[30] Meanwhile, the present document may be modified in various forms, and specific embodiments thereof will be described and illustrated in the drawings. However, the embodiments are not intended to limit the present document. The terms used in the following description are used to only describe specific embodiments and are not intended to limit the disclosure. An expression in the singular includes an expression in the plural unless it is expressly read otherwise. Terms such as “include” and “have” are intended to indicate that the features, numbers, steps, operations, elements, components, or combinations thereof used in the following description exist, and thus it is to be understood that the possibility of the existence or addition of one or more other features, numbers, steps, operations, elements, components or combinations thereof is not excluded.

[31] Next, examples of the present embodiment will be described in detail with reference to the accompanying drawings. In addition, the same reference numerals are used to indicate the same elements throughout the drawings, and the same descriptions of the same elements will be omitted.

[32] This document relates to video/image coding. For example, the methods/embodiments disclosed herein may relate to the Multipurpose Video Coding (VVC) standard (ITU-T Rec. H.266), the next generation video/image coding standard after VVC, or other standards related to video coding (eg, High Efficiency Video Coding (HEVC) standard (ITU-T Rec. H.265), Essential Video Coding (EVC) standard, AVS2 standard, and the like).

[33] This document proposes various embodiments of video/image encoding, and the embodiments above may also be performed in combination with each other unless otherwise specified.

[34] As used herein, video may refer to a sequence of images over time. A picture usually refers to a unit representing one image in a particular time frame, and a cut/tile refers to a unit that constitutes a part of a picture in encoding terms. The clipping/tile may include one or more coding tree units (CTUs). One picture may consist of one or more cutouts/mosaic elements. One picture can consist of one or more groups of mosaic elements. One tile group may include one or more tiles.

[35] Pixel or pel can mean the smallest unit that makes up one picture (or image). Also, 'sample' can be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a luma component pixel/pixel value or only a chrominance component pixel/pixel value.

[36] The unit may represent a basic image processing unit. The unit may include at least one of a specific picture area and information related to the area. One unit may include one luma block and two chrominance blocks (eg, cb, cr). Unit may be used interchangeably with terms such as block or area in some cases. In general, an M×N block may include samples (or arrays of samples) or a set (or array) of transform coefficients of M columns and N rows. Alternatively, a sample may denote a pixel value in the spatial domain, and when such a pixel value is converted to the frequency domain, it may denote a frequency domain conversion coefficient.

[37] As used herein, the designations “/” and “,” shall be interpreted to indicate “and/or.” For example, the expression “A/B” could mean “A and/or B”. Further, “A, B” may mean “A and/or B”. Additionally, “A/B/C” may mean “at least one of A, B, and/or C.” Also, “A/B/C” may mean “at least one of A, B and/or C”.

[38] Additionally, as used herein, the term “or” shall be interpreted to mean “and/or.” For example, the expression “A or B” may contain 1) only A, 2) only B, and/or 3) both A and B. In other words, in this document, the term “or” shall be interpreted to indicate “in addition or alternatively.”

[39] As used herein, “at least one of A and B” may mean “A only,” “B only,” or “both A and B.” Also, in the present specification, the expression “at least one of A or B” or “at least one of A and/or B” can be interpreted in the same way as “at least one of A and B”.

[40] Also, as used herein, “at least one of A, B, and C” may mean “A only,” “B only,” “C only,” or “any combination of A, B, and C.” Also, “at least one of A, B, or C” or “at least one of A, B, and/or C” may mean “at least one of A, B, and C.”

[41] Also, parentheses used herein may mean “for example.” Specifically, when “prediction (intra-prediction)” is specified, it may indicate that “intra-prediction” is proposed as an example of “prediction”. In other words, the term “prediction” in the present specification is not limited to “intra-prediction”, and “intra-prediction” may be offered as an example of “prediction”. Also, even when “prediction (ie, intra-prediction)” is specified, this may indicate that “intra-prediction” is being offered as an example of “prediction”.

[42] In the present description, technical features separately explained in one drawing may be implemented separately or may be implemented simultaneously.

[43] FIG. 1 illustrates an example of a video/image coding system to which the disclosure herein may be applied.

[44] With reference to FIG. 1, a video/image encoding system may include a source device and a receiving device. A source device may transmit encoded information or video/image data to a receiving device via a digital storage medium or network in the form of a file or stream.

[45] The source device may include a video source, an encoder, and a transmitter. The receiving device may include a receiver, a decoding device, and a rendering device. The encoding device may be called a video/image encoding device, and the decoding device may be called a video/image decoding device. The transmitter may be included in the encoding device. The receiver may be included in a decoding device. The imaging device may include a display, and the display may be configured as a separate device or an external component.

[46] The video source may obtain video/image through a video/image capture, synthesis, or generation process. The video source may include a video/image capturing device and/or a video/image generating device. The video/image capturing device may include, for example, one or more cameras, video/image archives including previously captured video/images, and the like. The video/image generating apparatus may include, for example, computers, tablets, and smartphones and may (electronically) generate video/images. For example, the virtual video/image may be generated through a computer or the like. In this case, the video/image capture process can be replaced by the associated data generation process.

[47] The encoding device can encode the input video/image. The encoding device may perform a series of processes such as prediction, transformation and quantization for compact and efficient encoding. The encoded data (encoded video/image information) may be output in the form of a bit stream.

[48] A transmitter may transmit encoded image/image information or data output in the form of a bitstream to a receiver of a receiving device via a digital storage medium or network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD and the like. The transmitter may include an element for generating a media file through a predefined file format and may include an element for transmission over a broadcast/communications network. The receiver can receive/extract the bit stream and transmit the received bit stream to the decoding device.

[49] A decoding apparatus may decode a video/image by performing a series of processes such as dequantization, deconversion, and prediction corresponding to an operation of the encoding apparatus.

[50] The renderer can render the decoded video/image. The rendered video/image can be displayed via the display.

[51] FIG. 2 is a diagram schematically illustrating the configuration of a video/image encoding apparatus to which the disclosure of the present document can be applied. Further, what is called a video encoding device may include an image encoding device.

[52] With reference to FIG. 2, encoding apparatus 200 may include or be configured with an image partitioning module 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-predictor 221 and an intra -predictor 222. Residual processor 230 may include transformer 232, quantizer 233, dequantizer 234, and inverse transformer 235. Residual processor 230 may further include subtractor 231. Adder 250 may be referred to as a reconstructor or reconstructed block generator. The image partitioning unit 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 that were described above may be configured by one or more hardware components (eg, encoder chipsets or processors) in accordance with an embodiment. In addition, memory 270 may include a decoded picture buffer (DPB) or may be configured with digital storage media. The hardware component may further include memory 270 as an internal/external component.

[53] The image splitter 210 may split an input image (or picture, frame) input to the encoder 200 into one or more processing units. For example, a processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively partitioned according to a quadtree-binary-tree-ternary (QTBTTT) structure from a coding tree unit (CTU) or a largest coding unit (LCU). For example, one encoding unit may be decomposed into multiple encoding units of greater depth based on a quadtree structure, a binary tree structure, and/or a ternary tree structure. In this case, for example, the quadtree structure is applied first, and the binary tree structure and/or ternary tree structure may be applied later. Alternatively, the binary tree structure can also be applied first. The encoding process of this document may be performed based on the final encoding unit, which is no longer split. In this case, the largest coding unit can be directly used as the final coding unit based on the coding efficiency according to the characteristics of the image, or, as necessary, the coding unit can be recursively divided into coding units of deeper depth, so that a coding unit having an optimal size , can be used as the final encoding unit. Here, the encoding process may include a prediction, transformation and reconstruction process, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transformation unit (TU). In this case, the prediction unit and the transformation unit may be separated or split from the above-mentioned final encoding unit. The prediction unit may be a sample prediction unit, and the transformation unit may be a unit for deriving a transform coefficient and/or a unit for deriving a residual signal from the transform coefficient.

[54] Unit may be used interchangeably with terms such as block or area in some cases. In general, an M×N block can represent samples consisting of M columns and N rows or a group of transform coefficients. A sample may generally represent a pixel or a pixel value, and may also represent only a luma component pixel/pixel value or represent only a chroma component pixel/pixel value. A sample can be used as a term corresponding to a pixel or pel configuring a single picture (or image).

[55] Subtractor 231 may generate a residual signal (residual block, residual samples, or residual sample array) by subtracting a prediction signal (predicted block, prediction samples, or prediction sample array) output from predictor 220 from an input image signal (original block, original samples or the original array of samples), and the generated residual signal is transmitted to transformer 232. Predictor 220 may perform a prediction on a target processing block (hereinafter referred to as a current block) and generate a predicted block including prediction samples for the current block. Predictor 220 may determine whether intra-prediction or inter-prediction is applied on the current block or CU. As described later in the description of each prediction mode, the predictor may generate various kinds of prediction-related information, such as prediction mode information, and transmit the generated information to the entropy encoder 240. The prediction information may be encoded in the entropy encoder 240 and output in the form of a bit stream. .

[56] Intra predictor 222 may predict the current block by referring to samples in the current picture. The accessed samples may be located adjacent to the current block or may be spaced apart from the current block in accordance with the prediction mode. In intra-prediction, prediction modes may include a plurality of undirected modes and a plurality of directed modes. The omnidirectional mode may include, for example, a DC mode or a planar mode. The directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes according to the granularity of the prediction direction. However, this is only an example, and more or less directional prediction modes can be used according to the setting. Intra predictor 222 may also determine the prediction mode applied to the current block using the prediction mode applied to the adjacent block.

[57] The inter-predictor 221 may output a predicted block of the current block based on a reference block (reference sample array) determined by a motion vector in the reference picture. Here, in order to reduce the amount of motion information transmitted in the inter-prediction mode, motion information may be predicted in block, sub-block, or sample units based on the correlation of motion information between an adjacent block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter-prediction direction information (L0 prediction, L1 prediction, Bi-prediction, etc.). In the case of inter-prediction, the neighbor block may include a spatial neighbor block existing in the current picture and a temporal neighbor block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporary adjacent block may be the same or different. A temporary neighbor block may be called a co-located reference block, a co-located CU (colCU), etc., and a reference picture including a temporary neighboring block may be called a co-located picture (colPic). For example, inter-predictor 221 may configure a list of motion information candidates based on neighboring blocks and generate information indicating which candidate is used to output the motion vector and/or reference picture index of the current block. Inter-prediction can be performed based on various prediction modes, and, for example, in the case of a skip mode and a merge mode, the inter-predictor 221 can use the motion information of a neighboring block as the motion information of the current block. In the skip mode, unlike the combine mode, the residual signal may not be transmitted. A motion vector prediction (MVP) mode may indicate the motion vector of a current block using the motion vector of an adjacent block as a motion vector predictor and signaling the motion vector difference.

[58] Predictor 220 may generate a prediction signal based on various prediction methods described below. For example, a predictor may not only apply intra-prediction or inter-prediction to predict one block, but also simultaneously apply both intra-prediction and inter-prediction. This may be called combined inter- and intra-prediction (CIIP). In addition, the predictor can perform intra-block copying (IBC) to predict a block. Intra-block copying may be used for encoding a content image/moving image of a game or the like, such as screen content encoding (SCC). IBC primarily performs prediction in the current picture, but can be performed similarly to inter-prediction in that the reference block is output in the current picture. That is, the IBC may use at least one of the inter-prediction methods described herein.

[59] The prediction signal generated by inter-predictor 221 and/or intra-predictor 222 can be used to generate a reconstructed signal or generate a residual signal. Converter 232 may generate transform coefficients by applying a transform method to the residual signal. For example, the transform method may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a graph-based transform (GBT), or a conditional nonlinear transform (CNT). Here, GBT means the transformation obtained from a graph when the relation information between pixels is represented by the graph. CNT refers to the transformation obtained from the prediction signal generated using all the previously reconstructed pixels. In addition, the transformation process can be applied to square blocks of pixels that are the same size, or can be applied to blocks that are variable in size rather than square.

[60] Quantizer 233 may quantize transform coefficients and transmit them to entropy encoder 240, and entropy encoder 240 may encode the quantized signal (quantized transform coefficient information) and output a bit stream. Information about the quantized transform coefficients may be called remainder information. Quantizer 233 may reorder the quantized block type transform coefficients into one-dimensional vector form based on the scanning order of the coefficients, and generate information about the quantized transform coefficients based on the quantized transform coefficients in one-dimensional vector form. Entropy encoder 240 may perform various encoding techniques, such as, for example, exponential Golomb coding, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), and the like. Entropy encoder 240 may encode information needed to reconstruct the video/image, together with or separately from the quantized transform coefficients (eg, syntax element values, etc.). Encoded information (eg, encoded video/image information) may be transmitted or stored in Network Abstraction Layer (NAL) units in the form of a bit stream. The video/image information may further include information about various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/image information may further include general restriction information. Herein, information and/or syntax elements transmitted/signaled as described below may be encoded by the above-described encoding process and included in the bitstream. The bitstream may be transmitted over a network or may be stored in a digital storage medium. The network may include a broadcast network and/or a communications network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD and the like. A transmitter (not shown) transmitting the signal output from the entropy encoder 240 and/or a storage module (not shown) storing the signal may be configured as an internal/external element of the encoder 200, and alternatively, the transmitter may be included in the entropy encoder 240.

[61] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, a residual signal (residual block or residual samples) may be recovered by applying dequantization and inverse transform to the quantized transform coefficients by dequantizer 234 and inverse transformer 235. Adder 250 adds the reconstructed residual signal to the prediction signal output from predictor 220 to generate a reconstructed signal ( restored picture, restored block, restored samples or restored array of samples). If there is no remainder for the target processing block, as is the case when the skip mode is applied, the predicted block can be used as a reconstructed block. The generated reconstructed signal can be used to intra-predict the next processing target block in the current picture, and can be used to inter-predict the next picture by filtering, as described below.

[62] Meanwhile, luma mapping with chroma scaling (LMCS) can be applied during the encoding and/or reconstruction process of a picture.

[63] Filter 260 may improve subjective/objective image quality by applying filtering to the reconstructed signal. For example, filter 260 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture and store the modified reconstructed picture in memory 270, specifically in DPB memory 270. Various filtering methods may include, for example, deblocking filtering, adaptive sampling offset (SAO), adaptive loop filter, bidirectional filter and the like. Filter 260 may generate various filtering-related information and transmit the generated information to entropy encoder 290, as described later in the description of each filtering method. Filtering-related information may be encoded by entropy encoder 290 and output in the form of a bitstream.

[64] The modified reconstructed picture supplied to the memory 270 can be used as a reference picture in the inter-predictor 221. When inter-prediction is applied by the encoding device, prediction mismatch between the encoding device 200 and the decoding device 300 can be avoided, and encoding efficiency can be improved. .

[65] DPB memory 270 The DPB may store a modified reconstructed picture for use as a reference picture in the inter-predictor 221. Memory 270 may store block motion information from which motion information in the current picture is derived (or encoded) and/or motion information blocks in the picture that have already been restored. The stored motion information may be transmitted to the inter-predictor 221 to be used as spatial neighbor motion information or temporal neighbor motion information. Memory 270 may store reconstructed samples of reconstructed blocks in the current picture and may transfer the reconstructed samples to intra-predictor 222.

[66] FIG. 3 is a diagram schematically illustrating the configuration of a video/image decoding apparatus to which the disclosure of the present document can be applied.

[67] With reference to FIG. 3, decoding apparatus 300 may include and be configured with an entropy decoder 310, a remainder processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-predictor 331 and an intra-predictor 332. Processor Residual 320 may include a dequantizer 321 and an inverse converter 322. The entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 that were described above may be configured by one or more hardware components (e.g., decoder chipsets or processors ) according to an embodiment. In addition, memory 360 may include a decoded picture buffer (DPB) or may be configured with a digital storage medium. The hardware component may further include memory 360 as an internal/external component.

[68] When a bitstream including video/image information is input, the decoding apparatus 300 can reconstruct an image in response to a process in which the video/image information is processed in the encoding apparatus shown in FIG. 2. For example, the decoding apparatus 300 may output units/blocks based on the block division related information obtained from the bitstream. The decoding apparatus 300 may perform decoding using a processing unit applied in the encoding apparatus. Thus, the processing unit for decoding may be, for example, an encoding unit, and the encoding unit may be divided according to a quadtree structure, a binary tree structure, and/or a ternary tree structure from the encoding tree unit or the largest encoding unit. One or more transformation units may be derived from a coding unit. Further, the reconstructed image signal decoded and output by the decoding apparatus 300 can be reproduced by the playback apparatus.

[69] The decoding apparatus 300 may receive a signal output from the encoding apparatus according to FIG. 2, in the form of a bitstream, and the received signal may be decoded by an entropy decoder 310. For example, the entropy decoder 310 may parse the bitstream to output information (e.g., video/image information) needed to reconstruct an image (or reconstruct a picture) . The video/image information may further include information about various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/image information may further include general restriction information. The decoding apparatus may further decode the picture based on the parameter set information and/or general constraint information. The signaled/received information and/or syntax elements described later herein may be decoded during the decoding process and obtained from the bit stream. For example, entropy decoder 310 decodes information in a bit stream based on an encoding method such as exponential Golomb coding, CAVLC, or CABAC, and outputs syntax elements required for image reconstruction and quantized values of transform coefficients for the remainder. More specifically, the CABAC entropy decoding method may receive a bin corresponding to each syntactic element in a bit stream, determine a context model using decoding information of a target syntactic element, decoding information for decoding a target block, or information of a symbol/bin decoded in a previous stage, and perform arithmetic decoding on a bin by predicting the probability of occurrence of the bin according to a certain context model, and generating a symbol corresponding to the meaning of each syntactic element. In this case, the CABAC entropy decoding method can update the context model using the decoded symbol/bin information for the context model of the next symbol/bin after determining the context model. Prediction-related information, among the information decoded by the entropy decoder 310, may be provided to the predictor 330, and information about the remainder on which entropy decoding has been performed in the entropy decoder 310, that is, quantized transform coefficients and related parameter information, may be input to dequantizer 321. In addition, filtering information among the information decoded by the entropy decoder 310 may be provided to the filter 350. Meanwhile, a receiver (not shown) for receiving a signal output from the encoding device may be further configured as an internal/external element decoding device 300, or the receiver may be a component of the entropy decoder 310. Meanwhile, the decoding device according to this document may be called a video/image/picture decoding device, and the decoding device may be classified into an information decoder (video/image/picture information decoder) and a sampling decoder (video/image/picture sampling decoder). The information decoder may include an entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverter 322, a predictor 330, an adder 340, a filter 350, and a memory 360.

[70] Dequantizer 321 may dequantize the quantized transform coefficients to output transform coefficients. The dequantizer 321 may reorder the quantized transform coefficients in the form of a two-dimensional block. In this case, reordering may be performed based on the scanning order of the coefficients performed by the encoder. The dequantizer 321 may perform dequantization on the quantized transform coefficients using a quantization parameter (eg, quantization step size information) and obtain the transform coefficients.

[71] The inverse converter 322 inversely converts the transform coefficients to obtain a residual signal (residual block, residual sample array).

[72] Predictor 330 may perform prediction on the current block and generate a predicted block including prediction samples of the current block. The predictor can determine whether intra-prediction or inter-prediction is applied to the current block based on the prediction information output from the entropy decoder 310, and can determine a particular intra/inter-prediction mode.

[73] Predictor 320 may generate a prediction signal based on various prediction methods described below. For example, a predictor may not only apply intra-prediction or inter-prediction to predict one block, but also simultaneously apply intra-prediction and inter-prediction. This may be called combined inter- and intra-prediction (CIIP). In addition, the predictor can perform intra-block copying (IBC) to predict a block. Intra-block copying may be used for encoding a content image/moving image of a game or the like, such as screen content encoding (SCC). IBC primarily performs prediction in the current picture, but can be performed similarly to inter-prediction in that the reference block is output in the current picture. That is, the IBC may use at least one of the inter-prediction methods described herein.

[74] Intra predictor 332 may predict the current block by reference to samples in the current picture. These samples may be located adjacent to the current block or may be spaced apart from the current block in accordance with the prediction mode. In intra-prediction, prediction modes may include a plurality of undirected modes and a plurality of directed modes. Intra predictor 332 may determine the prediction mode applied to the current block using the prediction mode applied to the adjacent block.

[75] The inter-predictor 331 may output a predicted block for the current block based on a reference block (reference sample array) determined by a motion vector in the reference picture. In this case, in order to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter-prediction direction information (L0 prediction, L1 prediction, Bi-prediction, etc.). In the case of inter-prediction, the neighbor block may include a spatial neighbor block existing in the current picture and a temporal neighbor block existing in the reference picture. For example, the inter-predictor 331 may construct a motion information candidate list based on neighboring blocks and output a motion vector of the current block and/or a reference picture index based on the received candidate selection information. Inter-prediction may be performed based on various prediction modes, and the prediction information may include information indicating an inter-prediction mode for the current block.

[76] Adder 340 may generate a reconstructed signal (reconstructed picture, reconstructed block, or reconstructed sample array) by adding the resulting residual signal to the prediction signal (predicted block or predicted sample array) output from predictor 330. If there is no residual for the target processing block, as in the case when the skip mode is applied, the predicted block can be used as a reconstructed block.

[77] The adder 340 may be referred to as a reconstructor or reconstructed block generator. The generated reconstructed signal can be used for intra-prediction of the next block to be processed in the current picture, and, as described below, can also be output by filtering or can be used for inter-prediction of the next picture.

[78] Meanwhile, luminance chroma scaling (LMCS) display can be applied in the picture decoding process.

[79] Filter 350 may improve subjective/objective image quality by applying filtering to the reconstructed signal. For example, filter 350 may generate a modified reconstructed picture by applying various filtering methods to the reconstructed picture and store the modified reconstructed picture in memory 360, specifically in DPB memory 360. Various filtering methods may include, for example, deblocking filtering, adaptive sampling offset , adaptive loop filter, bidirectional filter and the like.

[80] The (modified) reconstructed picture stored in the DPB memory 360 may be used as a reference picture in the inter-predictor 331. The memory 360 may store motion information of a block from which motion information in the current picture is derived (or decoded), and/or information about the movement of blocks in the picture that have already been restored. The stored motion information may be transmitted to the inter-predictor 331 for use as spatial neighbor motion information or temporal neighbor motion information. Memory 360 may store reconstructed samples of reconstructed blocks in the current picture and transfer the reconstructed samples to intra-predictor 332.

[81] In the present description, the embodiments described in the predictor 330, dequantizer 321, inverse transformer 322, and filter 350 of the decoding device 300 may be applied in a similar manner or correspond to the predictor 220, quantizer 234, inverse transformer 235, and filter 260 of the encoding device 200.

[82] As described above, in performing video encoding, prediction is performed to improve compression efficiency. By doing this, a predicted block including prediction samples for the current block as a block to be encoded (ie, an encoding target block) can be generated. Here, the predicted block includes prediction samples in a spatial domain (or pixel domain). The predicted block is output equally in the encoding device and the decoding device, and the encoding device can signal information (residual information) about the remainder between the original block and the predicted block, rather than the original sample value of the original block, to the decoding device, thereby improving the efficiency of image encoding. The decoding apparatus may output a residual block including residual samples based on the residual information, add the residual block and the predicted block to generate reconstructed blocks including the reconstructed samples, and generate a reconstructed picture including the reconstructed blocks.

[83] Residual information can be generated during the conversion and quantization process. For example, the encoding device may output a residual block between an original block and a predicted block, and perform a transform process on the residual samples (residual array of samples) included in the residual block to output transform coefficients, perform a quantization process on the transform coefficients to output quantized transform coefficients, and signal the associated residual information to the decoding device (via a bit stream). Here, the residual information may include quantized transform coefficient value information, location information, transform method, transform kernel, quantization parameter, and the like. The decoding apparatus may perform a dequantization/deconversion process based on the residual information and output residual samples (or residual blocks). The decoding apparatus can generate a reconstructed picture based on the predicted block and the residual block. The encoder may also dequantize/deconvert the quantized transform coefficients to reference inter-prediction of a later picture to output a residual block and generate a reconstructed picture from it.

[84] Herein, at least one of quantization/dequantization and/or transform/inverse transform may be omitted. When quantization/dequantization is omitted, the quantized conversion coefficient may be referred to as the conversion coefficient. When the transform/inverse transform is omitted, the conversion factor may be called the coefficient or residual coefficient, or may still be called the conversion factor for consistency of expression.

[85] Herein, the quantized transform coefficient and the transform coefficient may be referred to as the transform coefficient and the scaled transform coefficient, respectively. In this case, the residual information may include information about the transform coefficient(s), and the information about the transform coefficient(s) may be signaled through a residual encoding syntax. Transform coefficients may be derived based on the residual information (or information about the transform coefficient(s)), and scaled transform coefficients may be derived via inverse transform (scaling) on the transform coefficients. Residual samples can be inferred based on the inverse transform (transform) of the scaled transform coefficients. This may be applied/expressed in other parts of this document as well.

[86] The encoder/decoder predictor can output prediction samples by performing inter-prediction in units of blocks. An inter-prediction may be a prediction derived in a manner that depends on data elements (eg, sample values or motion information) of picture(s) other than the current picture. When inter-prediction is applied to the current block, the predicted block (prediction sample arrays) for the current block can be output based on the reference block (reference sample array) determined by the motion vector on the reference picture indicated by the reference picture index. In this case, in order to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information of the current block may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter-prediction type information (L0 prediction, L1 prediction, Bi-prediction, etc.). When inter-prediction is applied, neighboring blocks may include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporary adjacent block may be the same or different. A temporary neighbor block may be called a co-located reference block, a co-located CU (colCU), etc., and a reference picture including a temporary neighboring block may be called a co-located picture (colPic). For example, a motion information candidate list may be constructed based on neighboring blocks of the current block, and flag or index information indicating which candidate is selected (used) to output the motion vector and/or reference picture index of the current block may be signaled. Inter-prediction can be performed based on various prediction modes. For example, in the skip mode and the merge mode, the motion information of the current block may be the same as the motion information of the selected adjacent block. In the skip mode, unlike the combine mode, the residual signal may not be transmitted. In the case of a motion vector prediction (MVP) mode, the motion vector of a selected neighboring block can be used as a motion vector predictor, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be output using the sum of the motion vector predictor and the motion vector difference.

[87] The motion information may include L0 motion information and/or L1 motion information according to the type of inter-prediction (L0 prediction, L1 prediction, Bi-prediction, etc.). The motion vector in the L0 direction may be referred to as the L0 motion vector or MVL0, and the motion vector in the L1 direction may be referred to as the L1 motion vector or MVL1. Prediction based on motion vector L0 may be called L0 prediction, prediction based on motion vector L1 may be called L1 prediction, and prediction based on motion vector L0 and motion vector L1 may be called bi-prediction. Here, the motion vector L0 may indicate a motion vector associated with the reference picture list L0, and the motion vector L1 may indicate a motion vector associated with the reference picture list L1. The reference picture list L0 may include pictures that are earlier in the output order than the current picture as reference pictures, and the reference picture list L1 may include pictures that are later in the output order than the current picture. Previous pictures may be called forward (reference) pictures, and subsequent pictures may be called reverse (reference) pictures. The reference picture list L0 may further include pictures that are later in the output order than the current picture as reference pictures. In this case, previous pictures may be indexed first in the reference picture list L0, and subsequent pictures may be indexed later. The reference picture list L1 may further include pictures preceding the current picture in output order as reference pictures. In this case, subsequent pictures may be indexed first in the reference picture list 1, and previous pictures may be indexed later. The order of output can correspond to the order of picture order counting (POC).

[88] FIG. 4 shows an example of a hierarchical structure for a coded image/video.

[89] With reference to FIG. 4, the encoded image/video is divided into VCL (Video Coding Layer) which deals with the image/video decoding process itself, a subsystem which transmits and stores the encoded information, and a Network Abstraction Layer (NAL) which exists between the VCL and subsystems and is responsible for network abstraction functions.

[90] The VCL can generate VCL data including compressed image data (cut data), or generate parameter sets including a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), a parameter set video (Video Parameter Set: VPS), etc. or an extension auxiliary information (SEI) message further required for the image decoding process.

[91] In NAL, a NAL unit can be generated by adding header information (NAL unit header) to the raw byte sequence payload (RBSP) generated in the VCL. In this case, RBSP refers to slice data, parameter sets, SEI messages, etc., generated in the VCL. The NAL unit header may include NAL unit type information determined in accordance with the RBSP data included in the corresponding NAL unit.

[92] As shown in the drawing, a NAL unit can be divided into a VCL NAL unit and a non-VCL NAL unit according to the RBSP generated in the VCL. A VCL NAL unit may refer to a NAL unit including information (cut data) about an image, and a non-VCL NAL unit may refer to a NAL unit including information (a parameter set or SEI message) necessary for decoding the image.

[93] The above-described VCL NAL unit and the non-VCL NAL unit can be transmitted through the network by attaching header information in accordance with the subsystem data standard. For example, a NAL unit may be converted into a data form of a predefined standard such as H.266/VVC file format, Real Time Transport Protocol (RTP), Transport Stream (TS), etc., and transmitted over various networks.

[94] As described above, in a NAL unit, a NAL unit type may be determined according to an RBSP data structure included in a corresponding NAL unit, and information about this NAL unit type may be stored and signaled in a NAL unit header.

[95] For example, a NAL unit can be roughly classified into a VCL NAL unit type and a non-VCL NAL unit type depending on whether the NAL unit includes image information (cut data). The VCL NAL unit type can be classified according to the property and type of the picture included in the VCL NAL unit, and the non-VCL NAL unit type can be classified according to the parameter set type.

[96] The following is an example of a NAL unit type defined according to a parameter set type included in a non-VCL NAL unit type.

[97] - APS unit (Adaptation Parameter Set) NAL: Type for NAL unit including APS

[98] - DPS unit (Decode Parameter Set) NAL: Type for NAL unit including DPS

[99] - VPS unit (Video Parameter Set) NAL: Type for NAL unit including VPS

[100] - SPS unit (Sequence Parameter Set) NAL: Type for NAL unit including SPS

[101] - PPS unit (Picture Parameter Set) NAL: Type for NAL unit including PPS

[102] - PH unit (picture title) NAL: Type for NAL unit including PH

[103] The above-described NAL unit types have syntax information for the NAL unit type, and the syntax information may be stored and signaled in a NAL unit header. For example, the syntax information may be nal_unit_type, and NAL unit types may be determined by the value of nal_unit_type.

[104] Meanwhile, as described above, one picture may include a plurality of cutouts, and one cutout may include a cutout title and cutout data. In this case, one picture header can be added to multiple clippings (a clipping title and a clipping data set) in one picture. The picture title (picture title syntax) may include information/parameters generally applicable to the picture. In this document, the clipping may be mixed or replaced with a group of tiles. Also, in this document, the title of the cut may be mixed or replaced with the title of the group of tiles.

[105] The cut header (cut header syntax or cut header information) may include information/parameters generally applicable to the cut. APS (APS syntax) or PPS (PPS syntax) may include information/parameters generally applicable to one or more clippings or pictures. SPS (SPS syntax) may include information/parameters generally applicable to one or more sequences. VPS (VPS syntax) may include information/parameters generally applicable to multiple layers. DPS (DPS syntax) may include information/parameters generally applicable to the entire video. The DPS may include information/parameters related to Coded Video Sequence (CVS) concatenation. In this document, the high level syntax (HLS) may include at least one of APS syntax, PPS syntax, SPS syntax, VPS syntax, DPS syntax, picture header syntax, and cutout header syntax.

[106] Herein, image/video information encoded in the encoding device and signaled in the form of a bit stream to the decoding device may include, like picture partitioning related information in the picture, intra/inter-prediction information, residual information, in-loop filtering information, etc., information included in the cutout header, information included in the picture header, information included in the APS, information included in the PPS, information included in the SPS, information included in the VPS, and /or information included in the DPS. In addition, the image/video information may further include NAL unit header information.

[107] Meanwhile, in order to compensate for the difference between the original image and the reconstructed image due to an error occurring in a compression encoding process such as quantization, an in-loop filtering process may be performed on the reconstructed samples or reconstructed pictures as described above. As described above, in-loop filtering may be performed by an encoder filter and a decoder filter, and a deblocking filter, SAO, and/or an adaptive loop filter (ALF) may be applied. For example, the ALF process may be executed after the deblocking filtering process and the SAO process are executed. However, even in this case, the deblocking filtering process and/or the SAO process may be omitted.

[108] Next, image restoration and filtering will be described in detail. When encoding an image/video, a reconstructed block may be generated based on the intra-prediction/inter-prediction in a block unit, and a reconstructed picture including the reconstructed blocks may be generated. If the current picture/slice is an I-picture/slice, the blocks included in the current picture/slice can be reconstructed based on intra-prediction only. Meanwhile, if the current picture/slice is a P- or B-picture/slice, the blocks included in the current picture/slice can be reconstructed based on the intra-prediction or inter-prediction. In this case, intra-prediction can be applied to some blocks in the current picture/clipping, and inter-prediction can be applied to the remaining blocks.

[109] Intra prediction may represent a prediction for generating prediction samples for a current block based on reference samples in a picture to which the current block belongs (hereinafter referred to as the current picture). In the case where intra-prediction is applied to the current block, adjacent reference samples to be used for intra-prediction of the current block may be output. The adjacent reference samples of the current block may include a sample adjacent to the left boundary of the current block having a size of nW×nH, a total of 2×nH samples, a bottom-left neighbor, a sample adjacent to the top boundary of the current block, a total of 2×nW samples, neighboring ones on the top-right, and one sample neighboring on the top-left of the current block. Alternatively, the neighbor reference samples of the current block may include a top neighbor sample of a plurality of columns and a left neighbor sample of a plurality of rows. Alternatively, the adjacent reference samples of the current block may include a total of nH samples adjacent to the right boundary of the current block having size nW×nH, a total of nH samples adjacent to the right boundary of the current block, a total of nW samples adjacent to the bottom boundary of the current block, and one sample adjacent to the bottom-right of the current block.

[110] However, some of the adjacent reference samples of the current block may not yet have been decoded or may not be available. In this case, the decoder can configure adjacent reference samples used for prediction by replacing samples that are unavailable with available samples. Alternatively, neighboring reference samples to be used for prediction can be configured through interpolation of the available samples.

[111] In the case where neighboring reference samples are derived, (i) the prediction sample may be derived based on the average or interpolation of the neighboring reference samples of the current block, and (ii) the prediction sample may be derived based on the reference sample present in a particular (prediction) direction. , for a prediction sample among neighboring reference samples of the current block. Case (i) may be called non-directional mode or non-angular mode, and case (ii) may be called directional mode or angular mode. In addition, the prediction sample may be generated by interpolating the first neighboring sample with a second neighboring sample located in a direction opposite to the prediction direction of the intra-prediction mode of the current block based on the prediction sample of the current block among the neighboring reference samples. The above case can be called linear interpolation intra-prediction (LIP). In addition, chrominance prediction samples can be generated from luminance samples using a linear model. This case may be called LM mode. In addition, the prediction time sample of the current block may be output based on the filtered neighboring reference samples, and the prediction sample of the current block may be output by calculating the weighted sum of the prediction time sample and at least one reference sample output in accordance with the intra-prediction mode among existing neighboring reference samples, that is, unfiltered neighboring reference samples. The above case can be called location-dependent intra-prediction (PDPC). In addition, the prediction sample may be output using a reference sample located in the prediction direction on a reference sample line having the highest prediction accuracy among adjacent lines of a plurality of reference samples of the current block by selecting a corresponding line, and in this case, intra-prediction coding may be performed in a method for indicating (signaling) a reference sample line to be used to a decoding device. The above case may be called multi-reference line (MRL) intra-prediction or MRL-based intra-prediction. In addition, intra-prediction can be performed based on the same intra-prediction mode by dividing the current block into vertical or horizontal subdivisions, and adjacent reference samples can be output and used in the subdivision unit. That is, in this case, since the intra-prediction mode for the current block is equally applied to subdivisions, and adjacent reference samples are output and used in the subdivision unit, intra-prediction performance can be improved in some cases. This prediction method may be called intra-subdivision prediction (ISP) or ISP-based intra-prediction. The intra-prediction method described above may be called intra-prediction type in contrast to the intra-prediction mode in sections 1.2. The intra-prediction type may be referred to by various terms, such as intra-prediction method or additional intra-prediction mode. For example, the intra-prediction type (or additional intra-prediction mode) may include at least one of the above-described LIP, PDPC, MRL, and ISP. A general intra-prediction method excluding a special intra-prediction type such as LIP, PDPC, MRL or ISP may be called a normal intra-prediction type. The normal intra-prediction type can generally be applied when the special intra-prediction type is not applied, and prediction can be performed based on the intra-prediction mode described above. Meanwhile, post-filtering can be performed on the output prediction sample as needed.

[112] Specifically, the intra-prediction process may include the steps of determining an intra-prediction mode/type, deriving a neighboring reference sample, and deriving a prediction sample based on the intra-prediction mode/type. Additionally, a post-filtering step may be performed on the output prediction sample as needed.

[113] The reconstructed picture modified by the in-loop filtering process may be generated, and the modified reconstructed picture may be output as a decoded picture from the decoding apparatus. Further, the modified reconstructed picture can be stored in a decoded picture buffer or a memory of the encoder/decoder device, and then can be used as a reference picture in the inter-prediction process when the picture is encoded/decoded. The in-loop filtering process may include a deblocking filtering process, an adaptive sample offset (SAO) process, and/or an adaptive loop filter (ALF) process as described above. In this case, one or some of the deblocking filtering, the adaptive sample offset (SAO) process, the adaptive loop filter (ALF) process, and the bidirectional filter process may be applied sequentially, or all of these processes may be applied sequentially. For example, the SAO process may be executed after the deblocking filtering process is applied to the reconstructed picture. In addition, for example, the ALF process may be performed after the deblocking filtering process is applied to the reconstructed picture. This can be done in the same way by the encoding device.

[114] A deblocking filter is a filtering method that removes distortion that occurs at the boundaries between blocks in a reconstructed image. For example, the deblocking filtering process may derive a target boundary from the reconstructed picture, determine a boundary strength (bS) for the target boundary, and perform deblocking filtering for the target boundary based on the bS. bS may be determined based on the prediction mode of two blocks adjacent to the target boundary, the difference of motion vectors, whether the reference pictures are identical, and whether a non-zero significant coefficient is present.

[115] SAO is a method for compensating the offset difference between the reconstructed picture and the original picture in a sampling unit, and can be applied based on the type of stripe offset, edge offset, and the like, for example. According to SAO, samples are classified into different categories according to SAO types, and bias values can be added to the corresponding samples based on the categories. The filtering information for SAO may include information about whether SAO is applied, SAO type information, and SAO offset value information. SAO can be applied to the reconstructed image after applying deblocking filtering.

[116] Adaptive Loop Filter (ALF) is a filtering method for a reconstructed picture in a sampling unit based on filter coefficients according to the shape of the filter. The encoding device can determine whether ALF, the ALF shape and/or the ALF filter coefficient or the like is applied by comparing the reconstructed picture and the original picture with each other and can signal them to the decoding device. That is, the filtering information for ALF may include information about whether ALF is applied, ALF filter shape information, and ALF filter coefficient information. ALF can be applied to the reconstructed image after applying deblocking filtering.

[117] FIG. 5 illustrates a picture in accordance with an embodiment of the present document. An example picture in Fig. 5 can be divided into sub-pictures, cut-outs and mosaic elements.

[118] With reference to FIG. 5, the picture can be divided into sub-pictures. For example, a sub-picture may include one or more cutouts. A clipping can represent a rectangular area of a picture. In addition, the picture can be divided into mosaic elements. For example, a rectangular cutout may include only a portion (a subset) of one tile. That is, in FIG. 5, two rectangular cutouts are in the same tile, and the two rectangular cutouts may belong to different subpictures. Problems caused by the case according to FIG. 5, and their solutions will be described below.

[119] In an example, a picture/sub-picture may be encoded on a sub-picture(s)/cut(s)/tile(s) basis. The encoding apparatus may encode the current picture based on a subpicture/slice/tile structure, or the encoding apparatus may encode one or more subpictures (including slices/tiles) of the current picture, and may output a bitstream including the (encoded) information about the sub-image. The decoding apparatus may decode one or more sub-pictures in the current picture based on a bit stream including (encoded) information for the sub-picture(s)/cut(s)/tile(s).

[120] FIG. 6 illustrates a subpicture/slice/tile based encoding method according to an embodiment of the present document.

[121] The encoder may divide the (input) picture into a plurality of (or one or more) sub-pictures/cuts/tiles. Each sub-picture can be individually/independently encoded and a bitstream can be output. Here, the bitstream for a sub-picture may be called a substream, a subset, or a subbitstream. The sub-picture/cut/tile information may include the information/syntax element(s) described herein. For example, the cutout information may include information related to the number of cutouts signaled for each picture/subpicture and the width/height of the cutouts in the tiles. For example, the tile information may include information related to the number of tiles (for example, the number of columns of the tile and/or the number of rows of the tile) and information related to the size of each tile (for example, the width and/or height ).

[122] The encoder may encode one or more sub-pictures as sub-picture information. The encoder may encode one or more cuts/tiles as cut/tile information.

[123] FIG. 7 illustrates a subpicture/slice/tile based decoding method according to an embodiment of the present document.

[124] The decoder may decode one or more sub-pictures (including the cuts/tiles) and may output one or more decoded sub-pictures or a current picture including the sub-pictures. The bitstream may include substream(s) or bitstream(s) for subpicture(s). As described above, the subpicture/cut/tile information may be configured in a high level syntax (HLS) included in the bitstream. The decoder may output one or more sub-pictures based on the sub-picture information. The decoder may output one or more slices/tiles based on the slice/tile information. The decoder may decode all or some of the sub-pictures. The decoder may decode the sub-picture (including the current block (or CU)), CTU, slice and/or tile based on CABAC, prediction, residual processing (conversion and quantization) and in-loop filtering. Accordingly, the decoded sub-picture(s) may be output. The decoded sub-picture(s) may include reconstructed/decoded block(s). The decoded sub-pictures in the output sub-picture set (OPS) may be output together. As an example, if a picture belongs to a 360-degree or omnidirectional picture/video, some of them may be rendered, and, in this case, only some of all sub-pictures may be decoded, and some or all of the decoded sub-pictures may be rendered according to the window viewing or the user's viewing position. In addition, if information indicating whether in-loop sub-picture boundary filtering is enabled is included, the decoder may apply an in-loop filtering process (eg, deblocking filtering) to a sub-picture boundary located between two sub-pictures. For example, if the sub-picture boundary is the same as the picture boundary, an in-loop filtering process for the sub-picture boundaries may or may not be performed.

[125] In embodiments of this document, the image/video information may include HLS, and the HLS may include sub-picture(s)/cut(s)/tile(s) information. The sub-picture(s) information may include information representing one or more sub-pictures in the current picture. The cut(s) information may include information representing one or more cut(s) in the current picture, sub-picture, or tile. The tile(s) information may include information representing one or more tiles in the current picture, sub-picture, or clipping. The picture may include a tile including one or more cutouts, and/or a cutout including one or more tiles. In addition, the picture may include a sub-picture including one or more cutouts/tiles.

[126] The following tables present the syntax related to the above division (subpicture/cutout/tile). The subpicture(s)/cut(s)/tile(s) information may include syntax elements in the following tables.

[127] The following table presents the sequence parameter set (SPS) syntax based on picture division (subpicture/cut/tile).

[128] [Table 1]

[129] The following table presents picture parameter set (PPS) syntax based on picture division (subpicture/cut/tile).

[130] [Table 2]

[131] The following table represents the syntax of a cutout title based on picture division (subpicture/clipping/tile).

[132] [Table 3]

[133] FIG. 8 is a flowchart showing a filter-based encoding method in an encoding apparatus. The method according to Fig. 8 may include steps S800 to S830.

[134] In step S800, the encoding device may generate a reconstructed picture. Step S600 may be performed based on the above-mentioned process of generating a reconstructed picture (or reconstructed samples).

[135] In step S810, the encoder may determine whether in-loop filtering (on a virtual boundary) is applied based on information related to in-loop filtering. Here, the in-loop filtering may include at least one of the above-mentioned deblocking, SAO, and ALF filtering.

[136] In step S820, the encoding device may generate a modified reconstructed picture (modified reconstructed samples) based on the determination in step S810. Here, the modified reconstructed picture (modified reconstructed samples) may be a filtered reconstructed picture (filtered reconstructed samples).

[137] In step S830, the encoding device may encode image/video information including information related to in-loop filtering based on the in-loop filtering process.

[138] FIG. 9 is a flowchart showing a filtering-based decoding method in a decoding apparatus. The method according to Fig. 9 may include steps S900 to S930.

[139] In step S900, the decoding apparatus may obtain image/video information, including information related to in-loop filtering, from the bitstream. Here, the bitstream may be based on encoded image/video information transmitted from the encoder.

[140] In step S910, the decoding device may generate a reconstructed picture. Step S910 may be performed based on the above-described process for generating a reconstructed picture (or reconstructed samples).

[141] In step S920, the decoding apparatus may determine whether in-loop filtering (on a virtual boundary) is applied based on information related to in-loop filtering. Here, the in-loop filtering may include at least one of the above-mentioned deblocking, SAO, and ALF filtering.

[142] In step S930, the decoding apparatus may generate a modified reconstructed picture (modified reconstructed samples) based on the determination in step S920. Here, the modified reconstructed picture (modified reconstructed samples) may be a filtered reconstructed picture (filtered reconstructed samples).

[143] As described above, an in-loop filtering process may be applied to the reconstructed picture. In this case, a virtual boundary can be defined to further improve the subjective/objective visual quality of the reconstructed image, and an in-loop filtering process can be applied along the virtual boundary. For example, the virtual edge may include a discontinuous edge such as a 360-degree image, a VR image, a border, a picture-in-picture (PIP), and the like. For example, a virtual boundary may be present at a predetermined location, and its presence and/or location may be signaled. For example, the virtual boundary may be at the fourth from the top line of samples of the CTU line (specifically, for example, above the top fourth sample of the CTU line). As another example, information about the presence/absence and/or location of a virtual boundary may be signaled via HLS. The HLS may include SPS, PPS, picture header, and clip header, as described above.

[144] Next, high-level syntax signaling and semantics in accordance with embodiments of the present document will be described.

[145] An embodiment of the present document may include a method for controlling loop filters. The present method for controlling contour filters can be applied to a reconstructed picture. In-loop filters (loop filters) can be used to decode encoded bit streams. Loop filters may include deblocking, SAO, and ALF described above. The SPS may include flags related to deblocking, SAO, and ALF, respectively. The flags may represent whether the corresponding tools are enabled for encoding a coded layer video sequence (CLVS) or a coded video sequence (CVS) referencing the SPS.

[146] When loop filters are enabled for CVS, the application of loop filters can be controlled to avoid crossing specific boundaries. For example, you can control whether edge filters cross sub-image boundaries. You can also control whether edge filters cross tile boundaries. Additionally, you can control whether edge filters cross virtual boundaries. Here, virtual boundaries can be determined on the CTU based on the availability of the line buffer.

[147] Regarding whether the in-loop filtering process is performed on a virtual boundary, the in-loop filtering-related information may include at least one of an SPS virtual edge enable flag (SPS virtual edge enable flag), an SPS virtual edge presence flag , a flag for the presence of virtual boundaries of the picture header, a flag for the presence of virtual boundaries of the SPS picture header, and information about the location of the virtual boundary.

[148] In embodiments included herein, the virtual boundary location information may include x-coordinate information of a vertical virtual boundary and/or y-coordinate information of a horizontal virtual boundary. Specifically, the virtual boundary location information may include x-coordinate information of the vertical virtual boundary and/or y-coordinate information of the horizontal virtual boundary in a unit of luminance samples. In addition, the virtual boundary location information may include information about the number of pieces of information (syntactic elements) about the x-coordinate of the vertical virtual boundary that are present in the SPS. In addition, the virtual boundary information may include information about the number of pieces of information (syntactic elements) about the y-coordinate of the horizontal virtual boundary that are present in the SPS. In addition, the virtual boundary location information may include information about the number of pieces of information (syntactic elements) about the x-coordinate of the vertical virtual boundary that is present in the picture title. In addition, the virtual boundary location information may include information about the number of pieces of information (syntactic elements) at the y-coordinate of the horizontal virtual boundary that is present in the picture title.

[149] The following tables show exemplary syntax and semantics of a sequence parameter set (SPS) in accordance with the present embodiment.

[150] [Table 4]

[151] [Table 5]

subpics_present_flag equal to 1 specifies that the subpicture parameters are present in the SPS RBSP syntax. subpics_present_flag equal to 0 specifies that subpicture parameters are not present in the SPS RBSP syntax.
sps_num_subpics_minus1 plus 1 specifies the number of subpics. sps_num_subpics_minus1 must be in the range 0 to 254. If not present, sps_num_subpics_minus1 is considered to be 0.
subpic_ctu_top_left_x[i] specifies the horizontal position of the top left CTU of the i-th subpicture in CtbSizeY unit. The length of the syntax element is Ceil(Log2(pic_width_max_in_luma_samples/ CtbSizeY)) bits. If not present, the value of subpic_ctu_top_left_x[i] is considered to be 0.
subpic_ctu_top_left_y[i] specifies the vertical position of the top left CTU of the i-th subpicture in CtbSizeY unit. The length of the syntax element is Ceil(Log2(pic_height_max_in_luma_samples/ CtbSizeY)) bits. If not present, the value of subpic_ctu_top_left_y[i] is considered to be 0.
subpic_width_minus1[i] plus 1 specifies the width of the i-th subpicture in CtbSizeY units. The length of the syntax element is Ceil(Log2(pic_width_max_in_luma_samples/CtbSizeY)) bits. If not present, the value of subpic_width_minus1[i] is assumed to be Ceil(pic_width_max_in_luma_samples/ CtbSizeY)-1.
subpic_height_minus1[i] plus 1 specifies the height of the i-th subpicture in CtbSizeY units. The length of the syntax element is Ceil(Log2(pic_height_max_in_luma_samples/CtbSizeY)) bits. If not present, the value of subpic_height_minus1[i] is assumed to be Ceil(pic_height_max_in_luma_samples/ CtbSizeY)-1.
subpic_treated_as_pic_flag[i] equal to 1 specifies that the i-th subpicture of each encoded picture in CL VS is treated as a picture during the decoding process, excluding in-loop filtering operations. subpic_treated_as_pic_flag[i] equal to 0 specifies that the i-th subpicture of each encoded picture in CL VS is not treated as a picture during the decoding process, excluding in-loop filtering operations. If not present, the value of subpic_treated_as_pic_flag[i] is considered to be 0.
loop_filter_across_subpic_enabled_flag[i] equal to 1 specifies that in-loop filtering operations can be performed across the boundaries of the i-th subpicture in each encoded picture in CL VS.
loop_filter_across_subpic_enabled_flag[i] equal to 0 specifies that in-loop filtering operations are not performed across the boundaries of the i-th subpicture in each encoded picture in CL VS. If not present, loop_filter_across_subpic_enabled_pic_flag[i] is considered to be 0.
sps_loop_filter_across_virtual_boundaries_disabled_present_flag equal to 1 specifies that in-loop filtering operations are disabled across virtual boundaries in pictures referencing the SPS.
sps_loop_filter_across_virtual_boundaries_disabled_present_flag equal to 0 specifies that such disabling of in-loop filtering operations is not applied to pictures referencing SPS. In-loop filtering operations include deblocking filter, adaptive sample shift filter, and adaptive loop filter operations.
sps_sao_enabled_flag equal to 1 specifies that the adaptive sample shifting process is applied to the reconstructed image after the deblocking filter process. sps_sao_enabled_flag equal to 0 specifies that the adaptive sample shifting process is not applied to the reconstructed image after the deblocking filter process.
sps_alf_enabled_flag equal to 0 specifies that the adaptive loop filter is disabled. sps_alf_enabled_flag equal to 1 specifies that the adaptive loop filter is enabled.
sps_num_ver_virtual_boundaries specifies the number of sps_virtual_boundaries_pos_x[i] syntax elements that are present in the SPS. If sps_num_ver_virtual_boundaries is not present, it is considered to be 0.
sps_virtual_boundaries_pos_x[i] is used to compute the value of VirtualBoundariesPosX[i], which specifies the location of the i-th vertical virtual boundary in units of luminance samples. The value of sps_virtual_boundaries_pos_x[i] must be in the range 0 to Ceil(pic_width_in_luma_samples ÷ 8)-1, inclusive.
sps_num_hor_virtual_boundaries specifies the number of sps_virtual_boundaries_pos_y[i] syntax elements that are present in the SPS. If sps_num_hor_virtual_boundaries is not present, it is considered to be 0.
sps_virtual_boundaries_pos_y[i] is used to calculate the value of VirtualBoundariesPosY[i], which specifies the location of the i-th horizontal virtual boundary in units of luminance samples. The value of sps_virtual_boundaries_pos_y[i] must be in the range from 0 to Ceil(pic_height_in_luma_samples ÷ 8)-1, inclusive.

[152] The following tables present exemplary picture parameter set (PPS) syntax and semantics in accordance with the present embodiment.

[153] [Table 6]

[154] [Table 7]

no_pic_partition_flag equal to 1 specifies that picture partitioning is not applied to every picture referencing the PPS. no_pic_partition_flag equal to 0 specifies that each picture referencing a PPS can be split into more than one tile or slice.
loop_filter_across_tiles_enabled_flag equal to 1 specifies that in-loop filtering operations can be performed along the boundaries of tiles in pictures referencing PPS. loop_filter_across_tiles_enabled_flag equal to 0 specifies that in-loop filtering operations are not performed along the boundaries of tiles in pictures referencing PPS. In-loop filtering operations include deblocking filter, adaptive sample shift filter, and adaptive loop filter operations.
loop_filter_across_slices _enabled_flag equal to 1 specifies that in-loop filtering operations can be performed along the boundaries of slices in pictures referencing PPS. loop_filter_across_slices_enabled_flag equal to 0 specifies that in-loop filtering operations are not performed along slice boundaries in pictures referencing PPS. In-loop filtering operations include deblocking filter, adaptive sample shift filter, and adaptive loop filter operations.
deblocking_filter_control_present_flag equal to 1 specifies the presence of syntactic deblocking filter controls in the PPS. deblocking_filter_control_present_flag equal to 0 specifies the absence of syntactic controls for the deblocking filter in PPS.
deblocking_filter_override_enabled_flag equal to 1 specifies the presence of pic_deblocking_filter_override_flag in PHs referencing PPS, or slice_deblocking_filter_override_flag in slice headers referencing PPS.
deblocking_filter_override_enabled_flag equal to 0 specifies the absence of pic_deblocking_filter_override_flag in PHs referencing PPS, or slice_deblocking_filter_override_flag in slice headers referencing PPS. If not present, the value of deblocking_filter_override_enabled_flag is considered to be 0.
pps_deblocking_filter_disabled_flag equal to 1 specifies that the deblocking filter operation is not applied to slices referencing a PPS in which the slice_deblocking_filter_disabled_flag is not present. pps_deblocking_filter_disabled_flag equal to 0 specifies that the deblocking filter operation is applied to slices referencing a PPS in which the slice_deblocking_filter_disabled_flag is not present. If not present, pps_deblocking_filter_disabled_flag is considered to be 0.
pps_beta_offset_div2 and pps_tc_offset_div2 specify the default deblocking offsets for β and tC (divided by 2) that apply to cutouts referencing PPS unless the default deblocking offsets are overwritten by the deblocking offsets present in the cutout headers for the cutouts , referring to PPS. The values of pps_beta_offset_div2 and pps_tc_offset_div2 must both be in the range -6 to 6, inclusive. If not present, the values of pps_beta_offset_div2 and pps_tc_offset_div2 are considered to be 0.

[155] The following tables show exemplary syntax and semantics of a picture header in accordance with the present embodiment.

[156] [Table 8]

[157]

[158] [Table 9]

ph_loop_filter_across_virtual_boundaries_disabled_present_flag equal to 1 specifies that in-loop filtering operations are disabled along virtual boundaries in pictures associated with the PH.
ph_loop_filter_across_virtual_boundaries_disabled_present_flag equal to 0 specifies that such disabling of in-loop filtering operations is not applied to pictures associated with the PH. In-loop filtering operations include deblocking filter, adaptive sample shift filter, and adaptive loop filter operations.
ph_num_ver_virtual_boundaries specifies the number of syntactic elements ph_virtual_boundaries_pos_x[i] that are present in the RN.
ph_virtual_boundaries_pos_x[i] is used to calculate the value of VirtualBoundariesPosX[i], which specifies the location of the i-th vertical virtual boundary in units of luminance samples. The value of ph_virtual_boundaries_pos_x[i] must be in the range from 1 to Ceil(pic_width_in_luma_samples ÷ 8)-1, inclusive.
ph_num_hor_virtual_boundaries specifies the number of ph_virtual_boundaries_pos_y[i] syntactic elements that are present in the PH.
ph_virtual_boundaries_pos_y[i] is used to calculate the value of VirtualBoundariesPosY[i], which specifies the location of the i-th horizontal virtual boundary in units of luminance samples. The value of ph_virtual_boundaries_pos_y[i] must be in the range from 1 to Ceil(pic_height_in_luma_samples ÷ 8)-1, inclusive.
pic_sao_enabled_present_flag equal to 1 specifies that pic_sao_luma_flag and pic_sao_chroma_flag are present in the PH. pic_sao_enabled_present_flag equal to 0 specifies that pic_sao_luma_flag and pic_sao_chroma_flag are not present in the PH. If pic_sao_enabled_present_flag is not present, it is considered to be 0.
pic_sao_luma_enabled_flag equal to 1 specifies that SAO is enabled for the luma component in all slices associated with the PH; pic_sao_luma_enabled_flag equal to 0 specifies that the SAO for the luma component may be disabled for one or more or all of the slices associated with the PH.
pic_sao_chroma_enabled_flag equal to 1 specifies that SAO is enabled for the chroma component in all slices associated with the PH; pic_sao_chroma_enabled_flag equal to 0 specifies that SAO for the chroma component may be disabled for one or more or all of the slices associated with the PH.
pic_alf_enabled_present_flag equal to 1 specifies that pic_alf_enabled_flag, pic_num_alf_aps_ids_luma, pic_alf_aps_id_luma[i], pic_alf_chroma_idc and pic_alf_aps_id_chroma are present in the PH.
pic_alf_enabled_present_flag equal to 0 specifies that pic_alf_enabled_flag, pic_num_alf_aps_ids_luma, pic_alf_aps_id_luma[i], pic_alf_chroma_idc and pic_alf_aps_id_chroma are not present in the PH.
If pic_alf_enabled_present_flag is not present, it is considered to be 0.
pic_alf_enabled_flag equal to 1 specifies that the adaptive edge filter is enabled for all slices associated with the PH, and can be applied to the Y, Cb, or Cr color component of the slices.
pic_alf_enabled_flag equal to 0 specifies that the adaptive loop filter may be disabled for one or more or all of the cuts associated with the PH. If not present, pic_alf_enabled_flag
is considered equal to 0.
pic_num_alf_aps_ids_luma specifies the number of ALF APS referenced by the cuts associated with the PH.
pic_alf_aps_id_luma[i] specifies the adaptation_parameter_set_id of the i-th ALF APS referenced by the luma component of the slices associated with the PH.
The alf_luma_filter_signal_flag value of an APL NAL unit having an aps_params_type equal to ALF_APS and an adaptation_parameter_set_id equal to pic_alf_aps_id_luma[i] shall be equal to 1.
pic_alf_chroma_idc equal to 0 specifies that the adaptive contour filter is not applied to the Cb and Cr chroma components.
pic_alf_chroma_idc equal to 1 indicates that the adaptive edge filter is applied to the Cb chroma component.
pic_alf_chroma_idc equal to 2 indicates that an adaptive edge filter is applied to the Cr color component.
pic_alf_chroma_idc equal to 3 indicates that the adaptive contour filter is applied to the Cb and Cr color components. If pic_alf_chroma_idc is not present, then it is considered to be 0.
pic_alf_aps_id_chroma specifies the adaptation_parameter_set_id ALF APS referenced by the chroma component of the slices associated with the PH.
pic_deblocking_filter_override_present_flag equal to 1 specifies that pic_deblocking_filter_override_flag is present in the PH. pic_deblocking_filter_override_present_flag equal to 0 specifies that pic_deblocking_filter_override_flag is not present in the PH. If pic_deblocking_filter_override_present_flag is not present, it is considered to be 0.
pic_deblocking_filter_override_flag equal to 1 specifies that deblocking parameters are present in the PH. pic_deblocking_filter_override_flag equal to 0 specifies that deblocking parameters are not present in the PH. If not present, pic_deblocking_filter_override_flag is considered to be 0.

[159]

pic_deblocking_filter_disabled_flag equal to 1 specifies that the deblocking filter operation is not applied to cuts associated with the PH. pic_deblocking_filter_disabled_flag equal to 0 specifies that the deblocking filter operation is applied to cuts associated with the PH. If pic_deblocking_filter_disabled_flag is not present, it is considered equal to pps_deblocking_filter_disabled_flag.
pic_beta_offset_div2 and pic_tc_offset_div2 specify the deblocking parameter offsets for β and tC (divided by 2) for cuts associated with the PH. The values of pic_beta_offset_div2 and pic_tc_offset_div2 must both be in the range -6 to 6, inclusive. If not present, the values of pic_beta_offset_div2 and pic_tc_offset_div2 are considered to be equal to pps_beta_offset_div2 and pps_tc_offset_div2, respectively.

[160] The following tables show exemplary syntax and semantics of a fetch header in accordance with the present embodiment.

[161] [Table 10]

[162] [Table 11]

cu_chroma_qp_offset_enabled_flag equal to 1 specifies that cu_chroma_qp_offset_flag may be present in the transform unit and palette encoding syntax. cu_chroma_qp_offset_enabled_flag equal to 0 specifies that cu_chroma_qp_offset_flag is not present in the transform unit and palette encoding syntax. If not present, the value of cu_chroma_qp_offset_enabled_flag is considered to be 0.
slice_sao_luma_flag equal to 1 specifies that SAO is enabled for the luma component in the current slice; slice_sao_luma_flag equal to 0 specifies that SAO is disabled for the luma component in the current slice. If slice_sao_luma_flag is not present, it is considered equal to pic_sao_luma_enabled_flag.
slice_sao_chroma_flag equal to 1 specifies that SAO is enabled for the chroma component in the current slice; slice_sao_chroma_flag equal to 0 specifies that SAO is disabled for the chroma component in the current slice. If slice_sao_chroma_flag is not present, it is considered equal to pic_sao_chroma_enabled_flag.
slice_alf_enabled_flag equal to 1 specifies that the adaptive edge filter is enabled and can be applied to the Y, Cb and Cr color component in the slice. slice_alf_enabled_flag equal to 0 specifies that the adaptive edge filter is disabled for all color components in the slice. If not present, the value of slice_alf_enabled_flag is assumed to be equal to pic_alf_enabled_flag.
slice_num_alf_aps_ids_luma specifies the number of ALF APS that the slice refers to. If slice_alf_enabled_flag is 1 and slice_num_alf_aps_ids_luma is not present, the value of slice_num_alf_aps_ids_luma is assumed to be equal to the value of pic_num_alf_aps_ids_luma.
slice_alf_aps_id_luma[i] specifies the adaptation_parameter_set of the i-th ALF APS referenced by the slice luma component. The TemporalId of an APS NAL unit having an aps_params_type equal to ALF_APS and an adaptation_parameter_set_id equal to slice_alf_aps_id_luma[i] shall be less than or equal to the TemporalId of the NAL unit of the encoded slice. If slice_alf_enabled_flag is 1 and slice_alf_aps_id_luma[i] is not present, the value of slice_alf_aps_id_luma[i] is considered equal to the value of pic_alf_aps_id_luma[i].
The alf_luma_filter_signal_flag value of an APS NAL unit having an aps_params_type equal to ALF_APS and an adaptation_parameter_set_id equal to slice_alf_aps_id_luma[i] shall be equal to 1.
slice_alf_chroma_idc equal to 0 specifies that the adaptive edge filter is not applied to the Cb and Cr chroma components.
slice_alf_chroma_idc equal to 1 indicates that the adaptive edge filter is applied to the Cb chroma component.
slice_alf_chroma_idc equal to 2 indicates that an adaptive edge filter is applied to the Cr color component.
slice_alf_chroma_idc equal to 3 indicates that the adaptive contour filter is applied to the Cb and Cr color components. If slice_alf_chroma_idc is not present, it is assumed to be equal to pic_alf_chroma_idc.
slice_alf_aps_id_chroma specifies the adaptation_parameter_set ALF APS referenced by the slice chroma component. The TemporalId of an APS NAL unit having an aps_params_type equal to ALF_APS and an adaptation_parameter_set_id equal to slice_alf_aps_id_chroma shall be less than or equal to the TemporalId of the NAL unit of the encoded slice. If slice_alf_enabled_flag is 1 and slice_alf_aps_id_chroma is not present, then the value of slice_alf_aps_id_chroma is considered equal to the value of pic_alf_aps_id_chroma.
The alf_chroma_filter_signal_flag value of an APS NAL unit having an aps_params_type equal to ALF_APS and an adaptation_parameter_set_id equal to slice_alf_aps_id_chroma shall be equal to 1.
slice_deblocking_filter_override_flag equal to 1 specifies that deblocking options are present in the slice header.
slice_deblocking_filter_override_flag equal to 0 specifies that deblocking options are not present in the slice header. If not present, the value of slice_deblocking_filter_override_flag is assumed to be equal to pic_deblocking_filter_override_flag.
slice_deblocking_filter_disabled_flag equal to 1 specifies that the deblocking filter operation is not applied to the current slice. slice_deblocking_filter_disabled_flag equal to 0 specifies that the deblocking filter operation is applied to the current slice. If slice_deblocking_filter_disabled_flag is not present, it is considered equal to pic_deblocking_filter_disabled_flag.
slice_beta_offset_div2 and slice_tc_offset_div2 specify the deblocking parameter offsets for β and tC (divided by 2) for the current slice. The values of slice_beta_offset_div2 and slice_tc_offset_div2 must both be in the range -6 to 6, inclusive. If not present, the values of slice_beta_offset_div2 and slice_tc_offset_div2 are considered to be equal to pic_beta_offset_div2 and pic_tc_offset_div2, respectively.

[163] Next, information related to sub-pictures, information related to virtual boundaries that can be used for in-loop filtering, and signaling thereof will be described.

[164] In an example, two different rectangular cutouts may belong to different subpictures when sharing the same tile. In this case, a problem may arise in which the coding complexity increases.

[165] To facilitate picture division, an embodiment of the present document may include example conditions for a picture being divided into two or more sub-pictures. In the example, all CTUs in one tile may belong to the same sub-pictures. In another example, all CTUs in a subpicture may belong to the same tile. The above two examples can be used separately for image/video encoding, can be used sequentially, or can be used in combination. Moreover, in an embodiment of the present document, in the case where a sub-picture includes CTUs that are subsets of all CTUs in one tile, the sub-picture may not include CTUs belonging to another tile.

[166] In signaling for the current picture, if the value of subpic_present_flag is 1, the number of subpictures in each picture referencing the SPS may be 1 (the value of sps_num_subpics_minus1 is 0). This condition was adopted to support the use case of sub-picture extraction, in which the sub-picture is independently encoded, from a bitstream to form another bitstream even without changing values less than the parameter set values. Accordingly, even if the value of subpic_present_flag is 1, and the value of sps_num_subpics_minus1 is 0, subpic_ctu_top_left_x[0], subpic_ctu_top_left_y[0], subpic_width_minus1[0], subpic_height_minus1[0], subpic_treated_as_pic_flag[i] and/or loop_filter_across_subpic_enabled_flag[i] are still present. In such a situation, such syntax elements may be present, overlapping each other, and if a signal for an incorrect value is received at the corresponding syntax element, this may make the decoder operation unpredictable. For example, if the value of subpics_present_flag is 1 and the value of sps_num_subpics_minus1 is 0, this means that there is only one subpicture (the picture itself) present, and the value of subpic_treat_as_pic_flag[0] is 1. In this case, if the corresponding value is signaled as a value of 0, a the problem of inconsistency in the decoding process.

[167] To solve the above-described problem, an embodiment of the present document includes example conditions that may apply in the case where subpicture signaling is present (eg, the value of subpic_present_flag is 1) and only one subpicture is present in the picture (eg, the value of sps_num_subpics_minus1 is 0). The above example conditions may be as shown in the following table.

[168] [Table 12]

a) The signaling properties of the only sub-pictures in the picture are omitted and their values are output. In other words, the syntax elements subpic_ctu_top_left_x[0], subpic_ctu_top_left_y[0], subpic_width_minus1[0], subpic_height_minus1[0], subpic_treated_as_pic_flag[i] and loop_filter_across_subpic_enabled_flag[i] are not present in the current VCC definition, and their values are output as follows:
- subpic_ctu_top_left_x[0] is considered equal to 0
- subpic_ctu_top_left_y[0] is considered equal to 0
- subpic_width_minus1[0] is considered equal to Ceil(pic_width_max_in_luma_samples÷CtbSizeY))
- subpic_height_minus1[0] is considered equal to Ceil(pic_height_max_in_luma_samples÷CtbSizeY))
- subpic_treated_as_pic_flag[0] is considered equal to 1
- loop_filter_across_subpic_enabled_flag[0] is considered equal to 0
-
b) Sub-picture property values are limited as follows:
- The first CTU of the subpicture is the first CTU of the picture (ie, subpic_ctu_top_left_x[0] is constrained to be 0, subpic_ctu_top_left_y[0] is constrained to be 0).
- The width of the subpicture is equal to the width of the picture (ie, subpic_width_minus1[0] is limited to be equal to Ceil(pic_width_max_in_luma_samples÷CtbSizeY))).
- The height of the subpicture is equal to the height of the picture (ie subpic_height_minus1[0] is limited to be equal to Ceil(pic_height_max_in_luma_samples÷CtbSizeY))).
- The subpicture is an independently encoded subpicture (ie, subpic_treated_as_pic_flag[0] is constrained to be equal to 1).

[169] In the example, in the case where sub-picture signaling is present and the virtual boundary positions are present in the picture header, there is a problem that the picture header must be rewritten to identify whether the virtual boundary location signaling is correct in sub-picture retrieval scenarios and combining sub-pictures. This may defeat the design goal of subpicture extraction/merging, in which the bitstream does not need to be rewritten for a NAL unit of a level that is lower than the parameter set level.

[170] To solve the above problem, according to an embodiment of the present document, if sub-picture signaling is present (eg, if sub-picture signaling is present in the SPS), the virtual boundary position signaling may not be included in the picture header. As an example, if sub-picture signaling is present, information about the positions of the virtual boundaries may be included in a set of high-level parameters. For example, if sub-picture signaling is present, information about the positions of the virtual boundaries may be included in the SPS. In addition, if sub-picture signaling is present, information about the positions of the virtual boundaries may be included in the PPS.

[171] In an embodiment of this document, if subpicture ID signaling is present (if the value of sps_subpic_id_present_flag is 1, all subpictures may be independently encoded subpictures (the value of subpic_treated_as_pic_flag[i] is 1). In this case, locations (eg, SPS, PPS, or picture title) sub-picture ID alarms may not have any relationship.

[172] In accordance with embodiments of this document in conjunction with the tables above, it can be determined whether information related to virtual boundaries (eg, information related to virtual boundary positions) is signaled in the sequence parameter set based on whether sub-picture information. For example, in the case where sub-picture information is present in the corresponding sequence, information related to the virtual boundaries (eg, information related to the positions of the virtual boundaries) may be signaled in the sequence parameter set. Accordingly, the virtual boundary-based encoding method according to embodiments of the present document can be efficiently performed without rewriting or changing high-level syntax.

[173] FIG. 10 and fig. 11 schematically shows an example of a video/image encoding method and related components in accordance with embodiment(s) of the present document.

[174] The method disclosed in FIG. 10 may be performed by the encoding device disclosed in FIG. 2 or fig. 11. Specifically, for example, S1000 and S1010 in FIG. 10 may be executed by the encoding device remainder processor 230 according to FIG. 11, S1020 in FIG. 10 may be performed by the encoder filter 260 of FIG. 11, and S1030 in FIG. 10 may be performed by the entropy encoder 240 of the encoder according to FIG. 11. Additionally, although not shown in FIG. 10, prediction samples or prediction-related information may be output by the encoder predictor 220 of FIG. 10, and the bitstream may be generated from the residual information or prediction-related information by the entropy encoder 240 of the encoding device. The method disclosed in FIG. 10 may include the above-mentioned embodiments of the present document.

[175] With reference to FIG. 10, the encoder can output residual samples (S1000). The encoder may output residual samples for the current block, and residual samples for the current block may be output based on the original samples and prediction samples of the current block. Specifically, the encoding device may output prediction samples of current blocks based on the prediction mode. In this case, various prediction methods disclosed herein, such as inter-prediction or intra-prediction, can be applied. Residual samples can be derived from the prediction samples and the original samples.

[176] The encoder can output conversion coefficients. The encoder may output transform coefficients based on the transform process for the residual samples. For example, the conversion process may include at least one of DCT, DST, GBT or CNT.

[177] The encoder may output quantized transform coefficients. The encoding device may output quantized transform coefficients based on a quantization process for the transform coefficients. The quantized transform coefficients can be in the form of a one-dimensional vector, based on the scan order of the coefficients.

[178] The encoding device may generate residual information (S1010). The encoder may generate residual information based on the residual samples for the current block. The encoder may generate residual information representing the quantized transform coefficients. Residual information can be generated by various coding methods such as exponential Golomb coding, CAVLC and CABAC.

[179] The encoder may generate reconstructed samples. The encoder may generate reconstructed samples based on the residual information. Reconstructed samples can be generated by summing the residual samples based on the residual information with the prediction sample. Specifically, the encoder may perform prediction (intra- or inter-prediction) for the current block and may generate reconstructed samples based on the original samples and the prediction samples generated by the prediction.

[180] The reconstructed samples may include reconstructed luma samples and reconstructed chrominance samples. Specifically, the residual samples may include luma residual samples and chroma residual samples. Residual luminance samples can be generated based on the original luminance samples and the prediction luminance samples. Residual chroma samples may be generated based on the original chroma samples and the prediction chroma samples. The encoder may output transform coefficients for the residual luminance samples (luminance transform coefficients) and/or transform coefficients for the residual chroma samples (chrominance transform coefficients). The quantized transform coefficients may include quantized luma transform coefficients and/or quantized chrominance transform coefficients.

[181] The encoding device may generate information related to in-loop filtering for the reconstructed samples of the current picture (S1020). The encoder may perform an in-loop filtering process on the reconstructed samples, and may generate information related to the in-loop filtering based on the in-loop filtering process. For example, information related to in-loop filtering may include information about virtual boundaries as described above herein (SPS virtual boundaries enable flag, picture header virtual boundaries enable flag, SPS virtual boundaries present flag, picture header virtual boundaries present flag and information about the positions of virtual boundaries).

[182] The encoding device can encode video/image information (S1030). The image information may include residual information, prediction-related information, and/or in-loop filtering-related information. The encoded video/image information can be output in the form of a bit stream. The bit stream may be transmitted to the decoding device via a network or storage medium.

[183] The video/image information may include various pieces of information in accordance with an embodiment of the present document. For example, the video/image information may include information disclosed in at least one of Tables 1-12 described above.

[184] In an embodiment, the image information may include a sequence parameter set (SPS). Based on whether the SPS includes information related to the sub-picture, it can be determined whether the SPS includes additional information related to virtual boundaries.

[185] In an embodiment, additional information related to the virtual boundaries may include the number of virtual boundaries and the positions of the virtual boundaries.

[186] In an embodiment, additional information related to the virtual boundaries may include information about the number of vertical virtual boundaries, information about the positions of the vertical virtual boundaries, information about the number of horizontal virtual boundaries, and information about the positions of the horizontal virtual boundaries.

[187] In an embodiment, the picture information may include a subpicture presence flag (eg, subpic_present_flag). It can be determined whether the SPS includes information related to the sub-picture based on the sub-picture presence flag.

[188] In an embodiment, the picture information may include a sub-picture ID presence flag. Subpictures in the current picture may be independently encoded subpictures based on the value of the subpicture ID presence flag being 1.

[189] In an embodiment, the current picture may include sub-pictures and tiles. Coding tree units (CTUs) in one tile may belong to the same sub-picture.

[190] In an embodiment, the current picture may include sub-pictures and tiles. Coding tree units (CTUs) in the same sub-picture may belong to the same tile.

[191] In an embodiment, the SPS may include an SPS virtual boundary presence flag related to whether the SPS includes additional information related to the virtual boundaries. The value of the SPS virtual boundary presence flag may be determined to be 1 based on the fact that the SPS includes information related to the sub-picture.

[192] In an embodiment, the picture information may include picture header information. Additional information related to virtual boundaries may not be included in the picture title based on the fact that the SPS includes information related to the sub-picture.

[193] In an embodiment, the SPS may include additional information related to virtual boundaries based on the fact that the SPS includes information related to the sub-picture.

[194] FIG. 12 and fig. 13 schematically shows an example of a video/image decoding method and related components in accordance with embodiment(s) of the present document.

[195] The method disclosed in FIG. 12 may be performed by the decoding apparatus disclosed in FIG. 12 or fig. 13. Specifically, for example, S1200 in FIG. 12 may be performed by the entropy decoder 310 of the decoding device, S1210 may be performed by the remainder processor 320 and/or adder 340 of the decoding device, and S1220 may be performed by the filter 350 of the decoding device. The method disclosed in FIG. 12 may include the above-described embodiments of this document.

[196] With reference to FIG. 12, the decoding device can receive/receive video/image information (S1200). The video/image information may include residual information, prediction-related information, and/or in-loop filtering-related information. The decoding device can receive/receive video/image information via a bitstream.

[197] The video/image information may include various pieces of information in accordance with an embodiment of the present document. For example, the image/video information may include information disclosed in at least one of Tables 1-12 described above.

[198] The decoding apparatus may output the quantized transform coefficients. The decoding apparatus may output quantized transform coefficients based on the residual information. The quantized transform coefficients can be in the form of a one-dimensional vector based on the scanning order of the coefficients. The quantized transform coefficients may include quantized luma transform coefficients and/or quantized chrominance transform coefficients.

[199] The decoding device can output transform coefficients. The decoding apparatus may output transform coefficients based on a dequantization process for the quantized transform coefficients. The decoding apparatus may output luminance conversion coefficients by dequantization based on the quantized luminance conversion coefficients. The decoding apparatus may output chrominance conversion coefficients by dequantization based on the quantized chrominance conversion coefficients.

[200] The decoding device may generate/output residual samples. The decoding apparatus may output residual samples based on an inverse transform process for the transform coefficients. The decoding apparatus may output residual luminance samples through an inverse transform process based on luminance transform coefficients. The decoding apparatus may output residual chroma samples through an inverse transform process based on chroma transform coefficients.

[201] The decoding apparatus may generate/output reconstructed samples (S1210). For example, the decoding device may generate/output reconstructed luma samples and/or reconstructed chrominance samples. The decoding apparatus may generate reconstructed luma samples and/or reconstructed chrominance samples based on the residual information. The decoding apparatus may generate reconstructed samples based on the residual information. The reconstructed samples may include reconstructed luma samples and/or reconstructed chrominance samples. The luma components of the reconstructed sample may correspond to the reconstructed luma samples, and the chroma components of the reconstructed samples may correspond to the reconstructed chroma samples. The decoding apparatus may generate prediction luma samples and/or prediction chrominance samples through a prediction process. The decoding apparatus may generate reconstructed luminance samples based on the prediction luminance samples and residual luminance samples. The decoding apparatus may generate reconstructed chroma samples based on the prediction chroma samples and the residual chroma samples.

[202] The decoding apparatus may generate modified (filtered) reconstructed samples (S1220). The decoder may generate modified reconstructed samples based on an in-loop filtering process for the reconstructed samples. The decoder may generate modified reconstructed samples based on information related to in-loop filtering. The decoding apparatus may use a deblocking process, a SAO process, and/or an ALF process to generate modified reconstructed samples.

[203] In an embodiment, the image information may include a sequence parameter set (SPS). Based on whether the SPS includes information related to the sub-picture, it can be determined whether the SPS includes additional information related to virtual boundaries.

[204] In an embodiment, additional information related to the virtual boundaries may include the number of virtual boundaries and the positions of the virtual boundaries.

[205] In an embodiment, additional information related to the virtual boundaries may include information about the number of vertical virtual boundaries, information about the positions of the vertical virtual boundaries, information about the number of horizontal virtual boundaries, and information about the positions of the horizontal virtual boundaries.

[206] In an embodiment, the picture information may include a subpicture presence flag (eg, subpic_present_flag). It can be determined whether the SPS includes information related to the sub-picture based on the sub-picture presence flag.

[207] In an embodiment, the picture information may include a sub-picture ID presence flag. The subpictures in the current picture may be independently encoded subpictures based on the value of the subpicture ID presence flag being 1.

[208] In an embodiment, the current picture may include sub-pictures and tiles. Coding tree units (CTUs) in one tile may belong to the same sub-picture.

[209] In an embodiment, the current picture may include sub-pictures and tiles. Coding tree units (CTUs) in the same sub-picture may belong to the same tile.

[210] In an embodiment, the SPS may include an SPS virtual boundary presence flag related to whether the SPS includes additional information related to the virtual boundaries. The value of the SPS virtual boundary presence flag may be determined to be 1 based on the fact that the SPS includes information related to the sub-picture.

[211] In an embodiment, the picture information may include picture header information. Based on the fact that the SPS includes information related to the sub-picture, additional information related to the virtual boundaries may not be included in the picture title.

[212] In an embodiment, based on the fact that the SPS includes information related to the sub-picture, the SPS may include additional information related to virtual boundaries.

[213] If residual samples for the current block are present, the decoder may receive residual information for the current block. The residual information may include conversion factors for residual samples. The decoding apparatus may output residual samples (or a residual array of samples) for the current block based on the residual information. Specifically, the decoding apparatus may output quantized transform coefficients based on the residual information. The quantized transform coefficients can be in the form of a one-dimensional vector based on the scanning order of the coefficients. The decoding apparatus may output transform coefficients based on a dequantization process for the quantized transform coefficients. The decoding apparatus may output residual samples based on the transform coefficients.

[214] The decoding apparatus may generate reconstructed samples based on the (intra-) prediction samples and residual samples, and may output a reconstructed block or reconstructed picture based on the reconstructed samples. Specifically, the decoding apparatus may generate reconstructed samples based on the sum of the (intra-) prediction samples and the residual samples. Then, as necessary, the decoding apparatus may apply an in-loop filtering process, such as deblocking filtering and/or SAO processes, to the reconstructed picture to improve the subjective/objective quality of the picture.

[215] For example, a decoding apparatus may obtain image information including all or parts of the above-described pieces of information (or syntax elements) by decoding the bit stream or encoded information. In addition, the bitstream or encoded information may be stored on a computer-readable storage medium and may cause the above-described decoding method to be performed.

[216] Although in the above-described embodiments, the methods have been described on the basis of a flowchart in which the steps or blocks are listed in sequence, the steps herein are not limited to a certain order, and a certain step may be performed in another step or in a different order or simultaneously in relation to what is described above. Additionally, those skilled in the art will appreciate that steps in the flowcharts are not exclusive and another step may be included, or one or more steps in the flowchart may be removed without affecting scope of this disclosure.

[217] The above-described method according to the present disclosure may be implemented in the form of software, and the encoding device and/or decoding device according to the present disclosure may be included in a device for performing image processing, such as a television, computer, smartphone, set-top box , display device, etc.

[218] When embodiments of the present disclosure are implemented in software, the above-described method may be implemented by a module (process or function) that performs the above function. The module can be stored in memory and executed by the processor. The memory may be internal or external to the processor and may be coupled to the processor by various well known means. The processor may include an application specific integrated circuit (ASIC), other chipsets, logic circuitry, and/or data processing apparatus. The memory may include read only memory (ROM), random access memory (RAM), flash memory, memory card, storage medium, and/or other storage device. In other words, embodiments of the present disclosure may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional blocks shown in the respective drawings may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, implementation information (eg, instruction information) or algorithms may be stored in a digital storage medium.

[219] In addition, the decoding device and encoding device to which embodiments of this document are applied may be included in a multimedia broadcast transceiver, mobile communication terminal, home theater device, digital cinema device, surveillance camera, video talking device, real-time communication device such as video communication, mobile streaming device, storage medium, video recording camera, video on demand (VoD) service provider, video over-the-top (OTT) delivery device, internet streaming service provider, 3D video device, device virtual reality (VR), augmented reality (AR) device, video teleconferencing device, vehicle terminal (for example, vehicle terminal (including autonomous vehicle), aircraft terminal, ship terminal, etc.) and medical video device, and can be used to process image or data signals. For example, an OTT video device may include a game console, Bluray player, Internet-enabled television, home theater system, smartphone, tablet PC, digital video recorder (DVR).

[220] Moreover, the processing method to which embodiments of the present document are applied may be executed in the form of a computer-executable program and may be stored in a computer-readable recording medium. Multimedia data having a data structure in accordance with embodiments of the present document may also be stored in a computer-readable recording medium. The computer-readable recording medium includes all types of storage devices and distributed storage devices in which computer-readable data is stored. The computer-readable recording medium may include, for example, a Bluray disc (BD), a universal serial bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical storage device. . Computer-readable recording media also include media embodied in the form of a carrier wave (eg, transmission over the Internet). In addition, the bit stream generated by the encoding method may be stored in a computer-readable recording medium or transmitted over wired/wireless communication networks.

[221] In addition, embodiments of the present document may be implemented as a computer program product based on program code, and the program code may be executed in a computer in accordance with the embodiments of the present document. The program code may be stored on a computer-readable medium.

[222] FIG. 14 shows an example of a content streaming system in which the embodiment disclosed herein may be applied.

[223] With reference to FIG. 14, a content streaming system to which embodiments of the present document are applied may typically include an encoding server, a streaming server, a web server, a media storage device, a user device, and a media input device.

[224] The encoding server functions to compress content input from media input devices such as a smartphone, camera, video recorder, etc. into digital data to generate a bit stream and transmit it to the streaming server. As another example, in the case where a multimedia input device such as a smartphone, camera, video recorder, etc. directly generates a bitstream, the encoding server may be omitted.

[225] The bitstream may be generated by an encoding method or a bitstream generation method in which embodiments of the present document are applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[226] The streaming server transmits media data to the user device based on the user's request through a web server, which functions as a tool for informing the user about the service. When a user requests a desired service, the web server delivers the request to the streaming server and the streaming server transmits the media data to the user. In this regard, the content streaming system may include a separate control server, in which case, the control server functions to manage commands/responses between associated equipment in the content streaming system.

[227] The streaming server may receive content from a media storage and/or encoding server. For example, when content is received from an encoding server, the content may be received in real time. In this case, to provide a smooth streaming service, the streaming server may store the bit stream for a predetermined period of time.

[228] For example, user equipment may include a mobile phone, smartphone, laptop, digital broadcast terminal, personal digital assistant (PDA), portable media player (PMP), navigation device, slim PC, tablet PC, ultrabook, wearable device (eg , watch type terminal (smart watch), glasses type terminal (smart glasses), head mounted display (HMD)), digital TV, desktop computer, digital pointer and the like.

[229] Each of the servers in the content streaming system may operate as a distributed server, in which case, data received by each server may be processed in a distributed manner.

[230] The claims herein may be combined in various ways. For example, the technical features in the method claims herein may be combined to be implemented or performed in an apparatus, and the technical features in the device claims may be combined to be implemented or performed in a method. In addition, technical features in the method claims and device claims may be combined to be implemented or performed in the device. Also, technical features in the method claims and device claims can be combined to be implemented or performed in the method.

Claims (32)

1. An image decoding device containing: memory; And at least one processor coupled to the memory, wherein the at least one processor is configured to: obtaining image information including residual information via a bit stream, generating reconstructed samples of the current picture based on residual information, and generating modified reconstructed samples based on an in-loop filtering process for the reconstructed samples, wherein the picture information includes a sequence parameter set (SPS) and a picture header, and wherein, when the SPS includes information related to a sub-picture, information about the number of virtual boundaries and information about the positions of the virtual boundaries are not included in the picture header, but are included in the SPS. 2. The image decoding device according to claim 1, wherein the information about the number of virtual boundaries contains information about the number of vertical virtual boundaries and information about the number of horizontal virtual boundaries, and wherein the information about the positions of the virtual boundaries contains information about the positions of the vertical virtual boundaries and information about the positions of the horizontal virtual boundaries. 3. The image decoding device according to claim 1, wherein the image information includes a sub-picture presence flag, and wherein whether the SPS includes information related to the sub-picture is determined based on the presence flag of the sub-picture. 4. The image decoding device according to claim 1, wherein the current picture contains sub-pictures and tile elements, and where the coding tree units (CTUs) in one tile belong to the same sub-picture. 5. The image decoding device according to claim 1, wherein the current picture contains sub-pictures and tile elements, and where the coding tree units (CTUs) in the same sub-picture belong to the same tile. 6. The image decoding apparatus of claim 1, wherein the SPS includes an SPS virtual boundary presence flag related to whether the SPS includes virtual boundary number information and virtual boundary position information, and wherein the value of the SPS virtual boundary presence flag is 1 when the SPS includes information related to the sub-picture. 7. An image encoding device comprising: memory; And at least one processor coupled to the memory, wherein the at least one processor is configured to: generating residual samples for the current block, generating residual information based on residual samples for the current block, generating information related to in-loop filtering for the reconstructed samples of the current picture; And encoding image information including residual information and in-loop filtering related information, wherein the picture information includes a sequence parameter set (SPS) and a picture header, and wherein, when the SPS includes information related to a sub-picture, information about the number of virtual boundaries and information about the positions of the virtual boundaries are not included in the picture header, but are included in the SPS. 8. A transmission device comprising: at least one processor configured to obtain a bitstream, wherein the bitstream is generated by generating residual samples for the current block, generating residual information based on the residual samples for the current block, generating in-loop filtering related information for the reconstructed samples of the current picture, and encoding image information including residual information and information related to in-loop filtering; And a transmitter configured to transmit data containing a bit stream, wherein the picture information includes a sequence parameter set (SPS) and a picture header, and wherein, when the SPS includes information related to a sub-picture, information about the number of virtual boundaries and information about the positions of the virtual boundaries are not included in the picture header, but are included in the SPS.