RU2812029C2 - Method and device for image encoding/decoding based on mixed type nal unit and method for transmitting bit stream - Google Patents

Abstract

FIELD: image encoding.

SUBSTANCE: network abstraction layer (NAL) unit type information of the video coding layer (VCL) of the current frame and first flag information indicating whether or not a subframe included in the current frame is treated as one frame are obtained from the bitstream. The NAL unit type of each of the plurality of slices included in the current frame is determined based on the received NAL unit VCL type information. The plurality of slices are decoded based on the determined NAL unit type and the first flag information. Here, the current frame consists of two or more subframes based on at least some of a plurality of slices having different types of NAL units. Here, the first flag information has a predetermined value indicating that each of the two or more subframes is treated as one frame based on at least some of a plurality of slices having different types of NAL units.

EFFECT: improved efficiency of image encoding.

14 cl, 17 dwg

Description

Field of technology to which the invention relates

[1] The present invention relates to an image encoding/decoding method and equipment, and more specifically, to an image encoding and decoding method and equipment based on a mixed NAL unit type, and to a recording medium for storing a bitstream generated by the image encoding method/equipment true revelation of the essence.

State of the art

[2] Recently, the demand for high-quality, high-definition images, such as high-definition (HD) images and ultra-high-definition (UHD) images, has been increasing in various fields of technology. As the resolution and quality of image data increases, the amount of transmitted information or bits increases relatively compared to existing image data. Increasing the amount of information or bits transmitted results in increased transmission costs and storage costs.

[3] Accordingly, there is a need for high-performance image compression technology for efficiently transmitting, storing and reproducing information regarding high-quality, high-resolution images.

The essence of the invention

Technical problem

[4] An object of the present invention is to provide an image encoding/decoding method and equipment with improved encoding/decoding efficiency.

[5] Another object of the present invention is to provide an image encoding/decoding method and equipment based on a mixed NAL unit type.

[6] Another object of the present invention is to provide a method and equipment for encoding/decoding images based on two or more subframes having different types of NAL units.

[7] Another object of the present invention is to provide a non-volatile decoder-readable storage medium for storing a bitstream generated by an image encoding method or equipment according to the present disclosure.

[8] Another object of the present invention is to provide a recording medium storing a bitstream received, decoded and used to reconstruct an image by image decoding equipment according to the present disclosure.

[9] Another object of the present invention is to provide a method for transmitting a bitstream generated by an image encoding method or equipment according to the present disclosure.

[10] The technical problems solved by the present invention are not limited to the above technical problems, and other technical problems that are not described herein should become apparent to those skilled in the art from the following description.

Technical solution

[11] An image decoding method according to one aspect of the present invention comprises obtaining, from the bit stream, network abstraction layer (NAL) unit type information of a video coding layer (VCL) of the current frame and first flag information indicating whether or not a subframe included into the current frame, as a single frame, determining a NAL unit type of each of the plurality of slices included in the current frame based on the obtained NAL unit VCL type information, and decoding the plurality of slices based on the determined NAL unit type and the first flag information. The current frame may contain two or more subframes based on at least some of a plurality of slices having different types of NAL units, and the first flag information may have a predetermined value indicating that each of the two or more subframes is treated as one frame based on at least some of a plurality of slices having different types of NAL units.

[12] Image decoding equipment according to another aspect of the present invention includes a storage device and at least one processor. The at least one processor may obtain, from the bit stream, network abstraction layer (NAL) unit type information of the video coding layer (VCL) of the current frame and first flag information indicating whether or not a subframe included in the current frame is treated in as one frame, determine the NAL unit type of each of the plurality of slices included in the current frame based on the received NAL unit VCL type information, and decode the plurality of slices based on the determined NAL unit type and the first flag information. The current frame may contain two or more subframes based on at least some of a plurality of slices having different types of NAL units, and the first flag information may have a predetermined value indicating that each of the two or more subframes is treated as one frame based on at least some of a plurality of slices having different types of NAL units.

[13] An image encoding method according to another aspect of the present invention comprises segmenting a current frame into one or more subframes, determining a network abstraction layer (NAL) unit type of each of a plurality of slices included in the one or more subframes, and encoding the plurality of slices based on the determined NAL unit type. The current frame may be segmented into two or more subframes based on at least some of a plurality of slices having different types of NAL units, and each of the two or more subframes may be treated as a single frame based on at least some of multiple slices having different types of NAL units.

[14] In addition, a computer-readable recording medium according to another aspect of the present invention can store a bit stream generated by an image encoding method or image encoding equipment of the present invention.

[15] In addition, the transmission method according to another aspect of the present invention can transmit a bit stream generated by the image encoding equipment or the image encoding method of the present invention.

[16] The features briefly summarized above regarding the present invention are merely exemplary aspects of the following detailed description of the present invention and do not limit the scope of the present invention.

Advantages of the invention

[17] According to the present disclosure, it is possible to provide an image encoding/decoding method and equipment with improved encoding/decoding efficiency.

[18] In addition, according to the present disclosure, it is possible to provide an image encoding/decoding method and equipment based on a mixed NAL unit type.

[19] In addition, according to the present disclosure, it is possible to provide a method and equipment for encoding/decoding images based on two or more subframes having different mixed types of NAL units.

[20] In addition, according to the present disclosure, it is possible to provide a method for transmitting a bitstream generated by an image encoding method or equipment according to the present disclosure.

[21] In addition, according to the present disclosure, it is possible to provide a recording medium storing a bit stream generated by the image encoding method or equipment according to the present disclosure.

[22] In addition, according to the present disclosure, it is possible to provide a recording medium storing a bit stream received, decoded, and used to reconstruct an image by the image decoding equipment according to the present disclosure.

[23] Those skilled in the art will appreciate that the advantages that may be achieved through the present disclosure are not limited to those specifically described above, and other advantages of the present invention will be more clearly understood from the detailed description.

Brief description of drawings

[24] FIG. 1 is a view schematically illustrating a video coding system to which an embodiment of the present invention is applicable.

[25] FIG. 2 is a view schematically illustrating image encoding equipment to which an embodiment of the present invention is applicable.

[26] FIG. 3 is a view schematically illustrating image decoding equipment to which an embodiment of the present invention is applicable.

[27] FIG. 4 is a flowchart illustrating an image decoding procedure to which an embodiment of the present invention is applicable.

[28] FIG. 5 is a flowchart illustrating an image encoding procedure to which an embodiment of the present invention is applicable.

[29] FIG. 6 is a view illustrating an example of a multi-layer structure for a coded image/video.

[30] FIG. 7 is a view illustrating a frame parameter set (PPS) according to an embodiment of the present invention.

[31] FIG. 8 is a view illustrating a slice header according to an embodiment of the present invention.

[32] FIG. 9 is a view illustrating an example of a subframe.

[33] FIG. 10 is a view illustrating an example of a frame having a mixed NAL unit type.

[34] FIG. 11 is a view illustrating an example of a frame parameter set (PPS) according to an embodiment of the present invention.

[35] FIG. 12 is a view illustrating an example of a sequence parameter set (SPS) according to an embodiment of the present invention.

[36] FIG. 13 is a view illustrating a decoding order and an output order for each frame type.

[37] FIG. 14 is a flowchart illustrating a method for determining the NAL unit type of a current frame according to an embodiment of the present invention.

[38] FIG. 15 is a flowchart illustrating an image encoding method according to an embodiment of the present invention.

[39] FIG. 16 is a flowchart illustrating an image decoding method according to an embodiment of the present invention.

[40] FIG. 17 is a view illustrating a content streaming system to which an embodiment of the present invention is applicable.

Optimal mode for carrying out the invention

[41] Embodiments of the present invention are described in detail hereinafter with reference to the accompanying drawings so that they can be easily implemented by those skilled in the art. However, the present disclosure may be implemented in a variety of forms and is not limited to the embodiments described herein.

[42] In describing the present invention, if it is determined that a detailed description of a related known function or structure would lead to unnecessary ambiguity in the scope of the present invention, its detailed description is omitted. In the drawings, parts not related to the description of the present invention are omitted, and like reference numbers are added to like parts.

[43] In the present disclosure, when a component is “connected,” “coupled,” or “linked” to another component, this may include not only a direct connection-based relationship, but also an indirect connection-based relationship in which an intermediate component is present. In addition, when a component “includes” or “has” other components, it means that other components may be additionally included, not the exclusion of other components, unless otherwise noted.

[44] In the present disclosure, the terms “first”, “second”, etc. may be used only for the purpose of distinguishing one component from other components and does not limit the order or importance of the components unless otherwise noted. Accordingly, within the scope of the present invention, a first component in one embodiment may be referred to as a “second component” in another embodiment, and likewise, a second component in one embodiment may be referred to as a “first component” in another embodiment.

[45] In the present disclosure, components that are different from each other are intended to clearly describe each feature and do not mean that the components are necessarily separate. Thus, multiple components may be integrated and implemented in a single hardware or software module, or a single component may be distributed and implemented in multiple hardware or software modules. Therefore, even if not stated otherwise, such embodiments in which components are integrated or a component is distributed are also included within the scope of the present invention.

[46] In the present disclosure, the components described in the various embodiments do not necessarily mean essential components, and some components may be optional components. Accordingly, an embodiment consisting of a subset of the components described in the embodiment is also included within the scope of the present invention. In addition, embodiments including other components in addition to the components described in the various embodiments are included within the scope of the present invention.

[47] The present disclosure relates to image encoding and decoding, and the terms used in the present disclosure may have a general meaning commonly used in the art to which the present disclosure pertains, unless defined for the first time in the present disclosure.

[48] In the present disclosure, "frame" generally means a unit representing one image at a particular time period, and a slice/tile is an encoding unit constituting part of a frame, and one frame may consist of one or more slices/ tiles In addition, a slice/tile may include one or more coding tree units (CTUs).

[49] In the present disclosure, "pixel" or "pixel" can mean the smallest unit constituting one frame (or image). In addition, "sample" can be used as a term corresponding to a pixel. A sample may generally represent a pixel or a pixel value, and may represent only a pixel/pixel value of a luma component or only a pixel/pixel value of a chroma component.

[50] In the present disclosure, a "unit" may represent a basic unit of image processing. The unit may include at least one of a specific frame area and information associated with the area. The unit may be used interchangeably with terms such as "sample array", "block" or "zone" in some cases. In general, an MxN block may include samples (or arrays of samples) or a set (or array) of transform coefficients of M columns and N rows.

[51] In the present disclosure, "current block" may mean one of "current encoding block", "current encoding unit", "target encoding block", "target decoding block" or "target processing block". When prediction is performed, "current block" may mean "current prediction block" or "target prediction block". When transform (inverse transform)/quantization (dequantization) is performed, "current block" can mean "current transform block" or "transform target block". When filtering is performed, "current block" can mean "target filtering block".

[52] In addition, in the present disclosure, "current block" may mean a block including both a luminance component block and a chrominance component block, or a "current block luminance block" unless explicitly stated in as a block of color signals. The luminance signal component block of the current block may be expressed by including an explicit description of the luminance signal component block, such as "luminance signal block" or "current luminance signal block". In addition, the "current block chroma component block" may be expressed by including an explicit description of the chroma component block, such as "chrominance block" or "current chrominance block".

[53] In the present disclosure, the terms “/” and “,” are to be interpreted to indicate “and/or.” For example, the expression "A/B" and "A, B" can mean "A and/or B". Additionally, "A/B/C" and "A/B/C" may mean "at least one of A, B and/or C".

[54] In the present disclosure, the term “or” should be interpreted to indicate “and/or.” For example, the expression "A or B" may contain 1) only "A", 2) only "B", and/or 3) "both A and B". In other words, in the present disclosure, the term “or” is to be interpreted as indicating “in addition or alternatively.”

[55] Video Coding System Overview

[56] FIG. 1 is a view schematically illustrating a video encoding system according to the present disclosure.

[57] The video encoding system according to an embodiment may include encoding equipment 10 and decoding equipment 20. Encoding equipment 10 may deliver encoded information or video and/or image data to decoding equipment 20 in the form of a file or streaming via a digital storage medium or network.

[58] The encoding equipment 10 according to an embodiment may include a video source generator 11, an encoding unit 12, and a transmitter 13. The decoding equipment 20 according to an embodiment may include a receiver 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 may be referred to as a “video/image encoding unit,” and the decoding unit 22 may be referred to as a “video/image decoding unit.” The transmitter 13 may be included in the encoding module 12. The receiver 21 may be included in the decoding module 22. The rendering module 23 may include a display, and the display may be configured as a separate device or an external component.

[59] The video source generator 11 may obtain video/image through a video/image capturing, synthesizing, or shaping process. The video source generator 11 may include a video/image capture device and/or a video/image generation device. The video/image capturing device may include, for example, one or more cameras, video/image archives including previously captured video/images, and the like. The video/image generating apparatus may include, for example, computers, tablet computers and smartphones and may (electronically) generate video/images. For example, a virtual video/image may be generated via a computer or the like. In this case, the video/image capturing process can be replaced by the associated data generation process.

[60] The encoding unit 12 can encode the input video/image. The encoding unit 12 may perform a series of procedures such as prediction, transform and quantization for compression and encoding efficiency. The encoding unit 12 can output encoded data (encoded video/image information) in the form of a bitstream.

[61] The transmitter 13 may transmit encoded information or video/image data output in the form of a bitstream to the receiver 21 of the decoding equipment 20 via a digital storage medium or a network in the form of a file or streaming. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-Ray, HDD, SSD, and the like. The transmitter 13 may include an element for generating a multimedia file through a predefined file format and may include an element for transmission through a broadcast/communications network. The receiver 21 may retrieve/receive a bit stream from the storage medium or network and transmit the bit stream to the decoding unit 22.

[62] The decoding unit 22 can decode a video/image by performing a series of procedures such as dequantization, deconversion and prediction corresponding to the operation of the encoding unit 12.

[63] The rendering unit 23 may prepare a decoded video/image by rendering. The video/image prepared by rendering can be displayed through the display.

[64] General introduction to image encoding equipment

[65] FIG. 2 is a view schematically illustrating image encoding equipment to which an embodiment of the present invention is applicable.

[66] As shown in FIG. 2, the image encoding equipment 100 may include an image segmentation unit 110, a subtractor 115, a converter 120, a quantizer 130, a dequantizer 140, an inverse converter 150, an adder 155, a filter 160, a memory 170, an inter-prediction unit 180, an intra-prediction unit 185 and an entropy encoder 190. The inter-prediction module 180 and the intra-prediction module 185 may be collectively referred to as a “prediction module”. Converter 120, quantizer 130, dequantizer 140, and inverse converter 150 may be included in the residual processor. The residual processor may further include a subtractor 115.

[67] All or at least a portion of the plurality of components configuring the image encoding equipment 100 may be configured by a single hardware component (eg, an encoder or processor) in some embodiments. In addition, storage device 170 may include a decoded frame buffer (DPB) and may be configured with a digital storage medium.

[68] The image segmentation module 110 may segment an input image (either a frame or a movie frame) input to the image encoding equipment 100 into one more processing unit. For example, the processing unit may be called a "coding unit (CU)". A coding unit may be obtained by recursively segmenting a coding tree unit (CTU) or largest coding unit (LCU) according to a quadtree, binary tree, and ternary tree (QT/BT/TT) structure. For example, one coding unit may be segmented into multiple coding units of greater depth based on a quadtree structure, a binary tree structure, and/or a ternary structure. To segment a coding unit, a quadtree structure may first be applied, and subsequently a binary tree structure and/or a ternary structure may be applied. The encoding procedure of the present disclosure may be performed based on a final encoding unit that is no longer segmented. The largest coding unit may be used as the final coding unit, or a coding unit of greater depth obtained by segmenting the largest coding unit may be used as the final coding unit. Here, the encoding procedure may include a prediction, transformation and reconstruction procedure, which is described below. As another example, the processing unit of the encoding procedure may be a prediction unit (PU) or a transformation unit (TU). The prediction unit and the transformation unit may be split or segmented from the final encoding unit. The prediction unit may be a sample prediction unit, and the transformation unit may be a unit for extracting a transformation coefficient and/or a unit for extracting a residual signal from the transformation coefficient.

[69] The prediction module (inter-prediction module 180 or intra-prediction module 185) may perform prediction regarding a block to be processed (the current block) and generate a predicted block including prediction samples for the current block. The prediction module may determine whether intra prediction or inter prediction is applied based on the current block or CU. The prediction module may generate various information related to prediction of the current block and transmit the generated information to the entropy encoder 190. Information regarding the prediction may be encoded in the entropy encoder 190 and output in the form of a bit stream.

[70] The intra prediction module 185 may predict the current block by reference to the samples in the current frame. Samples for reference may be located in the surroundings of the current block or may be spaced apart according to an intra-prediction mode and/or intra-prediction technology. The intra prediction modes may include a plurality of non-directional modes and a plurality of directed modes. The omnidirectional mode may include, for example, DC mode and planar mode. The directional mode may include, for example, 33 directional prediction modes or 65 directional prediction modes according to the granularity of the prediction direction. However, this is merely an example; more or less directional prediction modes may be used depending on the setting. The intra prediction module 185 may determine the prediction mode applied to the current block by using the prediction mode applied to the adjacent block.

[71] The inter-prediction unit 180 may extract a predicted block for the current block based on a reference block (an array of reference samples) indicated by a motion vector for the reference frame. In this case, in order to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of the motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference frame index. The motion information may further include inter-prediction direction information (L0-prediction, L1-prediction, bi-prediction, etc.). In the case of inter-prediction, the neighbor block may include a spatial neighbor block present in the current frame and a temporal neighbor block present in the reference frame. The reference frame including the reference block and the reference frame including the temporary adjacent block may be identical or different. A temporary neighbor block may be referred to as a "co-located reference block", "co-located CU (colCU)", or the like. A reference frame including a temporary adjacent block may be referred to as a "co-located frame (colPic)". For example, the inter-prediction module 180 may configure a list of motion information candidates based on neighboring blocks and generate information indicating which candidate is used to retrieve the motion vector and/or reference frame index of the current block. Mutual prediction can be performed based on different prediction modes. For example, in the case of the skip mode and the merge mode, the inter-prediction unit 180 may use the motion information of a neighboring block as the motion information of the current block. In the case of the skip mode, unlike the combining mode, the residual signal may not be transmitted. In the case of the motion vector prediction (MVP) mode, the motion vector of an adjacent block can be used as a motion vector predictor, and the motion vector of the current block can be signaled by encoding the motion vector difference and an indicator for the motion vector predictor. The motion vector difference may mean the difference between the motion vector of the current block and the motion vector predictor.

[72] The prediction module can generate a prediction signal based on various prediction methods and prediction technologies described below. For example, the prediction module may not only apply intra-prediction or inter-prediction, but also simultaneously apply both intra-prediction and inter-prediction to predict the current block. A prediction method that simultaneously uses both intra prediction and inter prediction to predict the current block may be called "combined inter and intra prediction (CIIP)". In addition, the prediction module can perform intra-block copying (IBC) to predict the current block. In-block copying can be used to encode game image/video content and the like, such as screen content encoding (SCC). IBC is a method of predicting the current frame using a previously reconstructed reference block in the current frame at a location spaced from the current block by a predetermined distance. When IBC is applied, the location of the reference block in the current frame can be encoded as a vector (block vector) corresponding to a predetermined distance. IBC essentially performs prediction within the current frame, but can be performed similarly to inter-prediction in that the reference block is retrieved within the current frame. Thus, the IBC may use at least one of the inter-prediction technologies described in the present disclosure.

[73] The prediction signal generated by the prediction module can be used to generate a reconstructed signal or generate a residual signal. The subtractor 115 may generate a residual signal (a residual block or an array of residual samples) by subtracting a prediction signal (a prediction block or an array of prediction samples) output from the prediction module from an input image signal (a source block or an array of original samples). The generated residual signal can be transmitted to the converter 120.

[74] Converter 120 can generate conversion coefficients by applying conversion technology to the residual signal. For example, the transform technology may include at least one of a discrete cosine transform (DCT), a discrete sine transform (DST), a Karhunen-Loeve transform (KLT), a graph-based transform (GBT), or a conditionally nonlinear transform (CNT). ). Here, GBT means a graph-derived transformation when the relationship information between pixels is represented by the graph. CNT means the transformation obtained from the prediction signal generated using all the previously reconstructed pixels. In addition, the conversion process can be applied to square pixel blocks that are identical in size, or can be applied to blocks that are variable in size rather than square.

[75] Quantizer 130 may quantize the transform coefficients and pass them to entropy encoder 190. Entropy encoder 190 may encode the quantized signal (information regarding the quantized transform coefficients) and output a bit stream. Information regarding the quantized transform coefficients may be referred to as "residual information". The quantizer 130 may re-arrange the quantized block-type transform coefficients into one-dimensional vector form based on the scanning order of the coefficients, and generate information regarding the quantized transform coefficients based on the quantized transform coefficients in one-dimensional vector form.

[76] Entropy encoder 190 may implement various encoding techniques, such as, for example, exponential Golomb coding, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), and the like. Entropy encoder 190 may encode information needed to reconstruct video/images other than quantized transform coefficients (eg, syntax element values, etc.), together or separately. Encoded information (eg, encoded video/image information) may be transmitted or stored in network abstraction layer (NAL) units in the form of a bit stream. The video/image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a frame parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/image information may further include general restriction information. The signaling information, transmission information, and/or syntax elements described in the present disclosure may be encoded through the encoding procedure described above and included in the bit stream.

[77] The bitstream may be transmitted over a network or may be stored on a digital storage medium. The network may include a broadcast network and/or a communications network, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-Ray, HDD, SSD, and the like. A transmitter (not shown) transmitting the signal output from the entropy encoder 190 and/or a storage module (not shown) storing the signal may be included as an internal/external element of the encoding equipment 100, and alternatively, the transmitter may be included in the entropy encoder 190 .

[78] The quantized transform coefficients output from quantizer 130 can be used to generate a residual signal. For example, a residual signal (residual block or residual samples) may be recovered by applying dequantization and inverse transform to the quantized transform coefficients via dequantizer 140 and inverse transformer 150.

[79] An adder 155 adds the reconstructed residual signal with the prediction signal output from the inter-prediction module 180 or the intra-prediction module 185 so as to generate a reconstructed signal (reconstructed frame, reconstructed block, reconstructed sample array). If there is no remainder for a block to be processed, for example in a case in which the skip mode is applied, the predicted block can be used as a reconstructed block. The adder 155 may be referred to as a "recovery module" or a "recovered block generator." The generated reconstructed signal can be used to internally predict the next block to be processed in the current frame, and can be used to inter-predict the next frame through filtering, as described below.

[80] Filter 160 may enhance the subjective/objective quality of images by applying filtering to the reconstructed signal. For example, filter 160 may generate a modified reconstructed frame by applying various filtering techniques to the reconstructed frame and store the modified reconstructed frame in storage device 170, namely, a DPB of storage device 170. Various filtering techniques may include, for example, deblocking filtering. , filtering based on sampled adaptive offset, adaptive loop filter, bilateral filter, etc. The filter 160 may generate various filtering-related information and transmit the generated information to the entropy encoder 190, as described below in the description of each filtering method. Information associated with filtering may be encoded by entropy encoder 190 and output in the form of a bitstream.

[81] The modified reconstructed frame transferred to the storage device 170 can be used as a reference frame in the inter-prediction module 180. When mutual prediction is applied through the image encoding equipment 100, prediction mismatch between the image encoding equipment 100 and the image decoding equipment can be eliminated, and encoding efficiency can be improved.

[82] The DPB of the storage device 170 may store the modified reconstructed frame for use as a reference frame in the inter-prediction module 180 . The storage device 170 may store block motion information from which motion information in the current frame is retrieved (or encoded) and/or motion information of blocks in a frame that have already been reconstructed. The stored motion information may be transmitted to the inter-prediction unit 180 and used as spatial neighbor block motion information or temporal neighbor block motion information. Memory 170 may store reconstructed samples of reconstructed blocks in the current frame and may transmit reconstructed samples to intra prediction module 185.

[83] General introduction of image decoding equipment

[84] FIG. 3 is a view schematically illustrating image decoding equipment to which an embodiment of the present invention is applicable.

[85] As shown in FIG. 3, the image decoding equipment 200 may include an entropy decoder 210, a dequantizer 220, an inverse converter 230, an adder 235, a filter 240, a memory 250, an inter-prediction unit 260, and an intra-prediction unit 265. The inter-prediction module 260 and the intra-prediction module 265 may be collectively referred to as a “prediction module”. A dequantizer 220 and an inverse converter 230 may be included in the residual processor.

[86] All or at least a portion of the plurality of components configuring the image decoding equipment 200 may be configured by a hardware component (eg, a decoder or processor) according to an embodiment. In addition, storage device 250 may include a decoded frame buffer (DPB) or may be configured with a digital storage medium.

[87] The image decoding equipment 200, which receives the bit stream including video/image information, can reconstruct the image by performing a process corresponding to the process performed by the image encoding equipment 100 in FIG. 2. For example, the image decoding equipment 200 may perform decoding using a processing unit used in the image encoding equipment. Thus, a decoding processing unit, for example, may be an encoding unit. A coding unit may be obtained by segmenting a coding tree unit or a largest coding unit. The reconstructed image signal decoded and output by the decoding equipment 200 can be reproduced by the playback equipment.

[88] The image decoding equipment 200 may receive a signal output from the image encoding equipment of FIG. 2 in the form of a bit stream. The received signal may be decoded through entropy decoder 210. For example, entropy decoder 210 may parse the bitstream to extract information (eg, video/image information) needed to reconstruct images (or reconstruct frames). The video/image information may further include information regarding various parameter sets, such as an adaptation parameter set (APS), a frame parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). In addition, the video/image information may further include general restriction information. The image decoding equipment may further decode the frame based on information regarding the parameter set and/or general constraint information. The signaling/receiving information and/or syntax elements described in the present disclosure may be decoded by a decoding procedure and obtained from the bitstream. For example, entropy decoder 210 decodes information in a bitstream based on an encoding method, such as exponential Golomb, CAVLC, or CABAC encoding, and the output values of the syntax elements required for image reconstruction and the quantized values of the transform coefficients for the remainder. More specifically, the entropy CABAC decoding method may take a resolution element corresponding to each syntax element in the bitstream, determine a context model using information of a target decoding syntax element, decoding information of an adjacent block and a target decoding block, or information of a symbol/resolution element decoded on a previous one. stage, and perform arithmetic decoding for the resolution element by predicting the probability of occurrence of the permission element according to the determined context model and generating a symbol corresponding to the meaning of each syntactic element. In this case, the entropy CABAC decoding method can update the context model by using the decoded symbol/grant element information for the context model of the next symbol/grant element after determining the context model. The prediction-related information from the information decoded by the entropy decoder 210 may be provided to the prediction module (inter-prediction module 260 and intra-prediction module 265), and the residual value for which entropy decoding is performed in the entropy decoder 210, i.e. quantized conversion coefficients and related parameter information may be input to the dequantizer 220. In addition, information regarding filtering from information decoded by the entropy decoder 210 may be provided to the filter 240. Meanwhile, a receiver (not shown) for receiving a signal output from the equipment image encoding device may be further configured as an internal/external element of image decoding equipment 200, or the receiver may be a component of entropy decoder 210.

[89] Meanwhile, the image decoding equipment according to the present disclosure may be referred to as “video/image/frame decoding equipment.” The image decoding equipment can be classified into an information decoder (video/image/frame information decoder) and a sample decoder (video/image/frame sample decoder). The information decoder may include an entropy decoder 210. The sample decoder may include at least one of a dequantizer 220, an inverse converter 230, an adder 235, a filter 240, a memory 250, an inter-prediction module 160, or an intra-prediction module 265.

[90] Dequantizer 220 may dequantize the quantized transform coefficients and output the transform coefficients. Dequantizer 220 may re-arrange the quantized transform coefficients in the form of a two-dimensional block. In this case, rearrangement may be performed based on the order of coefficient scanning performed in the image encoding equipment. Dequantizer 220 may perform dequantization on the quantized transform coefficients by using a quantization parameter (eg, quantization step size information) and obtain the transform coefficients.

[91] The inverse converter 230 may inversely convert the transform coefficients to obtain a residual signal (residual block, residual sample array).

[92] Prediction module 230 may perform prediction for the current block and may generate a predicted block including prediction samples for the current block. The prediction module can determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoder 210, and can determine a particular intra/inter prediction mode (prediction technology).

[93] Identical to that described in the prediction module of the image encoding equipment 100, the prediction module can generate a prediction signal based on various prediction methods (techniques), which are described below.

[94] Intra prediction module 265 may predict the current block by reference to samples in the current frame. The description of the intra prediction module 185 applies equally to the intra prediction module 265.

[95] The inter-prediction module 260 may extract a predicted block for the current block based on a reference block (an array of reference samples) indicated by a motion vector for the reference frame. In this case, in order to reduce the amount of motion information transmitted in the inter-prediction mode, the motion information may be predicted in units of blocks, sub-blocks or samples based on the correlation of the motion information between a neighboring block and the current block. The motion information may include a motion vector and a reference frame index. The motion information may further include inter-prediction direction information (L0-prediction, L1-prediction, bi-prediction, etc.). In the case of inter-prediction, the neighbor block may include a spatial neighbor block present in the current frame and a temporal neighbor block present in the reference frame. For example, the inter-prediction module 260 may configure a motion information candidate list based on neighboring blocks and extract the current block's motion vector and/or reference frame index based on the received candidate selection information. The inter-prediction may be performed based on various prediction modes, and the information regarding the prediction may include information indicating the inter-prediction mode for the current block.

[96] The adder 235 can generate a reconstructed signal (recovered frame, reconstructed block, reconstructed sample array) by adding the resulting residual signal with a prediction signal (predicted block, predicted sample array) output from a prediction module (including inter-prediction module 260 and /or internal prediction module 265). If there is no remainder for a block to be processed, for example when a skip mode is applied, the predicted block can be used as the reconstructed block. The description of adder 155 is equally applicable to adder 235. Adder 235 may be referred to as a “recovery module” or a “recovered block generator.” The generated reconstructed signal can be used to internally predict the next block to be processed in the current frame, and can be used to inter-predict the next frame through filtering, as described below.

[97] Filter 240 may enhance the subjective/objective quality of images by applying filtering to the reconstructed signal. For example, filter 240 may generate a modified reconstructed frame by applying various filtering techniques to the reconstructed frame and store the modified reconstructed frame in storage device 250, namely, a DPB of storage device 250. Various filtering techniques may include, for example, deblocking filtering. , filtering based on sampled adaptive offset, adaptive loop filter, bilateral filter, etc.

[98] The (modified) reconstructed frame stored in the DPB of the storage device 250 can be used as a reference frame in the inter-prediction unit 260. Memory 250 may store block motion information from which motion information in the current frame is retrieved (or decoded) and/or motion information of blocks in a frame that have already been reconstructed. The stored motion information may be transmitted to the inter-prediction unit 260 so that it is used as spatial neighbor block motion information or temporal neighbor block motion information. Memory 250 may store reconstructed samples of reconstructed blocks in the current frame and transmit the reconstructed samples to intra prediction module 265.

[99] In the present disclosure, the embodiments described in the filter 160, the inter-prediction module 180, and the intra-prediction module 185 of the image encoding equipment 100 may be equally or correspondingly applied to the filter 240, the inter-prediction module 260, and the intra-prediction module 265 image decoding equipment 200.

[100] General procedure for image/video encoding

[101] When encoding images/videos, a frame configuring the image/video may be encoded/decoded according to the decoding order. The order of frames corresponding to the output order of the decoded frame may be set differently from the decoding order, and based on this, not only forward prediction but also backward prediction can be performed during inter prediction.

[102] FIG. 4 is a flowchart illustrating an image decoding procedure to which the embodiment(s) of the present invention is applicable.

[103] Each procedure shown in FIG. 4 may be performed by the image decoding equipment of FIG. 3. For example, step S410 may be performed by entropy decoder 210, step S420 may be performed by prediction units 265 and 260, step S430 may be performed by residual processor 220 and 230, step S440 may be performed by adder 235, and step S450 may be performed by filter 240. Step S410 may include the information decoding procedure described in the present disclosure, step S420 may include the inter/intra prediction procedure described in the present disclosure, step S430 may include the residual processing procedure described in the present disclosure. In essence, step S440 may include the block/frame recovery procedure described in the present disclosure, and step S450 may include the in-loop filtering procedure described in the present disclosure.

[104] Referring to FIG. 4, the frame decoding procedure may schematically include a procedure (S410) for obtaining image/video information (by decoding) from the bit stream, a frame reconstruction procedure (S420-S440), and an in-loop filtering procedure (S450) for the reconstructed frame. The frame reconstruction procedure may be performed based on the prediction samples and residual samples obtained through inter/intra prediction (S420) and residual processing (S430) (dequantization and inverse transformation of the quantized transform coefficient) described in the present disclosure. The modified reconstructed frame may be generated through an in-loop filtering procedure for the reconstructed frame generated through the frame restoration procedure. In this case, the modified reconstructed frame may be output as a decoded frame, stored in a decoded frame buffer (DPB) of storage device 250, and used as a reference frame in an inter-prediction procedure when decoding the frame subsequently. The procedure (S450) for in-circuit filtration can be omitted. In this case, the reconstructed frame may be output as a decoded frame, stored in the DPB of the memory 250, and used as a reference frame in an inter-prediction procedure when decoding the frame subsequently. The in-loop filtering procedure (S450) may include a deblocking filtering procedure, a sampled adaptive offset (SAO) filtering procedure, an adaptive loop filtering (ALF) procedure, and/or a bilateral filtering procedure as described above, some or all of which may fall. In addition, one or some of the deblocking filtering procedure, the sampled adaptive offset (SAO) filtering procedure, the adaptive contour filtering (ALF) procedure, and/or the bilateral filtering procedure may be applied sequentially, or all of them may be applied sequentially. For example, after a deblocking filtering procedure is applied to the reconstructed frame, an SAO procedure may be performed. Alternatively, after the deblocking filtering procedure is applied to the reconstructed frame, the ALF procedure may be executed. This can be done similarly even in encoding hardware.

[105] FIG. 5 is a flowchart illustrating an image encoding procedure to which an embodiment of the present invention is applicable.

[106] Each procedure shown in FIG. 5 may be performed by the image encoding equipment of FIG. 2. For example, step S510 may be performed by prediction units 185 and 180, step S520 may be performed by residual processor 115, 120, and 130, and step S530 may be performed by entropy encoder 190. Step S510 may include an inter/intra prediction procedure, described in the present disclosure, step S520 may include the residual processing procedure described in the present disclosure, and step S530 may include the information encoding procedure described in the present disclosure.

[107] Referring to FIG. 5, the frame encoding procedure may schematically include not only a procedure for encoding and outputting information for reconstructing frames (eg, prediction information, residual information, segmentation information, etc.) in the form of a bitstream, but also a procedure for generating the reconstructed frame for the current frame and a procedure (optional) for applying in-loop filtering to the reconstructed frame. The encoding equipment may extract the (modified) residual samples from the quantized transform coefficient through the dequantizer 140 and the inverse converter 150, and generate a reconstructed frame based on the prediction samples, which are the output of step S510, and the (modified) residual samples. The reconstructed frame generated in this manner may be equal to the reconstructed frame generated in the decoding equipment. The modified reconstructed frame may be generated through an in-loop filtering procedure for the reconstructed frame. In this case, the modified reconstructed frame can be stored in the decoded frame buffer of the storage device 170 and can be used as a reference frame in the inter-prediction procedure when encoding the frame subsequently, similar to the decoding equipment. As described above, in some cases, part or all of the in-loop filtration procedure may be omitted. When the in-loop filtering procedure is performed, the (in-loop) filtering-related information (parameter) can be encoded in the entropy encoder 190 and output in the form of a bitstream, and the decoding equipment can perform the in-loop filtering procedure using an identical method to that in the encoding equipment based on the associated filtering information.

[108] Through such an in-loop filtering procedure, noise generated during image/video encoding, such as blocking artifact and ringing artifact, can be reduced, and subjective/objective visual quality can be improved. In addition, by performing the in-loop filtering procedure in both the encoding equipment and the decoding equipment, the encoding equipment and the decoding equipment can extract the same prediction result, the reliability of frame encoding can be improved, and the amount of data that must be transmitted for frame encoding can be reduced.

[109] As described above, the frame restoration procedure may be performed not only in the image decoding equipment, but also in the image encoding equipment. A reconstructed block may be generated based on intra-prediction/inter-prediction in block units, and a reconstructed frame including the reconstructed blocks may be generated. When the current frame/slice/tile group is an I-frame/slice/tile group, the blocks included in the current frame/slice/tile group can only be reconstructed based on internal prediction. On the other hand, when the current frame/slice/tile group is a P- or B-frame/slice/tile group, blocks included in the current frame/slice/tile group can be reconstructed based on intra-prediction or inter-prediction. In this case, intra-prediction can be applied to some blocks in the current frame/slice/tile group, and inter-prediction can be applied to the remaining blocks. The color component of a frame may include a luma component and a chrominance component, and the methods and embodiments of the present invention are applicable to the luma component and the chrominance component unless expressly limited in the present disclosure.

[110] Example of coding layer structure

[111] The encoded video/image of the present invention can be processed, for example, according to the encoding layer and structure that is described below.

[112] FIG. 6 is a view illustrating a multi-layer structure for a coded image.

[113] The encoded image is classified into a video coding layer (VCL) for the process of decoding images and processing itself, a bottom system for transmitting and storing the encoded information, and a network abstraction layer (NAL) present between the VCL and the bottom system and responsible for the network adaptation function .

[114] In VCL, VCL data may be generated including compressed image data (slice data), or a parameter set may be generated including information such as a frame parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS) or additional enhancement information (SEI) message additionally required for the picture decoding process.

[115] In NAL, header information (NAL unit header) may be added to the primary byte data sequence (RBSP) generated in the VCL to form the NAL unit. In this case, RBSP means the slice data, parameter set, SEI message generated in the VCL. The NAL unit header may include NAL unit type information indicated according to the RBSP data included in the corresponding NAL unit.

[116] As shown in FIG. 6, a NAL unit can be classified into a VCL NAL unit and a non-VCL NAL unit according to the type of RBSP generated in the VCL. A VCL NAL unit may refer to a NAL unit including information regarding an image (slice data), and a non-VCL NAL unit may refer to a NAL unit including information (a set of parameters or SEI message) required to to decode the image.

[117] The VCL NAL unit and the non-VCL NAL unit may have header information attached and are transmitted over the network according to the underlying system data format. For example, a NAL unit may be modified into data having a predefined data format, such as H.266/VVC file format, RTP (Real Time Transport Protocol), or TS (Transport Stream), and transmitted over various networks.

[118] As described above, in a NAL unit, a NAL unit type may be indicated according to an RBSP data structure included in a corresponding NAL unit, and information regarding the NAL unit type may be stored in a NAL unit header and transmitted in signaling signals . For example, it can be broadly classified into a VCL NAL unit type and a non-VCL NAL unit type based on whether or not the NAL unit includes image information (slice data). The VCL NAL unit type can be classified according to the property and frame type, and the non-VCL NAL unit type can be classified according to the parameter set type.

[119] The following is an example of a NAL unit type indicated according to the type of parameter set/information included in the non-VCL NAL unit type.

[120] - DCI (Decoding Characteristics Information) NAL Unit Type (NUT): Type for the NAL Unit including the DCI

[121] - NUT VPS (Video Parameter Set): type for a NAL unit including a VPS

[122] - NUT SPS (Sequence Parameter Set): type for a NAL unit including an SPS

[123] - NUT PPS (frame parameter set): type for a NAL unit including a PPS

[124] - NUT APS (Adaptation Parameter Set): type for a NAL unit including an APS

[125] - NUT PH (frame header): Type for NAL unit including PH

[126] The above-described NAL unit types may have syntax information regarding the NAL unit types, and the syntax information may be stored in a NAL unit header and transmitted in signaling signals. For example, the syntax information may include nal_unit_type, and NAL unit types may be indicated by the nal_unit_type value.

[127] Meanwhile, as described above, one frame may include a plurality of slices, and one slice may include a slice header and slice data. In this case, one frame header may further be added to multiple slices (slice header and slice data set) in one frame. The frame header (frame header syntax) may include information/parameters typically applicable to the frame. The slice header (slice header syntax) may include information/parameters typically applicable to the slice. APS (APS Syntax) or PPS (PPS Syntax) may include information/parameters typically applicable to one or more slices or frames. SPS (SPS syntax) may include information/parameters typically applicable to one or more sequences. VPS (VPS syntax) can information/parameters typically applicable to multiple layers. DCI (DCI syntax) may include information/parameters associated with decoding characteristics.

[128] In the present disclosure, the high-level syntax (HLS) may include at least one of APS syntax, PPS syntax, SPS syntax, VPS syntax, DCI syntax, frame header syntax, or header syntax slices. In addition, in the present disclosure, low-level syntax (LLS) may include, for example, slice data syntax, CTU syntax, coding unit syntax, transformation unit syntax, etc.

[129] In the present disclosure, image/video information encoded by the encoding equipment and signaled to the decoding equipment in the form of a bit stream may include not only intra-frame segmentation related information, intra/inter prediction information, residual information, in-loop filtering information, but also slice header information, frame header information, APS information, PPS information, SPS information, VPS information and/or DCI information. In addition, the image/video information may further include general restriction information and/or information regarding the NAL unit header.

[130] General representation of transmission in the overhead signals of input points

[131] As described above, the VCL NAL unit may include slice data as an RBSP (primary byte sequence data). The slice data may be combined in a VCL NAL unit in bytes and may include one or more subsets. At least one random access (RA) entry point may be specified for subsets, and parallel processing may be performed based on the entry point.

[132] The Versatile Video Coding (VVC) standard (ITU-T Recommendation H.266) supports wave parallel processing (WPP), which is one of various parallel processing technologies. Multiple slices in a frame can be encoded/decoded in parallel based on WPP.

[133] To enable parallel processing characteristics, input point information may be signaled. The image decoding equipment can directly access the starting point of the data segment included in the NAL unit based on the information of the input points. Here, the starting point of a data segment can mean the starting point of tiles in a slice or the starting point of CTU rows in a slice.

[134] Input point information may be signaled in high-level syntax, such as a frame parameter set (PPS) and/or a slice header.

[135] FIG. 7 is a view illustrating a frame parameter set (PPS) according to an embodiment of the present invention, and FIG. 8 is a view illustrating a slice header according to an embodiment of the present invention.

[136] First, referring to FIG. 7, the frame parameter set (PPS) may include entry_point_offsets_present_flag as a syntax element indicating whether or not entry point information is transmitted in signaling signals.

[137] entry_point_offsets_present_flag may indicate whether or not entry point information is signaled in a slice header referencing a frame parameter set (PPS). For example, entry_point_offsets_present_flag having a first value (eg, 0) may indicate that the signaling of entry point information for a tile or specific CTU rows in a tile is not present in the slice header. In contrast, an entry_point_offsets_present_flag having a second value (eg, 1) may indicate that the signaling of entry point information for a tile or specific CTU rows in a tile is present in the slice header.

[138] Meanwhile, although the case in which entry_point_offsets_present_flag is included in the frame parameter set (PPS) is illustrated in FIG. 7 is an example, and therefore the embodiments of the present invention are not limited to this. For example, entry_point_offsets_present_flag may be included in the Sequence Parameter Set (SPS).

[139] Next, referring to FIG. 8, the slice header may include offset_len_minus1 and entry_point_offset_minus1[i] as syntax elements for identifying an entry point.

[140] offset_len_minus1 may indicate a value obtained by subtracting 1 from the length in bits of entry_point_offset_minus1[i]. The value of offset_len_minus1 can range from 0 to 31. In an example, offset_len_minus1 can be signaled based on the NumEntryPoints variable indicating the total number of entry points. For example, offset_len_minus1 may be signaled only when NumEntryPoints is greater than 0. In addition, offset_len_minus1 may be signaled based on the entry_point_offsets_present_flag described above with reference to FIG. 7. For example, when offset_len_minus1 can only be signaled when entry_point_offsets_present_flag has a second value (eg 1) (i.e. signaling entry point information is present in the slice header).

[141] entry_point_offset_minus1[i] may indicate the offset of the i-th entry point in bytes and may be expressed by adding 1 bit to offset_len_minus1. The slice data in the NAL unit may include a number of subsets identical to the value obtained by summing 1 with NumEntryPoints and an index value indicating that each of the subsets may have a range from 0 to NumEntryPoints. The first byte of slice data in a NAL unit may be represented by byte 0.

[142] When entry_point_offset_minus1[i] is signaled, emulation prevention bytes included in the slice data in the NAL unit may be counted as part of the slice data to identify subsets. Subset 0, which is the first subset of slice data, can be configured from byte 0 to entry_point_offset_minus1[0]. Likewise, subset k, which is the kth subset of slice data, can have the configuration from firstByte[k] to lastByte[k]. Here, firstByte[k] may be retrieved as shown in Equation 1 below, lastByte[k] may be retrieved as shown in Equation 2 below.

[143] equation 1

[144] equation 2

[145] In Equations 1 and 2, k can range from 1 to the value obtained by subtracting 1 from NumEntryPoints.

[146] The last subset (ie, the NumEntryPoints subset) of the slice data may consist of the remaining bytes of the slice data.

[147] Meanwhile, if the predefined synchronization process for context variables is not performed before decoding the CTU that includes the first CTB of the CTB row in each tile (for example, sps_entropy_coding_sync_enabled_flag==0), and the slice includes one or more full tiles ,Each subset of the slice header may consist of all the ,encoded bits for all CTUs in an identical tile. In this case, the total number of slice data subsets may be equal to the total number of tiles in the slice.

[148] In contrast, when the predefined synchronization process is not performed and the slice includes one subset for CTU rows in one tile, NumEntryPoints may be 0. In this case, one subset of the slice data may consist of all encoded bits for all CTU in the slice.

[149] In contrast, when a predefined synchronization process is performed (eg sps_entropy_coding_sync_enabled_flag==1), each subset may consist of all encoded bits for all CTUs of one CTU row in one tile. In this case, the total number of slice data subsets can be equal to the total number of CTU rows of each tile in the slice.

[150] General representation of a mixed type NAL unit

[151] In general, one NAL unit type can be specified per frame. As described above, syntax information indicating the type of the NAL unit may be stored in a NAL unit header for the NAL unit and signaled. For example, the syntax information may include nal_unit_type, and NAL unit types may be indicated by the nal_unit_type value.

[152] An example of the types of NAL units to which an embodiment of the present invention is applicable is shown in Table 1 below.

[153] Tab. 1

nal_unit_type Name nal_unit_type NAL unit content and RBSP syntactic structure NAL unit type class 0 TRAIL_NUT Coded Lag Frame Slice
slice_layer_rbsp() VCL 1 STSA_NUT Coded slice of STSA frame
slice_layer_rbsp() VCL 2 RADL_NUT Coded slice of RADL frame
slice_layer_rbsp() VCL 3 RASL_NUT Coded slice of RASL frame
slice_layer_rbsp() VCL 4...6 RSV_VCL_4...RSV_VCL_6 Reserved types of non-IRAP VCL NAL units VCL 7
8 IDR_W_RADL
IDR_N_LP Coded slice of IDR frame
slice_layer_rbsp() VCL 9 CRA_NUT Coded slice of CRA frame
silce_layer_rbsp() VCL 10 GDR_NUT Encoded slice of GDR frame
slice_layer_rbsp() VCL eleven
12 RSV_IRAP_11
RSV_IRAP_12 Reserved types of IRAP VCL NAL units VCL 13 DCI_NUT Decoding characteristics information
decoding_capability_information_rbsp() non-VCL 14 VPS_NUT Video Options Set
video_parameter_set_rbsp() non-VCL 15 SPS_NUT Sequence Parameter Set
seq_parameter_set_rbsp() non-VCL 16 PPS_NUT Frame Parameter Set
pic_parameter_set_rbsp() non-VCL 17
18 PREFIX_APS_NUT
SUFFIX_APS_NUT Set of adaptation parameters
adaptation_parameter_set_rbsp() non-VCL 19 PH_NUT Frame header
picture_header_rbsp() non-VCL 20 AUD_NUT AU separator
access_unit_delimiter_rbsp() non-VCL 21 EOS_NUT End of sequence
end_of_seq_rbsp() non-VCL 22 EOB_NUT End of bitstream
end_of_bitstream_rbsp() non-VCL 23
24 PREFIX_SEI_NUT
SUFFIX_SEI_NUT Additional improving information
sei_rbsp() non-VCL 25 FD_NUT Filling data
filler_data_rbsp() non-VCL 26
27 RSV_NVCL_26
RSV_NVCL_27 Reserved types of non-VCL NAL units non-VCL 28..31 UNSPEC_28...UNSPEC_31 Unspecified types of non-VCL NAL units non-VCL NUT: NAL unit type
STSA: Step-by-step access with switching to temporary sublayer
RADL: Random Access Ahead Decodable
RASL: Random Access Skip Ahead
IDR: Instant update when decoding
LP: Leading Frame
_W_RADL: With RADL
_N_LP: non-LP, no LP
CRA: Pure Random Access
GDR: Gradual decoding update
IRAP: Indoor Random Access Point

[154] Referring to Table 1, the VCL NAL unit type can be classified into NAL unit types 0 to 12 according to the attribute and frame type. In addition, the non-VCL NAL unit type can be classified into NAL unit types 13 to 31 according to the parameter set type.

[155] Detailed examples of the VCL NAL unit type are as follows.

[156] - IRAP (Indoor Random Access Point) based NAL unit type (NUT): The type for the NAL unit of an IRAP frame, which is set to the range IDR_W_RADL to CRA_NUT.

[157] - IDR (Instantaneous Decoding Update) based NUT: The type for the NAL unit of an IDR frame, which is set to IDR_W_RADL or IDR_N_LP.

[158] - CRA (Pure Random Access) based NUT: The type for the NAL unit for a CRA frame, which is set to CRA_NUT.

[159] - RADL (Random Access Decodable Ahead) NUT: The type for the NAL unit for a RADL frame, which is set to RADL_NUT.

[160] - RASL (Random Access Skip Ahead) NUT: The type for the NAL unit for the RASL frame, which is set to RASL_NUT.

[161] - Trailing NUT: The type for the NAL unit for the trailing frame, which is set to TRAIL_NUT.

[162] - GDR (Gradual Decoding Update) based NUT: Type for the NAL unit for a GDR frame, which is GDR_NUT.

[163] - STSA-based NUT: The type for the NAL unit for an STSA frame, which is set to STSA_NUT.

[164] Meanwhile, the VVC standard allows a single frame to include multiple slices having different types of NAL units. For example, one frame may include at least one first slice having a first NAL unit type and at least one second slice having a second NAL unit type different from the first NAL unit type. In this case, the frame NAL unit type may be referred to as a mixed NAL unit type. Since the VVC standard supports mixed NAL unit type, multiple frames can be more easily reconstructed/synthesized during content synthesis process, encoding/decoding process, etc.

[165] However, according to the current scheme for a mixed NAL unit type, there is a restriction that only two types of NAL units are allowed to mix. In addition, when one frame has a mixed NAL unit type, there is a restriction that one or more VCL NAL units of the frame must be of NAL unit type in the range from IDR_W_RADL to CRA_NUT, and the remaining VCL NAL units of the frame must be of NAL type -units in the range from TRAIL_NUT to RSV_VCL_6. As a result, although the mixed NAL unit type is useful in the image processing procedure, there is a problem that the mixed NAL unit type is not commonly used.

[166] To solve such a problem, according to embodiments of the present invention, two or more types of NAL units may be allowed to be mixed, and more different mixed types of NAL units may be provided.

[167] Embodiments of the present invention are described in detail hereinafter.

[168] When one frame includes two or more slices, and the NAL unit types of the slices are different, the frame may be limited such that it includes two or more subframes. Thus, when one frame has a mixed NAL unit type, the frame may include two or more subframes.

[169] A subframe may include one or more slices and construct a rectangular area within the frame. Each of the subframes included in the frame may have a different size from each other. On the other hand, for all frames belonging to the same sequence, the sizes and positions of specific individual subframes can be set equal to each other.

[170] FIG. 9 is a view illustrating an example of a subframe.

[171] Referring to FIG. 9, one frame can be segmented into 18 tiles in total. 12 tiles of the 18 tiles may be placed on the left side of the frame, and each of the 12 tiles may include one slice including a 4x4 CTU. In addition, 6 tiles may be placed on the right side of the frame, and each of the 6 tiles may include two slices comprising a 2x2 CTU and stacked in a vertical direction. As a result, a frame may include 24 subframes and 24 slices, and each subframe may include one slice.

[172] In one embodiment, each subframe in one frame may be treated as one frame to support a mixed NAL unit type. When a subframe is treated as one frame, the subframe can be independently encoded/decoded regardless of the encoding/decoding result of another subframe. Here, independent encoding/decoding may mean that the block segmentation structure (for example, single tree structure, dual tree structure, etc.), the type of prediction mode (for example, intra prediction, inter prediction, etc.) etc.), decoding order, etc. subframes differ in structure, type, order, etc. for another subframe. For example, when the first subframe is encoded/decoded based on the intra prediction mode, a second subframe adjacent to the first subframe and treated as one frame may be encoded/decoded based on the inter prediction mode.

[173] When one frame includes two or more independent subframes, and the subframes have different NAL unit types from each other, the frame may have a mixed NAL unit type.

[174] FIG. 10 is a view illustrating an example of a frame having a mixed NAL unit type.

[175] Referring to FIG. 10, one frame 1000 may include first to third subframes 1010-1030. Each of the first and third subframes 1010 and 1030 may include two slices. In contrast, the second subframe 1020 may include four slices.

[176] When each of the first to third subframes 1010-1030 is treated as one frame, the first to third subframes 1010-1030 may be independently encoded to construct different bit streams. For example, the encoded slice data of the first subframe 1010 may be encapsulated into one or more NAL units having a NAL unit type identical to the RASL_NUT NAL unit type to construct a first bit stream (bit stream 1). In addition, the encoded slice data of the second subframe 1020 may be encapsulated into one or more NAL units having a NAL unit type identical to the RADL_NUT NAL unit type to construct a second bit stream (bit stream 2). In addition, the encoded slice data of the third subframe 1030 may be encapsulated into one or more NAL units having a NAL unit type identical to the RASL_NUT NAL unit type to construct a third bit stream (3 bit stream). As a result, one frame 1000 may have a mixed NAL unit type in which RASL_NUT and RADL_NUT are mixed.

[177] In one embodiment, all slices included in each subframe in a frame may be constrained such that they are of the same NAL unit type. For example, both of the two slices included in the first subframe 1010 may have a NAL unit type identical to the NAL unit type RASL_NUT. In addition, all four slices included in the second subframe 1020 may have a NAL unit type identical to the NAL unit type RADL_NUT. In addition, both of the two slices included in the third subframe 1030 may have a NAL unit type identical to the NAL unit type RASL_NUT.

[178] Information regarding a subframe may be signaled in high-level syntax, such as a frame parameter set (PPS) and a sequence parameter set (SPS). In addition, information indicating whether or not to use a mixed NAL unit type may be signaled in a high-level syntax, such as a frame parameter set (PPS).

[179] FIG. 11 is a view illustrating an example of a frame parameter set (PPS) according to an embodiment of the present invention, and FIG. 12 is a view illustrating an example of a sequence parameter set (SPS) according to an embodiment of the present invention.

[180] First, referring to FIG. 11, the frame parameter set (PPS) may include pps_mixed_nalu_types_in_pic_flag as a syntax element indicating whether or not a mixed NAL unit type should be used.

[181] pps_mixed_nalu_types_in_pic_flag may indicate whether or not the current frame has a mixed NAL unit type. For example, pps_mixed_nalu_types_in_pic_flag having a first value (eg, 0) may indicate that the current frame does not have a mixed NAL unit type. In this case, the current frame may have an identical NAL unit type for all VCL NAL units, for example, a NAL unit type identical to the NAL unit type of the encoded slice NAL units. Conversely, pps_mixed_nalu_types_in_pic_flag having a second value (eg, 1) may indicate that the current frame is of a mixed NAL unit type.

[182] In an embodiment, the VCL NAL unit of the current frame may be constrained such that it does not have a NAL unit type identical to the GDR_NUT NAL unit type when the current frame has a mixed NAL unit type (eg, pps_mixed_nalu_types_in_pic_flag== 1).

[183] In an embodiment, when the current frame has a mixed NAL unit type (eg, pps_mixed_nalu_types_in_pic_flag==1), if one VCL NAL unit of the current frame has a NAL unit type (NAL unit type A) identical to the NAL unit type units IDR_W_RADL, IDR_N_LP or CRA_NUT, other VCL NAL units of the current frame may be constrained such that they have a NAL unit type identical to the NAL unit type IDR_W_RADL, IDR_N_LP, CRA_NUT or TRAIL_NUT.

[184] When the current frame has a mixed NAL unit type (for example, pps_mixed_nalu_types_in_pic_flag==1), each subframe in the current frame may have any of the VCL NAL unit types described above with reference to Table 1. For example, when the subframe in the current frame is an IDR subframe, the subframe may have a NAL unit type identical to the NAL unit type IDR_W_RADL or IDR_N_LP. Alternatively, when the subframe in the current frame is a trailing subframe, the subframe may have a NAL unit type identical to the TRAIL_NUT NAL unit type.

[185] pps_mixed_nalu_types_in_pic_flag having a second value (eg, 1) may indicate that frames referencing a frame parameter set (PPS) may include slices having different types of NAL units. Here, frames may be initiated from a subframe bitstream combining operation, in which the encoder provides matching of the bitstream structure and alignment between parameters of the original bitstreams. As an example of overlap, when reference frame list (RPL) syntax elements for a slice having a NAL unit type identical to the IDR_W_RADL or IDR_N_LP NAL unit type are not present in the slice header (for example, sps_idr_rpl_present_flag==0), and the current frame, containing a slice has a mixed NAL unit type (e.g. pps_mixed_nalu_types_in_pic_flag==1), the current frame may be restricted such that it does not include slices having a NAL unit type identical to the NAL unit type IDR_W_RADL or IDR_N_LP .

[186] Meanwhile, when there is a constraint such that a mixed NAL unit type does not apply to all frames in the Output Layer Set (OLS) (eg, gci_no_mixed_nalu_types_in_pic_constraint_flag==1), pps_mixed_nalu_types_in_pic_flag may have the first value (eg, 0).

[187] In addition, the picture parameter set (PPS) may include pps_no_pic_partition_flag as a syntax element indicating whether or not frame segmentation should be applied.

[188] pps_no_pic_partition_flag may indicate whether or not frame segmentation is applicable to the current frame. For example, pps_no_pic_partition_flag having a first value (eg 0) may indicate that the current frame cannot be partitioned. Conversely, pps_no_pic_partition_flag having a second value (eg, 1) may indicate that the current frame may be segmented into two or more tiles or slices. When the current frame has a mixed NAL unit type (for example, pps_mixed_nalu_types_in_pic_flag==1), the current frame can be constrained such that it has two or more tiles or slices (for example, pps_no_pic_partition_flag=1).

[189] In addition, the frame parameter set (PPS) may include pps_num_subpics_minus1 as a syntax element indicating the number of subpics.

[190] pps_num_subpics_minus1 may indicate a value obtained by subtracting 1 from the number of subpics included in the current frame. pps_num_subpics_minus1 may only be signaled when frame segmentation is applicable to the current frame (eg pps_no_pic_partition_flag==1). When pps_num_subpics_minus1 is not signaled, the value of pps_num_subpics_minus1 may be logically output as 0. Meanwhile, a syntax element indicating the number of subframes may be signaled in a high-level syntax other than a frame parameter set (PPS), such as in a parameter set sequences (SPS).

[191] In an embodiment, when the current frame includes only one subframe (for example, pps_num_subpics_minus1==0), the current frame may be limited such that it does not have a mixed NAL unit type (for example, pps_mixed_nalu_types_in_pic_flag==0) . Thus, when the current frame has a mixed NAL unit type (eg, pps_mixed_nalu_types_in_pic_flag==1), the current frame may be constrained such that it includes two or more subframes (eg, pps_num_subpics_minus1>0).

[192] Next, referring to FIG. 12, the sequence parameter set (SPS) may include sps_subpic_treated_as_pic_flag[i] as a syntactic element associated with the treatment of the subframe during encoding/decoding.

[193] sps_subpic_treated_as_pic_flag[i] may indicate whether or not each subframe in the current frame is treated as a single frame. For example, sps_subpic_treated_as_pic_flag[i] having a first value (eg, 0) may indicate that the i-th subframe in the current frame is not treated as a single frame. In contrast, sps_subpic_treated_as_pic_flag[i] having a second value (eg, 1) may indicate that the i-th subframe in the current frame is treated as a single frame during the encoding/decoding process except for in-loop filtering operations. When sps_subpic_treated_as_pic_flag[i] is not signaled, sps_subpic_treated_as_pic_flag[i] may be inferred to have a second value (eg, 1).

[194] In an embodiment, when the current frame includes two or more subpics (eg, pps_num_subpics_minus1>0), and at least one of the subpics is not treated as one frame (eg, sps_subpic_treated_as_pic_flag[i]==0 ), the current frame may be constrained such that it does not have a mixed NAL unit type (eg pps_mixed_nalu_types_in_pic_flag=0). Thus, when the current frame has a mixed NAL unit type (eg, pps_mixed_nalu_types_in_pic_flag==1), all subframes in the current frame can be constrained such that they are treated as a single frame (eg, sps_subpic_treated_as_pic_flag[i]=1).

[195] Hereinafter, this document describes in detail the type of NAL unit according to embodiments of the present invention for each frame type.

[196] (1) IRAP (Indoor Random Access Point) based frame

[197] An IRAP frame is a randomly available frame and may have an identical NAL unit type, such as IDR_W_RADL, IDR_N_LP, or CRA_NUT, as described above with reference to Table 1. An IRAP frame may not refer to a non-IRAP frame. frame for inter-prediction during the decoding process. An IRAP frame may include an IDR (Instantaneous Decoding Update) based frame and a CRA (Clean Random Access) based frame.

[198] The first frame in the bit stream in decoder order may be limited to an IRAP frame or a GDR (Gradual Decoding Update) frame. For a single layer bitstream, when a significant set of parameters to be referenced is available although frames preceding the IRAP frame in decoding order are not decoded at all, the IRAP frame and all non-RASL frames after the IRAP frame in decoding order can be decoded correctly.

[199] In an embodiment, the IRAP frame may not have a mixed NAL unit type. Thus, for an IRAP frame, pps_mixed_nalu_types_in_pic_flag may have the first value (eg, 0), and all slices in an IRAP frame may have the same NAL unit type in the range IDR_W_RADL to CRA_NUT. As a result, when the first slice to be decoded in a frame has a NAL unit type in the range of IDR_W_RADL to CRA_NUT, the frame can be determined to be an IRAP frame.

[200] (2) CRA (pure random access) based frame

[201] A CRA frame is one of the IRAP frames and may have a NAL unit type identical to the CRA_NUT NAL unit type, as described above with reference to Table 1. A CRA frame may not refer to a frame other than a CRA frame. frame for inter-prediction during the decoding process.

[202] The CRA frame may be the first frame in the bit stream in decoding order or the frame after the first frame. A CRA frame may be associated with RADL or RASL frames.

[203] When NoIncorrectPicOutputFlag has a second value (eg, 1) for a CRA frame, RASL frames associated with the CRA frame refer to frames that are not present in the bitstream and may therefore not be decoded and, as a result, may not be output by image decoding equipment. Here, the NoIncorrectPicOutputFlag may indicate whether or not frames preceding the recovery point-based frame in decoding order are output before the recovery point-based frame. For example, NoIncorrectPicOutputFlag having a first value (eg, 0) may indicate that frames preceding the recovery point-based frame in decoding order are output before the recovery point-based frame. In this case, the CRA frame may not be the first frame in the bitstream or the first frame after the NAL unit of the end of sequence (EOS) in the decoding order, which may mean that random access does not occur. In contrast, NoIncorrectPicOutputFlag having a second value (eg, 1) may indicate that frames preceding the recovery point frame in decoding order may not be output before the recovery point frame. In this case, the CRA frame may be the first frame in the bit stream or the first frame after the NAL unit of the end of sequence (EOS) in the decoding order, which may indicate that random access occurs. Meanwhile, NoIncorrectPicOutputFlag may be referred to as NoOutputBeforeRecoveryFlag in some embodiments.

[204] For all frame units (PUs) following the current frame in decoding order in CLVS (coded layer video sequence), a list of 0 reference frames (for example, RefPicList[0]) and a list of 1 reference frames (for example, RefPicList[1] ) for a single slice included in a CRA subframe belonging to frame units (PUs) may be constrained such that they do not include any frame preceding the frame including the CRA subframe in decoding order within the active recording . Here, a frame unit (PU) may mean a set of NAL units including a plurality of NAL units that are mutually associated according to a predetermined classification rule and are continuous in decoding order for one encoded frame.

[205] (3) IDR (Instantaneous Decoding Refresh) based frame

[206] An IDR frame is one of the IRAP frames and may have a NAL unit type identical to the IDR_W_RADL or IDR_N_LP NAL unit type, as described above with reference to Table 1. The IDR frame may not refer to a frame other than IDR frames for inter-prediction during decoding process.

[207] An IDR frame may be the first frame in the bitstream in decoding order, or may be a frame after the first frame. Each IDR frame may represent the first CVS (coded video sequence) frame in the decoding order.

[208] When an IDR frame has a NAL unit type identical to the IDR_W_RADL NAL unit type for each NAL unit, the IDR frame may have associated RADL frames. In contrast, when the IDR frame has a NAL unit type identical to the NAL unit type IDR_N_LP for each NAL unit, the IDR frame may have unassociated leading frames. Meanwhile, the IDR frame may not be associated with RASL frames.

[209] For all frame units (PUs) following the current frame in decoding order in CLVS (coded layer video sequence), a list of 0 reference frames (for example, RefPicList[0]) and a list of 1 reference frames (for example, RefPicList[1] ) for one slice included in an IDR subframe belonging to frame units (PUs) may be restricted such that they do not include any frame preceding the frame including the IDR subframe in decoding order within the active recording .

[210] (4) RADL (Random Access Decodable Advance) frame

[211] The RADL frame is one of the leading frames and may have a NAL unit type identical to the RADL_NUT NAL unit type, as described above with reference to Table 1.

[212] The RADL frame may not be used as a reference frame in the process of decoding a lagging frame having an identical associated IRAP frame. When field_seq_flag has the first value (eg, 0) for a RADL frame, the RADL frame may precede all non-leading frames that have an identical associated IRAP frame in decoding order. Here, field_seq_flag may indicate whether a CLVS (coded layer video sequence) transmits a frame indicating fields or a frame indicating movie frames. For example, field_seq_flag having a first value (eg, 0) may indicate that CLVS is transmitting a frame indicating movie frames. Conversely, a field_seq_flag having a second value (eg, 1) may indicate that CLVS is transmitting a frame indicating the fields.

[213] (5) RASL (random access skip forward) frame

[214] The RASL frame is one of the leading frames and may have a NAL unit type identical to the RASL_NUT NAL unit type, as described above with reference to Table 1.

[215] In an example, all RASL frames may be forward frames of an associated CRA frame. When NoIncorrectPicOutputFlag has a second value (eg, 1) for a CRA frame, RASL frames refer to frames that are not present in the bitstream and therefore may not be decoded and, as a result, may not be output by the image decoding equipment.

[216] The RASL frame may not be used as a reference frame in the decoding process of non-RASL frames. However, when a RADL frame belonging to a layer identical to that of a RASL frame and associated with an identical CRA frame is present, the RASL frame can be used as a co-located reference frame for inter-prediction of a RADL subframe included in the RADL frame .

[217] When field_seq_flag has the first value (eg, 0) for a RASL frame, the RASL frame may precede all non-leading frames of the CRA frame associated with the RASL frame in decoding order.

[218] (6) Lag frame

[219] A lagging frame is a non-IRAP frame after an associated IRAP frame or GDR frame in output order and may not be an STSA frame. In addition, the lagging frame may come after the associated IRAP frame in the decoding order. Thus, a lagging frame that comes after an associated IRAP frame in output order, but precedes an associated IRAP frame in decoding order, may not be resolved.

[220] (7) GDR (Gradual Decoding Update) based frame

[221] A GDR frame is a randomly available frame and may have a NAL unit type identical to the NAL unit type GDR_NUT, as described above with reference to Table 1.

[222] (8) STSA based frame

[223] An STSA frame is a randomly available frame and may have a NAL unit type identical to the STSA_NUT NAL unit type, as described above with reference to Table 1.

[224] The STSA frame may not reference frames that have a TemporalId identical to the TemporalId of the STSA frame for inter-prediction. Here, the TemporalId may be an identifier indicating a temporal layer, for example, a temporal sublayer in scalable video encoding. In an embodiment, STSA frames may be constrained such that they have a TemporalId greater than 0.

[225] Frames having a TemporalID identical to the TemporalID of an STSA frame and following the STSA frame in decoding order may not reference frames having a TemporalID identical to the TemporalID of the STSA frame and preceding the STSA frame in decoding order for inter-prediction . The STSA frame may activate up-switching from the immediately lower sublayer of the current sublayer to which the STSA frame belongs to the current sublayer.

[226] FIG. 13 is a view illustrating a decoding order and an output order for each frame type.

[227] A plurality of frames may be classified into an I frame, a P frame, or a B frame according to a prediction method. The I frame may refer to a frame to which only intra prediction is applicable, and may be decoded without reference to another frame. The I frame may be referred to as an inner frame and may include the above-described IRAP frame. A P frame may refer to a frame to which intra prediction and unidirectional inter prediction are applicable, and may be decoded by referring to another frame. A B frame may refer to a frame to which intra prediction and bidirectional/unidirectional inter prediction are applicable, and may be decoded using one or two other frames. The P frame and B frame may be referred to as a cross frame and may include the above-described RADL frame, a RASL frame, and a lagging frame.

[228] The cross frame may further be classified into a leading frame (LP) or a non-leading frame (NLP) according to the decoding order and the output order. A forward frame may refer to a frame that comes after an IRAP frame in decoding order and precedes an IRAP frame in output order, and may include the above-described RADL frame and a RASL frame. A non-leading frame may refer to a frame that comes after an IRAP frame in decoding order and output order, and may include the lagging frame described above.

[229] In FIG. 13, the name of each frame may indicate its frame type. For example, an I5 frame may be an I frame, and B0, B2, B3, B4, and B6 frames may be B frames, and P1 and P7 frames may be P frames. In addition, in FIG. 13, each arrow may indicate a reference direction between frames. For example, a B0 frame may be decoded by reference to a P1 frame.

[230] Referring to FIG. 13, the I5 frame may be an IRAP frame, such as a CRA frame. When random access for an I5 frame occurs, the I5 frame may be the first frame in the decoding order.

[231] B0 and P1 frames may precede the I5 frame in decoding order and construct a separate video sequence relative to the I5 frame. B2, B3, B4, B6 and P7 frames can come after the I5 frame in decoding order and construct one video sequence together with the I5 frame.

[232] B2, B3 and B4 frames come after the I5 frame in decoding order and precede the I5 frame in output order and can therefore be classified as a leading frame. The B2 frame may be decoded by reference to the I5 frame preceding the P1 frame in the decoding order. Accordingly, when random access occurs from an I5 frame, the B2 frame may not be decoded correctly by reference to a P1 frame that is not present in the bit stream. A frame type such as a B2 frame may be referred to as a RASL frame. In contrast, a B3 frame may be decoded by reference to a B3 frame preceding an I5 frame in the decoding order. Accordingly, when random access occurs from an I5 frame, the B3 frame can be correctly decoded by reference to the previously decoded I5 frame. In addition, B4 may be decoded by reference to an I5 frame and a B4 frame preceding the B3 frame in the decoding order. Accordingly, when random access occurs from an I5 frame, a B4 frame can be correctly decoded by referring to a previously decoded I5 frame and a B3 frame. The frame type, for example B3 and B4 frames, may be referred to as a RADL frame.

[233] Meanwhile, B6 and P7 frames come after the I5 frame in the decoding order and output order, and therefore can be classified into a non-leading frame. A B6 frame can be decoded by reference to an I5 frame and a P7 frame preceding it in the decoding order. Accordingly, when random access occurs from an I5 frame, the B6 frame can be correctly decoded by referring to the previously decoded I5 frame and P7 frame.

[234] In a single video sequence, the decoding process and the output process may be performed in different orders based on the frame type. For example, when one video sequence includes an IRAP frame, a leading frame, and a non-leading frame, the decoding process may be performed in the order of the IRAP frame, leading frame, and non-leading frame, and the output process may be performed in the order of leading frame, IRAP frame, and non-leading frame. .

[235] FIG. 14 is a flowchart illustrating a method for determining the NAL unit type of a current frame according to an embodiment of the present invention.

[236] Referring to FIG. 14, the image encoding equipment can determine whether or not the current frame includes two or more subframes (S1410). The segmentation information of the current frame may be signaled using one or more syntax elements in high-level syntax. For example, through the Frame Parameter Set (PPS) described above with reference to FIG. 7, no_pic_partition_flag indicating whether the current frame is segmented or not, and pps_num_subpics_minus1 indicating the number of subframes included in the current frame may be signaled. When the current frame includes two or more subframes, no_pic_partition_flag may have a first value (for example, 0), and pps_num_subpics_minus may have a value greater than 0.

[237] When the current frame does not include two or more subframes (None S1410), the image encoding equipment may determine that the current frame has one type of NAL unit (S1450). Thus, all slices in the current frame can have any of the types of NAL units described above with reference to Table 1.

[238] In contrast, when the current frame includes two or more subframes (Yes S1410), the image encoding equipment can determine whether or not each subframe in the current frame is treated as one frame (S1420). Information indicating whether or not a subframe is treated as a single frame may be signaled using a predefined syntax element in a high-level syntax, for example, through a sequence parameter set (SPS) described above with reference to FIG. 12, sps_subpic_treated_as_pic_flag[i], indicating whether or not a subframe is treated as a single frame, may be signaled. When each subframe in the current frame is treated as one frame, sps_subpic_treated_as_pic_flag[i] may have a second value (for example, 1).

[239] When each subframe in the current frame is not treated as one frame (No) S1420, the image encoding equipment can determine that the current frame has one type of NAL unit (S1450).

[240] In contrast, when each subframe in the current frame is treated as one frame (Yes S1420), the image encoding equipment can determine whether or not at least some of the plurality of slices in the current frame have different types of NAL units. (S1430).

[241] When at least some of the plurality of slices in the current frame have different NAL unit types (Yes in S1430), the image encoding equipment can determine that the current frame has a mixed NAL unit type (S1440).

[242] In contrast, when all of the plurality of slices in the current frame have the same NAL unit type (“No” in S1430), the image encoding equipment can determine that the current frame has the same NAL unit type (S1450).

[243] When the current frame has a mixed NAL unit type, the image encoding equipment may generate a sub-bit stream for each sub-frame included in the current frame. In this case, the plurality of sub-bit streams generated for the current frame may constitute one bit stream.

[244] Information indicating whether or not the current frame has a mixed NAL unit type may be signaled using a predefined syntax element in the high-level syntax. For example, through the above-described Frame Parameter Set (PPS) described above with reference to FIG. 11, pps_mixed_nalu_types_in_pic_flag indicating whether or not the current frame has a mixed NAL unit type may be signaled. In this case, the picture decoding equipment can determine whether or not the current frame has a mixed NAL unit type based on the pps_mixed_nalu_types_in_pic_flag. For example, when pps_mixed_nalu_types_in_pic_flag has a first value (eg, 0), the picture decoding equipment may determine that the current frame is of one NAL unit type. In contrast, when pps_mixed_nalu_types_in_pic_flag has a second value (eg, 1), the picture decoding equipment may determine that the current frame is of a mixed NAL unit type.

[245] Meanwhile, all slices in each subframe included in the current frame may have the same NAL unit type. Thus, each subframe can have only one type of NAL unit. In addition, when the current frame has a mixed NAL unit type, at least some subframes included in the current frame may have different NAL unit types. For example, a first subframe included in the current frame may have a first NAL unit type, and a second subframe included in the current frame may have a second NAL unit type different from the first NAL unit type.

[246] The type of each subframe included in the current frame may be determined based on the NAL unit type of each subframe. For example, as described above with reference to Table 1, a subframe type having a NAL unit type such as TRAIL_NUT may be determined as a trailing subframe. In addition, the type of a subframe having a NAL unit type such as STSA_NUT may be defined as an STSA subframe, and the type of a subframe having a NAL unit type such as GDR_NUT may be defined as a GDR subframe. In addition, the type of a subframe having a NAL unit type such as RADL_NUT may be defined as a RADL subframe, and the type of a subframe having a NAL unit type such as RASL_NUT may be defined as a RASL subframe. In addition, the type of a subframe having a NAL unit type such as IDR_W_RADL or IDR_N_LP may be defined as an IDR subframe, and the type of a subframe having a NAL unit type such as CRA_NUT may be defined as a CRA subframe.

[247] According to embodiments of the present invention, when the current frame includes two or more subframes and each subframe is treated as one frame, the current frame may have a mixed NAL unit type. Therefore, in various applications or use cases, the efficiency of image encoding/decoding can be further improved.

[248] Hereinafter, this document will describe in detail an image encoding/decoding method according to an embodiment of the present invention with reference to FIGS. 15 and fig. 16.

[249] FIG. 15 is a flowchart illustrating an image encoding method according to an embodiment of the present invention.

[250] The image encoding method of FIG. 15 may be implemented by the image encoding equipment of FIG. 2. For example, step S1510 may be performed by the image segmentation unit 110, and steps S1520 and S1530 may be performed by the entropy encoder 190.

[251] Referring to FIG. 15, the image encoding equipment may segment the current frame into two or more subframes (S1510).

[252] In an embodiment, if the current frame is of a mixed NAL unit type, the current frame may be limited such that it includes two or more subframes. Thus, based on at least some of the plurality of slices included in the current frame having different types of NAL units, the current frame may be segmented into two or more subframes.

[253] The segmentation information of the current frame may be signaled using one or more syntax elements in high-level syntax. For example, through the Frame Parameter Set (PPS) described above with reference to FIG. 7, flag information no_pic_partition_flag indicating whether or not the current frame is segmented, and flag information pps_num_subpics_minus1 indicating the number of subframes included in the current frame may be encoded/transmitted in overhead signals. In this case, when the current frame is segmented into two or more subframes, no_pic_partition_flag may have the first value (for example, 0), and pps_num_subpics_minus may have a value greater than 0.

[254] In an embodiment, when the current frame has a mixed NAL unit type, each subframe included in the current frame may be treated as one frame. For example, a current frame having a mixed NAL unit type may include a first subframe and a second subframe, and each of the first subframe and the second subframe may be independently encoded/decoded except for an in-loop filtering operation.

[255] Information regarding whether or not a subframe included in the current frame is treated as one frame may be encoded/signaled using a predefined syntax element in the high-level syntax. For example, sps_subpic_treated_as_pic_flag[i] flag information indicating whether or not the i-th subframe included in the current frame is treated as one frame may be encoded/signalled via the Sequence Parameter Set (SPS) described above with reference to fig. 12. In this case, when the i-th subframe included in the current frame is not treated as one frame, sps_subpic_treated_as_pic_flag[i] may have the first value (for example, 0). On the other hand, when the i-th subframe included in the current frame is treated as one frame, sps_subpic_treated_as_pic_flag[i] may have a second value (eg, 1).

[256] The image encoding equipment may determine the NAL unit type of each of the plurality of slices included in the current frame (S1520).

[257] In an embodiment, all slices included in each subframe in the current frame may be of the same NAL unit type. For example, all slices included in the first subframe of the current frame may have a first NAL unit type, and all slices included in the second subframe of the current frame may have a second NAL unit type. In this case, when the current frame has a mixed NAL unit type, the second NAL unit type may be different from the first NAL unit type. Alternatively, when the current frame has one NAL unit type, the second NAL unit type may be identical to the first NAL unit type.

[258] In an embodiment, whether or not at least some of the plurality of slices included in the current frame have different NAL unit types (i.e., whether or not the current frame has a mixed NAL unit type) , may be encoded/signalled using a syntax element. For example, pps_mixed_nalu_types_in_pic_flag flag information indicating whether or not the current frame has a mixed NAL unit type may be encoded/signalled via the Frame Parameter Set (PPS) described above with reference to FIG. 11. In this case, when the current frame has one NAL unit type, pps_mixed_nalu_types_in_pic_flag may have the first value (for example, 0). In contrast, when the current frame has a mixed NAL unit type, pps_mixed_nalu_types_in_pic_flag may have a second value (for example, 1).

[259] In an embodiment, the value of pps_mixed_nalu_types_in_pic_flag may be determined based on whether or not the mixed NAL unit type is applicable to all frames in the Output Layer Set (OLS). For example, when the mixed NAL unit type is not applicable to all frames in the Output Layer Set (OLS), pps_mixed_nalu_types_in_pic_flag may be constrained such that it has the first value (eg, 0). In contrast, when the mixed NAL unit type is applicable to all frames in the Output Layer Set (OLS), pps_mixed_nalu_types_in_pic_flag may have a first value (eg, 0) or a second value (eg, 1).

[260] Meanwhile, whether or not a mixed NAL unit type is applicable to all frames in the Output Layer Set (OLS) may be encoded/signaled using a predefined syntax element in the high-level syntax. For example, flag information gci_no_mixed_nalu_types_in_pic_constraint_flag indicating whether a mixed NAL unit type is constrained or not may be encoded/signaled through general_constraints_info including general constraint information. In this case, when the mixed NAL unit type is applicable to all frames in the output layer set (OLS), gci_no_mixed_nalu_types_in_pic_constraint_flag may have the first value (eg, 0). In contrast, when the mixed NAL unit type is not applicable to all frames in the output layer set (OLS), gci_no_mixed_nalu_types_in_pic_constraint_flag may have a second value (eg, 1).

[261] In an embodiment, the value of pps_mixed_nalu_types_in_pic_flag may be determined based on whether or not the current frame includes only one subframe. For example, when the current frame includes only one subframe, pps_mixed_nalu_types_in_pic_flag may be constrained such that it has the first value (eg, 0). On the other hand, when the current frame includes two or more subframes, pps_mixed_nalu_types_in_pic_flag may have a first value (eg, 0) or a second value (eg, 1).

[262] In an embodiment, the value of pps_mixed_nalu_types_in_pic_flag may be determined based on whether or not each of all subframes included in the current frame is treated as a single frame. For example, when at least one of the subframes included in the current frame is not treated as a single frame, pps_mixed_nalu_types_in_pic_flag may be constrained to have a first value (eg, 0). In contrast, when each of all subframes included in the current frame is treated as one frame, pps_mixed_nalu_types_in_pic_flag may have a first value (eg, 0) or a second value (eg, 1).

[263] The image encoding equipment may encode subframes in the current frame based on the NAL unit type determined in S1520 (S1530). At this time, the encoding process for each slice may be performed in coding units (CU) based on a predetermined prediction mode as described above. Meanwhile, each subframe can be independently encoded to construct different bitstreams (subframes). For example, a first sub-bitstream including encoding information of the first subframe may be constructed, and a second sub-bitstream including encoding information of the second subframe may be constructed.

[264] FIG. 16 is a flowchart illustrating an image decoding method according to an embodiment of the present invention.

[265] The image decoding method of FIG. 16 may be implemented by the image decoding equipment of FIG. 3. For example, steps S1610 and S1620 may be performed by the entropy decoder 210, and step S1630 may be performed by the dequantizer 220 to the intra prediction unit 265.

[266] Referring to FIG. 16, the image decoding equipment can obtain, from the bitstream, VCL NAL unit type information of the current frame and first flag information indicating whether or not a subframe included in the current frame is treated as one frame (S1610).

[267] The NAL unit type information of the current frame may include a NAL unit type value for a VCL NAL unit including encoded image data (eg, slice data) of the current frame. The NAL unit type value may be obtained by parsing the nal_unit_type syntax element included in the NAL unit header for the VCL NAL unit.

[268] The first flag information may be obtained by parsing a high-level syntax such as sps_subpic_treated_as_pic_flag[i] included in the Sequence Parameter Set (SPS). In this case, when the i-th subframe included in the current frame is not treated as one frame, sps_subpic_treated_as_pic_flag[i] may have the first value (for example, 0). On the other hand, when the i-th subframe included in the current frame is treated as one frame, sps_subpic_treated_as_pic_flag[i] may have a second value (eg, 1).

[269] The image decoding equipment can determine the NAL unit type of each of the plurality of slices in the current frame based on the VCL NAL unit type information of the current frame (S1620).

[270] In an embodiment, when at least some of the plurality of slices in the current frame have different NAL unit types (i.e., when the current frame has a mixed NAL unit type), the current frame includes two or more subframes

[271] In an embodiment, when at least some of the plurality of slices in the current frame have different types of NAL units, each of all subframes included in the current frame may be treated as one frame. In this case, for all subframes included in the current frame, the first flag information (eg, sps_subpic_treated_as_pic_flag[i]) may have a second value (eg, 1).

[272] In an embodiment, all slices included in each subframe in the current frame may have the same NAL unit type. For example, all slices included in the first subframe of the current frame may have a first NAL unit type, and all slices included in the second subframe of the current frame may have a second NAL unit type. In this case, when the current frame has a mixed NAL unit type, the second NAL unit type may be different from the first NAL unit type. In contrast, when the current frame has one NAL unit type, the second NAL unit type may be identical to the first NAL unit type.

[273] In an embodiment, whether or not at least some of the plurality of slices in the current frame have different NAL unit types can be determined based on high-level syntax, for example, second flag information (eg, pps_mixed_nalu_types_in_pic_flag) obtained from Frame Parameter Set (PPS). For example, when the second flag information has the first value (eg, 0), all slices in the current frame may have the same NAL unit type. In contrast, when the second flag information has a second value (eg, 1), at least some of the plurality of slices in the current frame may have different types of NAL units.

[274] In an embodiment, the second flag information (eg, pps_mixed_nalu_types_in_pic_flag) may have a predetermined value based on whether or not the mixed NAL unit type is applicable to all frames in the Output Layer Set (OLS). For example, when the mixed NAL unit type is not applicable to all frames in the output layer set (OLS), the second flag information may have a first value (eg, 0). In this case, the image decoding equipment may logically output the value of the second flag information as constituting the first value (eg, 0) without separately parsing the second flag information. In contrast, when the mixed NAL unit type is applicable to all frames in the output layer set (OLS), the second flag information may have a first value (eg, 0) or a second value (eg, 1).

[275] In an embodiment, the second flag information (eg, pps_mixed_nalu_types_in_pic_flag) may have a predetermined value based on whether or not the current frame includes only one subframe. For example, when the current frame includes only one subframe, the second flag information may only have the first value (eg, 0). In this case, the image decoding equipment may logically output the value of the second flag information as constituting the first value (eg, 0) without separately parsing the second flag information. In contrast, when the current frame includes two or more subframes, the second flag information may have a first value (eg, 0) or a second value (eg, 1).

[276] In an embodiment, the second flag information (eg, pps_mixed_nalu_types_in_pic_flag) may have a predetermined value based on whether or not each of all subframes included in the current frame is treated as a single frame. For example, when at least one of the subframes included in the current frame is not treated as a single frame, pps_mixed_nalu_types_in_pic_flag may have only the first value (eg, 0). In this case, the image decoding equipment may logically output the value of the second flag information as constituting the first value (eg, 0) without separately parsing the second flag information. In contrast, when each of all subframes included in the current frame is treated as one frame, pps_mixed_nalu_types_in_pic_flag may have a first value (eg, 0) or a second value (eg, 1).

[277] Meanwhile, the type of each subframe included in the current frame may be determined based on the NAL unit type of each subframe. For example, as described above with reference to Table 1, a subframe type having a NAL unit type such as TRAIL_NUT may be determined as a trailing subframe. In addition, the type of a subframe having a NAL unit type such as STSA_NUT may be defined as an STSA subframe, and the type of a subframe having a NAL unit type such as GDR_NUT may be defined as a GDR subframe. In addition, the type of a subframe having a NAL unit type such as RADL_NUT may be defined as a RADL subframe, and the type of a subframe having a NAL unit type such as RASL_NUT may be defined as a RASL subframe. In addition, the type of a subframe having a NAL unit type such as IDR_W_RADL or IDR_N_LP may be defined as an IDR subframe, and the type of a subframe having a NAL unit type such as CRA_NUT may be defined as a CRA subframe.

[278] The image decoding equipment can decode a plurality of slices in the current frame based on the first flag information obtained in step S1610 and the NAL unit type determined in S1620 (S1630). At this time, the decoding process of each slice may be performed in coding units (CU) based on a predetermined prediction mode as described above.

[279] According to an embodiment of the present invention, the current frame may have two or more types of NAL units based on the subframe structure.

[280] The name of a syntax element described in the present disclosure may include information regarding the position at which the corresponding syntax element is signaled. For example, a syntax element beginning with "sps_" may indicate that the corresponding syntax element is signaled in the Sequence Parameter Set (SPS). In addition, a syntax element starting with "pps_", "ph_", "sh_"6 may indicate that the corresponding syntax element is signaled in the frame parameter set (PPS), frame header and slice header, respectively.

[281] Although the exemplary methods of the present invention described above are presented as a sequence of operations for clarity of description, it is not intended to limit the order in which the steps are performed, and the steps may be performed simultaneously or in a different order as necessary. To implement the method of the present disclosure, the described steps may further include other steps, may include the remaining steps except some steps, or may include other additional steps except some steps.

[282] In the present disclosure, image encoding equipment or image decoding equipment that performs a predetermined operation (step) may perform an operation (step) of confirming a condition or situation for executing the corresponding operation (step). For example, if it is described that a predetermined operation is performed when a predetermined condition is satisfied, the image encoding equipment or the image decoding equipment may perform the predetermined operation after determining whether or not the predetermined condition is satisfied.

[283] The various embodiments of the present invention are not intended to be a list of all possible combinations and are intended to describe characteristic aspects of the present invention, and the matters described in the various embodiments may be applied independently or in combination of two or more of them.

[284] Various embodiments of the present invention may be implemented in hardware, firmware, software, or a combination of the above. If the present invention is implemented in hardware, the present disclosure may be implemented using application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs). FPGA), general processors, controllers, microcontrollers, microprocessors, etc.

[285] In addition, the image decoding equipment and image encoding equipment to which embodiments of the present invention are applied may be included in a multimedia broadcast transceiver device, mobile communication terminal, home theater system video device, digital cinema system video device, surveillance camera, device video chat device, real-time communication device such as video calling, mobile streaming device, storage media, video recording camera, video on demand (VoD) service device, OTT video (OTT) device over networks), Internet streaming service device, 3D video device, video telephony system video device, medical video device, etc. and can be used to process video or data signals. For example, OTT video devices may include a gaming console, Blu-ray player, Internet-enabled television, home theater system, smartphone, tablet PC, digital video recorder (DVR), etc.

[286] FIG. 17 is a view illustrating a content streaming system to which an embodiment of the present invention is applicable.

[287] Referring to FIG. 17, the content streaming system to which the embodiment(s) of this document applies may mainly include an encoding server, a streaming server, a web server, a media storage, a user device, and a media input device.

[288] The encoding server compresses content input from media input devices such as a smartphone, camera, video recorder, etc. into digital data to form a bitstream, and transmits the bitstream to the streaming server. As another example, when media input devices such as smartphones, cameras, video recorders, etc. directly generate a bitstream, the encoding server may be omitted.

[289] The bitstream may be generated by an image encoding method or image encoding equipment to which an embodiment of the present invention is applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[290] The streaming server transmits media data to the user device based on the user's request through the web server, and the web server serves as a medium for informing the user regarding the service. When a user requests a desired service from a web server, the web server may deliver it to the streaming server, and the streaming server may stream the media data to the user. In this case, the content streaming system may include a separate control server. In this case, the control server serves to manage the command/response between devices in the content streaming system.

[291] The streaming server may receive content from a media storage and/or encoding server. For example, when content is received from an encoding server, the content may be received in real time. In this case, in order to provide a smooth streaming service, the streaming server may maintain the bit stream for a predetermined time.

[292] Examples of a user device may include a mobile phone, a smartphone, a laptop computer, a digital broadcast terminal, a personal digital assistant (PDA), a portable media player (PMP), a navigation device, a tablet PC, tablet PCs, ultrabooks, wearable devices (such as smart watches, smart glasses, head-mounted displays), digital TVs, desktop computers, digital signage system, etc.

[293] Each server in a content streaming system may operate as a distributed server, in which case data received from each server may be distributed.

[294] The scope of the invention includes software or computer executable instructions (e.g., operating system, application, firmware, program, etc.) for enabling operations according to the methods of various embodiments in hardware or a computer, non-transitory computer-readable medium, having such software or instructions stored and executed in hardware or a computer.

Industrial applicability

[295] Embodiments of the present invention can be used to encode or decode an image.

Claims (36)

1. Decoding equipment for decoding images, wherein the decoding equipment comprises: Memory device; at least one processor, wherein the at least one processor is configured to: obtain, from the bitstream, unit type information of the network abstraction layer (NAL) of the video coding layer (VCL) of the current frame and first flag information indicating whether or not a subframe included in the current frame is treated as one frame; determine the NAL unit type of each of the plurality of slices included in the current frame based on the received VCL NAL unit type information; decode multiple slices based on a specific NAL unit type and first flag information, wherein the current frame contains two or more subframes based on at least some of the plurality of slices having different types of NAL units, and wherein the first flag information has a predetermined value indicating that each of the two or more subframes is treated as one frame based on at least some of a plurality of slices having different types of NAL units. 2. The decoding equipment of claim 1, wherein all slices in each subframe included in the current frame are of the same NAL unit type. 3. The decoding equipment of claim 1, wherein whether or not at least some of the plurality of slices have different types of NAL units is determined based on second flag information obtained from a frame parameter set (PPS). 4. The decoding equipment of claim 3, wherein the second flag information has a predetermined value indicating that all of the plurality of slices are of the same NAL unit type based on the non-applicability of a mixed NAL unit type to all frames in the set of output layers ( OLS). 5. The decoding equipment of claim 3, wherein the second flag information has a predetermined value indicating that all of the plurality of slices have the same NAL unit type based on the current frame including only one subframe. 6. The decoding equipment of claim 3, wherein the second flag information has a predetermined value indicating that all of the plurality of slices are of the same NAL unit type based on at least one of the subframes included in the current frame, not interpreted as one frame. 7. The decoding equipment of claim 1, wherein the subframe type of each subframe included in the current frame is determined based on the NAL unit type of each subframe. 8. Encoding equipment for encoding images, wherein the encoding equipment comprises: Memory device; at least one processor, wherein the at least one processor is configured to: segment the current frame into one or more subframes; determine a network abstraction layer (NAL) unit type of each of the plurality of slices included in the one or more subframes; encode multiple slices based on a specific NAL unit type, wherein the current frame is segmented into two or more subframes based on at least some of a plurality of slices having different types of NAL units, wherein each of the two or more subframes is treated as one frame based on at least some of the plurality of slices having different types of NAL units. 9. The encoding equipment of claim 8, wherein all slices in each subframe included in the current frame are of the same NAL unit type. 10. The encoding equipment of claim 8, wherein whether or not at least some of the plurality of slices have different types of NAL units is encoded based on predetermined flag information included in the frame parameter set (PPS). 11. The encoding equipment of claim 10, wherein the flag information has a predetermined value indicating that all of the plurality of slices are of the same NAL unit type based on the non-applicability of a mixed NAL unit type to all frames in the output layer set (OLS). ). 12. The encoding equipment of claim 10, wherein the flag information has a predetermined value indicating that all of the plurality of slices have the same NAL unit type based on the current frame including only one subframe. 13. The encoding equipment of claim 10, wherein the flag information has a predetermined value indicating that all of the plurality of slices are of the same NAL unit type based on at least one of the subframes included in the current frame, not treated as one frame. 14. Equipment for transmitting image data, the equipment comprising: at least one processor configured to receive a bit stream generated by an encoding method performed by the encoding equipment for encoding images; a transmitter configured to transmit a bit stream, in this case, the encoding method contains stages in which: segmenting the current frame into one or more subframes; determining a network abstraction layer (NAL) unit type of each of the plurality of slices included in the one or more subframes; encode multiple slices based on a specific NAL unit type, wherein the current frame is segmented into two or more subframes based on at least some of a plurality of slices having different types of NAL units, wherein each of the two or more subframes is treated as one frame based on at least some of the plurality of slices having different types of NAL units.

Images