KR20230055488A - A method and an apparatus for processing high resolution videos - Google Patents

Abstract

According to an embodiment of the present invention, provided is an image decoding method, which includes the steps of: receiving a bitstream including image information; decoding subpicture signaling information from the bitstream; identifying resolution adjustment information for each subpicture from the subpicture signaling information; decoding one or more subpictures using the resolution adjustment information for each subpicture; and reconstructing a recovered image using the decoded subpicture.

Description

High-resolution image processing method and apparatus thereof

The present invention relates to an image processing method and an apparatus therefor. More specifically, the present invention relates to a method and apparatus for processing a high-resolution image.

As digital image processing and computer graphics technology have recently developed, research on virtual reality (VR) technology that reproduces the real world and allows a realistic experience is being actively conducted.

In particular, recent VR systems such as HMD (Head Mounted Display) can not only provide a 3D stereoscopic image to both eyes of the user, but also track the viewpoint in all directions, providing realistic virtual reality that can be viewed 360 degrees. It is receiving a lot of attention in that it can provide reality (VR) video content.

However, since 360 VR content is composed of simultaneous omnidirectional multi-view image information in which temporal and binocular images are spatially and compositely synchronized, two large images synchronized with respect to the binocular space of all viewpoints must be used in the production and transmission of images. It is encoded, compressed, and transmitted. This increases the complexity and bandwidth burden, and in particular, in the decoding apparatus, decoding is performed even for an area that is not actually viewed outside the user's point of view, so that unnecessary processes are wasted.

Accordingly, there is a need for an encoding method that reduces the transmission data amount and complexity of an image and is efficient in terms of bandwidth and battery consumption of a decoding device.

In order to solve this problem, recently, a technique of setting a location where a user's gaze stays as a viewport and processing an adaptive resolution based thereon has emerged.

H.265/HEVC technology, one of the international standards for video compression, mainly compresses and restores 360 video using a tile division technique. It is possible to reconstruct with a high bitrate according to , and to reduce the total amount of data by encoding some tiles with a low bitrate.

However, although the tile division method can support a variable bit rate, since actual images are configured by mapping images obtained through each camera in various ways into one large image, it is not efficient to compress an image resolution of 8K or higher. There is a problem.

The present invention is to solve the above problems, to provide a high-resolution video processing method and apparatus capable of efficiently reducing the amount of transmission data, bandwidth and processing operations of an image through variable compression of the image resolution. It has a purpose.

An image decoding method according to an embodiment of the present invention for solving the above problems includes receiving a bitstream including image information; decoding subpicture signaling information from the bitstream; identifying resolution adjustment information for each subpicture from the subpicture signaling information; decoding one or more subpictures using the resolution adjustment information for each subpicture; and reconstructing a restored image using the decoded subpicture.

In addition, an image encoding method according to an embodiment of the present invention for solving the above problems includes dividing an original image into subpictures; determining a coding resolution for each divided sub-picture; encoding an original image according to the encoding resolution for each sub-picture; generating the subpicture signaling information including resolution adjustment information for each subpicture; and constructing a bitstream including an encoded image and the subpicture signaling information.

According to an embodiment of the present invention, a bitstream may be configured by dividing an original video into subpictures and generating subpicture signaling information including resolution adjustment information for each subpicture according to a coding resolution for each divided subpicture. and can process signaling and decoding using it.

Accordingly, the present invention can provide a high-resolution image processing method and apparatus capable of efficiently reducing the amount of transmission data, bandwidth, and processing operations of an image through variable compression of the image resolution.

1 is a block diagram showing the configuration of an image encoding apparatus according to an embodiment of the present invention.
2 is a block diagram for explaining an embodiment of a method of performing inter prediction in an image encoding apparatus.
3 is a block diagram showing the configuration of a video decoding apparatus according to an embodiment of the present invention.
4 is a block diagram for explaining an embodiment of a method of performing inter prediction in a video decoding apparatus.
5 is a flowchart illustrating an operation of an image encoding apparatus according to an embodiment of the present invention.
6 is a flowchart illustrating an operation of a video decoding apparatus according to an embodiment of the present invention.
7 is a comparison diagram between an original image and a case where the resolution of each subpicture is adjusted according to an embodiment of the present invention.
8 is an exemplary diagram illustrating a relationship between resolution adjustment of each sub-picture and GOP units according to an embodiment of the present invention.
9 is an exemplary diagram illustrating a relationship between resolution adjustment for each subpicture and a NAL packet structure according to an embodiment of the present invention.
10 is a diagram for explaining a reduction in the number of tiles according to resolution adjustment of each subpicture according to an embodiment of the present invention.

EMBODIMENT OF THE INVENTION Hereinafter, embodiment of this invention is described concretely with reference to drawings. In describing the embodiments of the present specification, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description thereof will be omitted.

It is understood that when a component is referred to as being "connected" or "connected" to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. In addition, the description of "including" a specific configuration in the present invention does not exclude configurations other than the corresponding configuration, and means that additional configurations may be included in the practice of the present invention or the scope of the technical spirit of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

In addition, components appearing in the embodiments of the present invention are shown independently to represent different characteristic functions, and do not mean that each component is made of separate hardware or a single software component unit. That is, each component is listed and included as each component for convenience of explanation, and at least two components of each component can be combined to form a single component, or one component can be divided into a plurality of components to perform a function, and each of these components can be divided into a plurality of components. Integrated embodiments and separated embodiments of components are also included in the scope of the present invention as long as they do not depart from the essence of the present invention.

In addition, some of the components may be optional components for improving performance rather than essential components that perform essential functions in the present invention. The present invention can be implemented by including only components essential to implement the essence of the present invention, excluding components used for performance improvement, and a structure including only essential components excluding optional components used for performance improvement. Also included in the scope of the present invention.

1 is a block diagram showing the configuration of an image encoding apparatus according to an embodiment of the present invention. The image encoding apparatus 10 includes a picture divider 110, a transform unit 120, a quantization unit 130, a scanning unit 131, entropy encoding unit 140, intra prediction unit 150, inter prediction unit 160, inverse quantization unit 135, inverse transformation unit 125, post-processing unit 170, picture storage unit 180 ), a subtraction unit 190 and an addition unit 195 are included.

Referring to FIG. 1 , the picture divider 110 analyzes an input video signal, divides a picture into coding units, determines a prediction mode, and determines the size of the prediction unit for each coding unit.

Also, the picture division unit 110 sends a prediction unit to be encoded to the intra prediction unit 150 or the inter prediction unit 160 according to a prediction mode (or prediction method). Also, the picture division unit 110 sends a prediction unit to be encoded to the subtraction unit 190.

Here, a picture of an image is composed of a plurality of tiles or slices, and the tiles or slices may be divided into a plurality of Coding Tree Units (CTUs), which are basic units for dividing a picture.

Also, a plurality of tiles or slices according to an embodiment of the present invention may constitute one or more tile or slice groups, and such groups may constitute subpictures dividing a picture into rectangular regions.

Also, the coding tree unit may be divided into one or more coding units (CUs), which are basic units in which inter prediction or intra prediction is performed.

A coding unit (CU) may be divided into one or more prediction units (PUs), which are basic units in which prediction is performed.

In this case, the encoding apparatus 10 determines either inter prediction or intra prediction as a prediction method for each of the divided coding units (CUs), but predicts a prediction block differently for each prediction unit (PU). can create

Meanwhile, a coding unit (CU) may be divided into one or two or more transform units (TUs), which are basic units in which transformation of a residual block is performed.

In this case, the picture segmentation unit 110 may transfer image data to the subtraction unit 190 in units of blocks divided as described above (eg, a prediction unit (PU) or a transformation unit (TU)).

For example, a coding tree unit (CTU) having a maximum size of 256x256 pixels may be divided into four coding units (CUs) having a square shape by dividing into a quad tree structure.

Each of the four coding units (CUs) having the shape of a square may be subdivided into a quad tree structure, and the depth (Depth) of the coding units (CUs) split into the quad tree structure as described above is any one of 0 to 3. It can have a single integer value.

A coding unit (CU) may be divided into one or two or more prediction units (PUs) according to a prediction mode.

In addition, the coding unit (CU) may be divided into a quad tree or binary tree structure and divided into transform units (TUs) having a square or rectangular shape.

In addition, for more accurate and efficient encoding and decoding, the encoding apparatus 10 according to an embodiment of the present invention divides a quad tree and a binary (binary) tree into an edge region of a coding unit in which a length in a specific direction is long. Coding units may be divided into a ternary tree or triple tree structure that can easily divide .

Meanwhile, each of the transform units (TUs) having a square or rectangular shape may be re-divided into a quad tree or binary tree structure, and the depth of the transform unit (TU) divided into the quad tree or binary tree structure is It can have an integer value from 0 to 3.

Here, when the coding unit (CU) is in the inter prediction mode, the prediction unit (PU) and the transform unit (TU) divided from the corresponding coding unit (CU) may have a partition structure independent of each other.

When a coding unit (CU) is in intra prediction mode, a transform unit (TU) divided from the corresponding coding unit (CU) cannot be larger than the size of the prediction unit (PU).

In addition, the transform unit (TU) divided as described above may have a maximum size of 64x64 pixels.

The transform unit 120 transforms a residual block, which is a residual signal between an input original block of the prediction unit (PU) and a prediction block generated by the intra predictor 150 or the inter predictor 160, and the transform is It may be performed using a unit (TU) as a basic unit.

In the transformation process, different transformation matrices may be determined according to the prediction mode (intra or inter), and since the residual signal of intra prediction has a direction according to the intra prediction mode, the transformation matrix may be adaptively determined according to the intra prediction mode. there is.

A transformation unit may be transformed by two (horizontal and vertical) one-dimensional transformation matrices, and for example, in the case of inter prediction, one or two predetermined transformation matrices may be determined.

On the other hand, in the case of intra prediction, if the intra prediction mode is horizontal, since the probability that the residual block has a vertical direction increases, a DCT-based integer matrix is applied in the vertical direction, and a DST-based or DST-based integer matrix is applied in the horizontal direction. Apply KLT-based integer matrix. When the intra prediction mode is vertical, a DST-based or KLT-based integer matrix may be applied in the vertical direction, and a DCT-based integer matrix may be applied in the horizontal direction.

In addition, in the case of the DC mode, a DCT-based integer matrix can be applied to both directions.

And, in the case of intra prediction, a transform matrix may be adaptively determined based on the size of a transform unit (TU).

The quantization unit 130 determines a quantization step size for quantizing coefficients of the residual block transformed by the transformation matrix, and the quantization step size may be determined for each quantization unit having a predetermined size or larger.

The size of the quantization unit may be 8x8 or 16x16, and the quantization unit 130 quantizes the coefficients of the transform block using a quantization matrix determined according to a quantization step size and a prediction mode.

Also, the quantization unit 130 may use a quantization step size of a quantization unit adjacent to the current quantization unit as a quantization step size predictor of the current quantization unit.

The quantization unit 130 searches the left quantization unit, the upper quantization unit, and the upper left quantization unit of the current quantization unit in order, and generates a quantization step size predictor of the current quantization unit using one or two effective quantization step sizes. there is.

For example, the quantization unit 130 determines the first effective quantization step size retrieved in the above order as the quantization step size predictor, or determines the average value of two valid quantization step sizes retrieved in the above order as the quantization step size predictor, or If only one quantization step size is valid, this can be determined as a quantization step size predictor.

When the quantization step size predictor is determined, the quantization unit 130 transmits a difference value between the quantization step size predictor of the current quantization unit and the quantization step size predictor to the entropy encoding unit 140 .

Meanwhile, all of the left coding unit, the above coding unit, and the top left coding unit of the current coding unit do not exist. Alternatively, a coding unit previously present in the coding order within the largest coding unit may exist.

Therefore, in quantization units adjacent to the current coding unit and the maximum coding unit, a quantization step size of a quantization unit immediately preceding in coding order may be a candidate.

In this case, priorities are set in the order of 1) the left quantization unit of the current coding unit, 2) the upper quantization unit of the current coding unit, 3) the upper left quantization unit of the current coding unit, and 4) the immediately previous quantization unit in the coding order. It can be. The order may be reversed, and the upper left quantization unit may be omitted.

Meanwhile, the quantized transform block as described above is transmitted to the inverse quantization unit 135 and the scanning unit 131.

The scanning unit 131 scans the coefficients of the quantized transform block and converts them into 1-dimensional quantization coefficients. In this case, since the coefficient distribution of the transform block after quantization may be dependent on the intra prediction mode, the scanning method is applied to the intra prediction mode. can be determined according to

In addition, the coefficient scanning method may be determined differently according to the size of the transform unit, and the scan pattern may be changed according to the directional intra prediction mode. In this case, the scan order of quantization coefficients may be scanned in the reverse direction.

When the quantized coefficients are divided into a plurality of subsets, the same scan pattern may be applied to the quantization coefficients in each subset, and zigzag scan or diagonal scan may be applied to the scan pattern between the subsets.

Meanwhile, the scan pattern is preferably scanned from the main subset including DC to the remaining subsets in a forward direction, but the reverse direction is also possible.

Also, a scan pattern between subsets may be set identically to a scan pattern of quantized coefficients within the subset, and the scan pattern between subsets may be determined according to an intra prediction mode.

On the other hand, the encoding device 10 includes information indicating the position of the last non-zero quantization coefficient in the transform unit PU and the position of the last non-zero quantization coefficient in each subset in a bitstream so that the decoding device ( 20) can be sent.

The inverse quantization unit 135 inversely quantizes the quantized coefficients quantized as described above, and the inverse transformation unit 125 performs inverse transformation in units of transform units (TUs) to restore the inversely quantized transform coefficients into residual blocks in the spatial domain. can do.

The adder 195 may generate a reconstructed block by adding the residual block reconstructed by the inverse transform unit 125 and the prediction block received from the intra predictor 150 or the inter predictor 160.

In addition, the post-processing unit 170 performs a deblocking filtering process to remove a blocking effect occurring in the reconstructed picture, and a sample adaptive offset (Sample Adaptive Offset: Post-processing, such as a SAO) application process and an adaptive loop filtering (ALF) process for compensating for a difference value from an original video with a coding unit, may be performed.

The deblocking filtering process may be applied to a boundary of a prediction unit (PU) or a transform unit (TU) having a size greater than or equal to a predetermined size.

For example, the deblocking filtering process may include determining a boundary to be filtered, determining a boundary filtering strength to be applied to the boundary, determining whether to apply a deblocking filter, When it is determined to apply the deblocking filter, selecting a filter to be applied to the boundary may be included.

On the other hand, whether or not the deblocking filter is applied depends on i) whether the boundary filtering strength is greater than 0 and ii) the degree of change in pixel values at the boundary of two blocks (P block, Q block) adjacent to the boundary to be filtered. It may be determined by whether the indicated value is smaller than the first reference value determined by the quantization parameter.

It is preferable that the said filter is at least two or more. When the absolute value of the difference between two pixels located at the block boundary is greater than or equal to the second reference value, a filter that performs relatively weak filtering is selected.

The second reference value is determined by the quantization parameter and the boundary filtering strength.

In addition, the sample adaptive offset (SAO) application process is to reduce the distortion between pixels in the image to which the deblocking filter is applied and original pixels, and the sample adaptive offset (SAO) application process is performed in units of pictures or slices. It can be decided whether or not to do so.

A picture or slice may be divided into a plurality of offset regions, and an offset type may be determined for each offset region, and the offset type may include a predetermined number (eg, 4) of edge offset types and two band offsets. Can contain types.

For example, if the offset type is an edge offset type, an edge type to which each pixel belongs is determined and a corresponding offset is applied, and the edge type may be determined based on a distribution of values of two pixels adjacent to the current pixel. there is.

The adaptive loop filtering (ALF) process may perform filtering based on a value obtained by comparing a reconstructed image that has undergone a deblocking filtering process or an adaptive offset application process with an original video.

The picture storage unit 180 receives post-processed image data from the post-processing unit 170 and restores and stores the image in units of pictures. The picture may be an image in units of frames or an image in units of fields.

The inter prediction unit 160 may perform motion estimation using at least one reference picture stored in the picture storage unit 180 and determine a reference picture index and a motion vector representing the reference picture.

In this case, a prediction block corresponding to a prediction unit to be encoded may be extracted from a reference picture used for motion estimation among a plurality of reference pictures stored in the picture storage 180 according to the determined reference picture index and motion vector. there is.

The intra prediction unit 150 may perform intra prediction encoding using reconstructed pixel values inside a picture including a current prediction unit.

The intra prediction unit 150 may perform intra prediction by receiving a current prediction unit to be predictively encoded and selecting one of a preset number of intra prediction modes according to the size of a current block.

The intra predictor 150 may adaptively filter reference pixels to generate an intra prediction block, and generate reference pixels using available reference pixels when a reference pixel is not available.

The entropy encoding unit 140 may entropy encode the quantization coefficients quantized by the quantization unit 130, the intra prediction information received from the intra prediction unit 150, and the motion information received from the inter prediction unit 160. .

FIG. 2 is a block diagram showing an embodiment of a configuration for performing inter prediction in the encoding device 10. The illustrated inter prediction encoder includes a motion information determining unit 161 and a motion information encoding mode determining unit 162 , a motion information encoder 163, a prediction block generator 164, a residual block generator 165, a residual block encoder 166, and a multiplexer 167.

Referring to FIG. 2 , the motion information determining unit 161 determines motion information of a current block, the motion information includes a reference picture index and a motion vector, and the reference picture index is one of previously encoded and reconstructed pictures. can represent

A reference picture indicating one of the reference pictures belonging to list 0 (L0) when the current block is unidirectional inter-prediction coded, and indicating one of the reference pictures of list 0 (L0) when the current block is bi-directional predictive coded. index and a reference picture index indicating one of the reference pictures of List 1 (L1).

In addition, when the current block is bi-directionally predicted-coded, it may include an index indicating one or two pictures among reference pictures of a composite list (LC) generated by combining list 0 and list 1.

A motion vector represents a position of a prediction block within a picture indicated by each reference picture index, and the motion vector may be in a pixel unit (integer unit) or a sub-pixel unit.

For example, the motion vector may have a precision of 1/2, 1/4, 1/8, or 1/16 pixels, and if the motion vector is not an integer unit, a prediction block may be generated from pixels in integer units. can

The motion information encoding mode determination unit 162 may determine an encoding mode for the motion information of the current block, and the encoding mode may be exemplified by any one of a skip mode, a merge mode, and an AMVP mode.

The skip mode is applied when there is a skip candidate having the same motion information as that of the current block and the residual signal is 0. can be applied when is equal to

The merge mode is applied when there is a merge candidate having the same motion information as the motion information of the current block. applies in case Meanwhile, the merge candidate and the skip candidate may be the same.

The AMVP mode is applied when the skip mode and merge mode are not applied, and an AMVP candidate having a motion vector most similar to that of the current block may be selected as an AMVP predictor.

However, the encoding mode may adaptively include a more subdivided motion compensation prediction encoding mode as a process other than the exemplified method. Adaptively determined motion compensation prediction modes include not only the aforementioned AMVP mode, merge mode, and skip mode, but also FRUC (FRAME RATE UP-CONVERSION) mode, which is currently proposed as a new motion compensation prediction mode, and BI-DIRECTIONAL OPTICAL FLOW (BIO) ) mode, AMP (AFFINE MOTION PREDICTION) mode, OBMC (OVERLAPPED BLOCK MOTION COMPENSATION) mode, DMVR (DECODER-SIDE MOTION VECTOR REFINEMENT) mode, ATMVP (Alternative temporal motion vector prediction) mode, STMVP (Spatial-temporal motion vector prediction) At least one of a mode and a local illumination compensation (LIC) mode may be further included, and the block may be adaptively determined according to a predetermined condition.

The motion information encoding unit 163 may encode the motion information according to the method determined by the motion information encoding mode determination unit 162 .

For example, the motion information encoder 163 may perform a merge motion vector encoding process when the motion information encoding mode is a skip mode or a merge mode, and may perform an AMVP encoding process when the motion information encoding mode is the AMVP mode.

The prediction block generation unit 164 generates a prediction block using motion information of the current block, and when the motion vector is an integer unit, copies a block corresponding to a position indicated by the motion vector in a picture indicated by a reference picture index to the current block. Generates a prediction block of

Meanwhile, when the motion vector is not an integer unit, the prediction block generator 164 may generate pixels of a prediction block from pixels of an integer unit in a picture indicated by a reference picture index.

In this case, a predicted pixel may be generated using an 8-tap interpolation filter for a luminance pixel, and a predicted pixel may be generated using a 4-tap interpolation filter for a chrominance pixel.

The residual block generating unit 165 generates a residual block using a current block and a prediction block of the current block, and when the size of the current block is 2Nx2N, a residual block is generated using the current block and a prediction block having a size of 2Nx2N corresponding to the current block. blocks can be created.

Meanwhile, when the size of the current block used for prediction is 2NxN or Nx2N, after obtaining a prediction block for each of two 2NxN blocks constituting 2Nx2N, the final prediction block of size 2Nx2N is obtained by using the two 2NxN prediction blocks. can be created

In addition, a 2Nx2N-sized residual block may be generated using the 2Nx2N-sized prediction block, and overlap smoothing is applied to pixels of the boundary portion in order to resolve the discontinuity of the boundary portion of the two 2NxN-sized prediction blocks. can

The residual block encoder 166 divides the residual block into one or more transform units (TUs), and each transform unit (TU) can be transform-encoded, quantized, and entropy-encoded.

The residual block encoder 166 may transform the residual block generated by the inter prediction method using an integer-based transform matrix, and the transform matrix may be an integer-based DCT matrix.

Meanwhile, the residual block encoder 166 uses a quantization matrix to quantize coefficients of the residual block transformed by the transformation matrix, and the quantization matrix may be determined by a quantization parameter.

The quantization parameter is determined for each coding unit (CU) having a predetermined size or more, and when the current coding unit (CU) is smaller than the predetermined size, the first coding unit (CU) among coding units (CUs) within the predetermined size in coding order ( Since only the quantization parameters of the CU) are coded and the quantization parameters of the remaining coding units (CUs) are identical to the above parameters, coding may not be performed.

Also, the coefficients of the transform block may be quantized using a quantization matrix determined according to the quantization parameter and the prediction mode.

A quantization parameter determined for each coding unit (CU) having a predetermined size or larger may be predicted and coded using a quantization parameter of a coding unit (CU) adjacent to the current coding unit (CU).

A quantization parameter predictor of the current coding unit (CU) may be generated using one or two valid quantization parameters by searching in the order of the left coding unit (CU) and the upper coding unit (CU) of the current coding unit (CU). there is.

For example, the first effective quantization parameter searched in the above order may be determined as the quantization parameter predictor, and the first valid quantization parameter may be quantized by searching in the order of the left coding unit (CU) and the immediately previous coding unit (CU) in the coding order. It can be determined by the parameter predictor.

Coefficients of the quantized transform block are scanned and converted into one-dimensional quantization coefficients, and a scanning method may be set differently according to an entropy encoding mode.

For example, in the case of CABAC encoding, inter-prediction coded quantization coefficients may be scanned in one predetermined method (zigzag or raster scan in a diagonal direction), and in case of CAVLC encoding, scanning in a different method from the above method. It can be.

For example, the scanning method may be determined according to zigzag in the case of inter and the intra prediction mode in the case of intra, and the coefficient scanning method may be determined differently according to the size of the transformation unit.

Meanwhile, the scan pattern may vary according to the directional intra prediction mode, and the scan order of quantization coefficients may be reversed.

The multiplexer 167 multiplexes the motion information encoded by the motion information encoder 163 and the residual signals encoded by the residual block encoder 166.

The motion information may vary according to the coding mode. For example, in case of skip or merge, it may include only an index indicating a predictor, and in case of AMVP, it may include a reference picture index of the current block, a differential motion vector, and an AMVP index. .

Hereinafter, an embodiment of the operation of the intra prediction unit 150 shown in FIG. 1 will be described in detail.

First, the intra prediction unit 150 receives prediction mode information and the size of the prediction unit (PU) from the picture divider 110, and sets reference pixels to the picture storage unit to determine the intra prediction mode of the prediction unit (PU). It can be read from (180).

The intra prediction unit 150 determines whether a reference pixel is generated by examining whether unavailable reference pixels exist, and the reference pixels can be used to determine an intra prediction mode of a current block.

When the current block is located on the upper boundary of the current picture, pixels adjacent to the upper side of the current block are not defined, and when the current block is located on the left boundary of the current picture, pixels adjacent to the left of the current block are not defined, The pixels may be determined to be not usable pixels.

Also, when the current block is located on a slice boundary and pixels adjacent to the upper or left side of the slice are not pixels to be encoded and reconstructed first, it may be determined that they are not usable pixels.

As described above, when pixels adjacent to the left side or above the current block do not exist or pixels that have been previously encoded and reconstructed do not exist, the intra prediction mode of the current block may be determined using only available pixels.

Meanwhile, a reference pixel at an unavailable location may be generated using available reference pixels of the current block. For example, when pixels of the upper block are not available, some or all of the left pixels may be used to generate the upper pixel. can be created, and vice versa.

That is, a reference pixel is generated by copying an available reference pixel at a location closest to a predetermined direction from a reference pixel at an unavailable location, or when there is no reference pixel available in a predetermined direction, the closest available reference pixel in the opposite direction exists. A reference pixel may be created by copying an available reference pixel of a location.

Meanwhile, even when there are pixels above or to the left of the current block, they may be determined as unavailable reference pixels according to the coding mode of the block to which the pixels belong.

For example, if a block to which a reference pixel adjacent to the upper side of the current block belongs is a block reconstructed by inter-encoding, the pixels may be determined as unavailable pixels.

In this case, a block adjacent to the current block is intra-coded, and usable reference pixels may be generated using pixels belonging to the reconstructed block, and the information that the encoding apparatus 10 determines available reference pixels according to the encoding mode is provided. It is transmitted to the decryption device 20.

The intra prediction unit 150 determines an intra prediction mode of a current block by using the reference pixels, and the number of intra prediction modes allowable for the current block may vary depending on the size of the block.

The intra prediction mode may include at least one non-directional mode (non-directional mode) and a plurality of directional modes (directional modes).

One or more non-directional modes may be DC modes and/or planar modes. When the DC mode and the planar mode are included as non-directional modes, a preset number of intra prediction modes may exist regardless of the size of the current block.

In the planar mode, a prediction block of the current block is formed using at least one pixel value (or a predicted value of the pixel value, hereinafter referred to as a first reference value) located at the bottom-right of the current block and reference pixels. is created

Meanwhile, the configuration of the video decoding apparatus 20 according to an embodiment of the present invention may be derived from the configuration of the video encoding apparatus 10 described with reference to FIGS. 1 and 2, and for example, FIGS. An image may be decoded by reversely performing the steps of the image encoding method described with reference to .

3 is a block diagram showing the configuration of a video decoding apparatus according to an embodiment of the present invention. The decoding apparatus 20 includes an entropy decoding unit 210, an inverse quantization/inverse transform unit 220, an adder 270, It includes a post-processing unit 250, a picture storage unit 260, an intra prediction unit 230, a motion compensated prediction unit 240, and an intra/inter conversion switch 280.

The entropy decoding unit 210 receives and decodes the bit stream encoded by the video encoding apparatus 10, separates it into an intra prediction mode index, motion information, a quantization coefficient sequence, etc., and converts the decoded motion information into a motion compensation prediction unit ( 240).

The entropy decoding unit 210 transfers the intra prediction mode index to the intra prediction unit 230 and the inverse quantization/inverse transformation unit 220, and transfers the inverse quantization coefficient sequence to the inverse quantization/inverse transformation unit 220.

The inverse quantization/inverse transformation unit 220 transforms the quantization coefficient sequence into a 2-dimensional array of inverse quantization coefficients, and may select one of a plurality of scanning patterns for the conversion. For example, the prediction mode of the current block (i.e., , intra prediction or inter prediction) and an intra prediction mode, a scanning pattern may be selected.

The inverse quantization/inverse transform unit 220 restores the quantization coefficient by applying a quantization matrix selected from among a plurality of quantization matrices to the inverse quantization coefficient of the two-dimensional array.

Meanwhile, different quantization matrices are applied according to the size of a current block to be reconstructed, and a quantization matrix may be selected for blocks of the same size based on at least one of a prediction mode and an intra prediction mode of the current block.

The inverse quantization/inverse transformation unit 220 restores a residual block by inversely transforming the reconstructed quantization coefficient, and the inverse transformation process may be performed using a transform unit (TU) as a basic unit.

The adder 270 restores an image block by adding the residual block reconstructed by the inverse quantization/inverse transformation unit 220 and the prediction block generated by the intra prediction unit 230 or the motion compensation prediction unit 240.

The post-processing unit 250 may perform post-processing on the reconstructed image generated by the adder 270 to reduce deblocking artifacts caused by image loss due to the quantization process by filtering or the like.

The picture storage unit 260 is a frame memory for storing a locally decoded image on which filter post-processing is performed by the post-processing unit 250 .

The intra prediction unit 230 restores the intra prediction mode of the current block based on the intra prediction mode index received from the entropy decoding unit 210 and generates a prediction block according to the restored intra prediction mode.

The motion compensation prediction unit 240 generates a prediction block for a current block from a picture stored in the picture storage unit 260 based on the motion vector information, and when decimal precision motion compensation is applied, a selected interpolation filter is applied to the prediction block. can create

The intra/inter conversion switch 280 may provide the prediction block generated by either the intra prediction unit 230 or the motion compensation prediction unit 240 to the adder 270 based on the encoding mode.

4 is a block diagram showing an embodiment of a configuration for performing inter prediction in the video decoding apparatus 20. The inter prediction decoder includes a demultiplexer 241, a motion information encoding mode determination unit 242, and a merge mode motion Information decoding unit 243, AMVP mode motion information decoding unit 244, selection mode motion information decoding unit 248, prediction block generator 245, residual block decoding unit 246, and reconstruction block generator 247 includes

Referring to FIG. 4 , the demultiplexer 241 demultiplexes currently encoded motion information and encoded residual signals from the received bitstream, and transmits the demultiplexed motion information to the motion information encoding mode determination unit 242. and the demultiplexed residual signal may be transmitted to the residual block decoder 246.

The motion information encoding mode determination unit 242 determines the motion information encoding mode of the current block, and if skip_flag of the received bitstream has a value of 1, it is determined that the motion information encoding mode of the current block is encoded in the skip encoding mode. can do.

The motion information encoding mode determining unit 242 determines the motion information encoding mode of the current block when skip_flag of the received bitstream has a value of 0 and the motion information received from the demultiplexer 241 has only a merge index. It can be determined that is encoded in the merge mode.

In addition, the motion information coding mode determination unit 242 determines that the skip_flag of the received bitstream has a value of 0 and the motion information received from the demultiplexer 241 has a reference picture index, a differential motion vector, and an AMVP index. In this case, it can be determined that the motion information encoding mode of the current block is encoded in the AMVP mode.

The merge mode motion information decoding unit 243 is activated when the motion information encoding mode determination unit 242 determines that the motion information encoding mode of the current block is skip or merge mode. It can be activated when the information encoding mode determination unit 242 determines that the motion information encoding mode of the current block is the AMVP mode.

The selected mode motion information decoding unit 248 may decode motion information in a prediction mode selected from motion compensation prediction modes other than the aforementioned AMVP mode, merge mode, and skip mode. The selected prediction mode may include a more precise motion prediction mode than the AMVP mode, and may be adaptively determined according to predetermined conditions (eg, block size and block division information, signaling information presence, block location, etc.) . The selection prediction mode is, for example, FRUC (FRAME RATE UP-CONVERSION) mode, BIO (BI-DIRECTIONAL OPTICAL FLOW) mode, AMP (AFFINE MOTION PREDICTION) mode, OBMC (OVERLAPPED BLOCK MOTION COMPENSATION) mode, DMVR (DECODER-SIDE) mode. It may include at least one of a MOTION VECTOR REFINEMENT (MOTION VECTOR REFINEMENT) mode, an alternative temporal motion vector prediction (ATMVP) mode, a spatial-temporal motion vector prediction (STMVP) mode, and a local illumination compensation (LIC) mode.

The prediction block generation unit 245 generates a prediction block of a current block using motion information reconstructed by the merge mode motion information decoding unit 243 or the AMVP mode motion information decoding unit 244.

When the motion vector is an integer unit, a prediction block of the current block may be generated by copying a block corresponding to a position indicated by the motion vector in a picture indicated by a reference picture index.

On the other hand, if the motion vector is not an integer unit, pixels of a prediction block are generated from pixels of an integer unit within a picture indicated by a reference picture index. In this case, an 8-tap interpolation filter is used for luminance pixels and chrominance pixels A prediction pixel may be generated using a 4-tap interpolation filter.

The residual block decoder 246 entropy-decodes the residual signal and inverse-scans the entropy-decoded coefficients to generate a 2-dimensional quantized coefficient block. The inverse scanning method may vary depending on the entropy decoding method.

For example, the inverse scanning method may be applied as a raster inverse scan method in a diagonal direction in case of CABAC-based decoding, and as a zigzag inverse scan method in case of CAVLC-based decoding. Also, the inverse scanning method may be determined differently according to the size of the prediction block.

The residual block decoder 246 inversely quantizes the generated coefficient block using an inverse quantization matrix, and may restore quantization parameters to derive the quantization matrix. Here, the quantization step size may be reconstructed for each coding unit equal to or greater than a predetermined size.

The residual block decoder 260 inversely transforms the inverse quantized coefficient block to restore a residual block.

The reconstruction block generator 270 generates a reconstruction block by adding the prediction block generated by the prediction block generator 250 and the residual block generated by the residual block decoder 260.

Hereinafter, an embodiment of a process of reconstructing a current block through intra prediction will be described with reference to FIG. 7 again.

First, the intra prediction mode of the current block is decoded from the received bitstream, and for this, the entropy decoding unit 210 restores the first intra prediction mode index of the current block by referring to one of a plurality of intra prediction mode tables. can

As a table shared by the encoding apparatus 10 and the decoding apparatus 20 of the plurality of intra prediction mode tables, any one table selected according to the distribution of intra prediction modes for a plurality of blocks adjacent to the current block may be applied. .

For example, if the intra prediction mode of the left block of the current block and the intra prediction mode of the upper block of the current block are the same, the first intra prediction mode index of the current block is restored by applying the first intra prediction mode table, and the first intra prediction mode index of the current block is restored. Otherwise, the first intra prediction mode index of the current block may be restored by applying the second intra prediction mode table.

As another example, when the intra prediction modes of the upper block and the left block of the current block are both directional intra prediction modes, the direction of the intra prediction mode of the upper block and the direction of the intra prediction mode of the left block If the angle is within the predetermined angle, the first intra prediction mode index of the current block is restored by applying the first intra prediction mode table, and if the angle is out of the predetermined angle, the first intra prediction mode index of the current block is restored by applying the second intra prediction mode table. can also be restored.

The entropy decoding unit 210 transmits the first intra prediction mode index of the reconstructed current block to the intra prediction unit 230 .

Upon receiving the index of the first intra prediction mode, the intra predictor 230 may determine the maximum possible mode of the current block as the intra prediction mode of the current block when the index has a minimum value (ie, 0). .

Meanwhile, when the index has a value other than 0, the intra predictor 230 compares the index indicated by the maximum possible mode of the current block with the first intra prediction mode index, and as a result of the comparison, the first intra prediction mode index. If the index is not smaller than the index indicated by the maximum possible mode of the current block, an intra-prediction mode corresponding to a second intra-prediction mode index obtained by adding 1 to the first intra-prediction mode index is determined as the intra-prediction mode of the current block; Otherwise, the intra prediction mode corresponding to the first intra prediction mode index may be determined as the intra prediction mode of the current block.

The intra prediction modes allowable for the current block may include at least one non-directional mode (non-directional mode) and a plurality of directional modes (directional modes).

One or more non-directional modes may be DC modes and/or planar modes. Also, any one of a DC mode and a planner mode may be adaptively included in the set of allowable intra prediction modes.

To this end, information specifying a non-directional mode included in the set of allowable intra prediction modes may be included in a picture header or slice header.

Next, the intra prediction unit 230 reads reference pixels from the picture storage unit 260 to generate an intra prediction block, and determines whether unavailable reference pixels exist.

The determination may be made according to whether reference pixels used to generate an intra-prediction block by applying the decoded intra-prediction mode of the current block exist.

Next, when it is necessary to generate a reference pixel, the intra prediction unit 230 may generate reference pixels at unavailable locations using previously reconstructed available reference pixels.

The definition of an unavailable reference pixel and the method of generating the reference pixel may be the same as the operation of the intra prediction unit 150 according to FIG. 1, but an intra prediction block is generated according to the decoded intra prediction mode of the current block. Reference pixels used to do this may be selectively reconstructed.

In addition, the intra prediction unit 230 determines whether or not to apply a filter to reference pixels to generate a prediction block, that is, whether to apply filtering to reference pixels to generate an intra prediction block of the current block. It may be determined based on the decoded intra prediction mode and the size of the current prediction block.

Since the problem of blocking artifacts increases as the size of the block increases, the number of prediction modes for filtering reference pixels can be increased as the size of the block increases. For this, the reference pixel may not be filtered.

When it is determined that a filter needs to be applied to the reference pixel, the intra prediction unit 230 filters the reference pixels using a filter.

At least two or more filters may be adaptively applied according to the level of difference between the reference pixels. The filter coefficient of the filter is preferably symmetrical.

In addition, the above two or more filters may be adaptively applied according to the size of the current block. When applying the filter, a filter with a narrow bandwidth is used for small blocks and a filter with a wide bandwidth is used for large blocks. may be applied.

In the case of DC mode, a prediction block is generated with the average value of reference pixels, so there is no need to apply a filter. It may not be necessary to apply a filter to a reference pixel even in a horizontal mode in which direction is correlated.

In this way, since whether filtering is applied is also related to the intra prediction mode of the current block, the reference pixel may be adaptively filtered based on the intra prediction mode of the current block and the size of the prediction block.

Next, the intra prediction unit 230 generates a prediction block using a reference pixel or filtered reference pixels according to the reconstructed intra prediction mode, and generation of the prediction block is the same as the operation in the encoding apparatus 10. Therefore, a detailed description thereof will be omitted.

The intra predictor 230 determines whether to filter the generated prediction block, and the filtering can be determined using information included in a slice header or a coding unit header or according to an intra prediction mode of a current block.

When it is determined that the generated prediction block is to be filtered, the intra predictor 230 may generate a new pixel by filtering a pixel at a specific position of the generated prediction block using available reference pixels adjacent to the current block. .

For example, in the DC mode, among prediction pixels, prediction pixels that come into contact with reference pixels may be filtered using the reference pixels that come into contact with the prediction pixels.

Therefore, prediction pixels are filtered using one or two reference pixels according to the position of the prediction pixels, and prediction pixel filtering in the DC mode can be applied to prediction blocks of all sizes.

Meanwhile, in the vertical mode, among prediction pixels of a prediction block, prediction pixels adjacent to a left reference pixel may be changed using reference pixels other than an upper pixel used to generate the prediction block.

Similarly, in the horizontal mode, among generated prediction pixels, prediction pixels contacting an upper reference pixel may be changed using reference pixels other than the left pixel used to generate the prediction block.

The current block may be reconstructed using the predicted block of the current block reconstructed in this way and the residual block of the current block decoded.

5 is a flowchart illustrating an operation of an image encoding apparatus according to an embodiment of the present invention.

Referring to FIG. 5 , the video encoding apparatus 10 according to an embodiment of the present invention determines the encoding resolution for each sub-picture to be an optimized value and performs a variable encoding process, and the encoding information is stored in the decoding apparatus 20 It is signaled as , so that efficient decryption is processed.

According to an embodiment of the present invention for this purpose, in dividing a picture, the picture division unit 110 determines a subpicture division structure that can be coded and decoded independently of other division units and a resolution for each subpicture. , signaling information including adaptive resolution information corresponding thereto may be configured.

Furthermore, the picture divider 110 according to an embodiment of the present invention may divide the picture into subpictures according to the arrangement of an ultra-high resolution image such as a 360-degree video, and configuration information corresponding to each subpicture. By encoding and transmitting to the decoding apparatus 20, independent processing of tiles included in a subpicture is possible. Here, a subpicture is formed by dividing a picture, and may represent one or more rectangular areas configurable as tiles or slices. Therefore, although the division area according to the embodiment of the present invention is mainly described as a subpicture, it is a rectangular area dividing a picture, and each subpicture according to signaling of configuration information for a subpicture set including the subpictures. Independent processing of pictures can be determined.

Here, independence may mean that encoding and decoding processes including intra prediction, inter prediction, transform, quantization, entropy, and filter can be processed as an independent picture regardless of divided subpictures. However, this may not mean that all encoding and decoding processes are completely independently performed for each subpicture, and the current subpicture is encoded and decoded by selectively using information of other subpictures during inter prediction or in-loop filtering. may be decrypted.

Accordingly, referring to FIG. 5 , the encoding apparatus 10 according to an embodiment of the present invention divides an original video into subpictures (S101), and determines a coding resolution for each subpicture that can maximize transmission efficiency. (S103).

Then, the encoding apparatus 10 according to the embodiment of the present invention encodes the original video according to the encoding resolution for each sub-picture (S105).

Then, the encoding apparatus 10 according to the embodiment of the present invention generates subpicture signaling information including resolution control information (S107).

And, the encoding apparatus 10 according to the embodiment of the present invention configures a bitstream including the encoded video and subpicture signaling information (S109).

Meanwhile, referring to FIG. 6 , the decoding apparatus 20 according to an embodiment of the present invention obtains subpicture signaling information including resolution adjustment information from a bitstream received from the encoding apparatus 10 (S201) ( S203).

Thereafter, the decoding apparatus 20 processes subpicture decoding of different resolutions based on the subpicture signaling information (S205).

Then, the decoding device 20 processes reconstruction of a restored image using the decoded subpictures (S207).

7 is a comparison diagram between an original image and a case where the resolution of each subpicture is adjusted according to an embodiment of the present invention.

More specifically, referring to FIG. 7 , when encoding a high-resolution video, the encoding apparatus 10 according to an embodiment of the present invention compresses an important region where the user's gaze stays and other regions with relatively different resolutions. In this way, the video can be streamed and restored.

In particular, prior art such as HEVC proposes a method of compressing existing high-resolution images by varying bit-rates for each region (tile) in order to efficiently compress them, but this does not significantly increase the data compression rate. Since this method is different from changing the resolution, the encoding apparatus 10 according to an embodiment of the present invention may provide compression processing of a high-resolution image according to changing the resolution for each sub-picture.

The main characteristics of high-resolution images are that the amount of data to be processed is enormous due to the large resolution and that the user can focus on a specific limited area rather than viewing the entire large resolution at once. That is, the visual sensitivity is lowered for the rest of the region other than the central region of the gaze. Accordingly, the encoding apparatus 10 according to an embodiment of the present invention can provide a method for encoding a high-resolution image by utilizing the above two characteristics.

Furthermore, the encoding device 10 and the decoding device 20 according to an embodiment of the present invention may perform 360-degree image processing, which is a video for VR (virtual reality), which is a representative field of high-resolution video, but the present invention is VR , 360-degree images, etc., and may also be applied to general two-dimensional high-resolution images.

In particular, as a technology for encoding high-resolution video, H.266/VVC international standard technology has been recently enacted, and an embodiment of the present invention may also partially borrow international standard technologies in encoding and decoding processes. However, the picture division unit 110 according to an embodiment of the present invention can bring about improved coding and decoding efficiency by supporting a function of varying coding resolution for each subpicture, which is not supported in the current H.266/VVC.

As described above, even in the VVC standard, a subpicture can be defined as an independent area division for compositing with other content, and can be treated as a completely independent image to indicate an area that can be coded/decoded.

In the embodiment of the present invention as shown in FIG. 7 , the encoding device 10 may divide six faces of a 360 image (cube map format, A) into respective subpictures.

Referring to FIG. 7, the resolution of the original video is set to 1920x1152, the resolution of one side (sub-picture) is set to 640x576, and the size of the coding tree unit (CTU) is 64x64. The size of a sub-picture with a resolution of 1/2 of 7 may be set to 320x288. The above settings are for explanation, and the present invention is not limited to numbers. For example, the CTU size may be separately signaled as header information.

And, as shown in FIG. 7(B), the encoding apparatus 10 according to the embodiment of the present invention divides one original video into several subpictures, and then each subpicture has a different resolution. (resolution) can be encoded.

For example, as shown in FIG. 7(B), if a viewport of a 360 image is located in subpictures 1 and 2, the encoding device 10 determines that subpictures 1 and 2 have the original resolution ( resolution), and subpictures 3 through 6 can be coded with a resolution corresponding to 1/2 of the original (resolution). This is possible because in the present invention, subpictures are treated as a completely independent image and can be coded/decoded, and at this time, the resolution change between subpictures according to the viewport change is adaptive. It could be possible.

Here, the subpicture signaling information may be exemplified by header information for each subpicture, and the header information is in the form of a high level syntax and may be included in picture header information, picture parameter information, or sequence parameter information of an encoded video bitstream. there is. In addition, the header information, as another type of high level syntax, may be included in a supplemental enhancement information (SEI) message of an encoded video bitstream and transmitted.

In addition, subpicture signaling information whose resolution is adjusted for each subpicture may be defined and included in a sequence parameter set (SPS) defined in, for example, HEVC, VVC, and the like. Table 1 below is an exemplary syntax diagram of a sequence parameter set including subpicture signaling information according to an embodiment of the present invention.

seq_parameter_set_rbsp(　) { sps_subpic_info_present_flag if( sps_subpic_info_present_flag ) { sps_num_subpics_minus1 if( sps_num_subpics_minus1 > 0 ) { sps_independent_subpics_flag sps_subpic_same_size_flag } for( i = 0; sps_num_subpics_minus1 > 0 && i <= sps_num_subpics_minus1; i++ ) { if( !sps_subpic_same_size_flag |　| i =　= 0 ) { if( i > 0 && sps_pic_width_max_in_luma_samples > CtbSizeY ) sps_subpic_ctu_top_left_x [ i ] if( i > 0 && sps_pic_height_max_in_luma_samples > CtbSizeY ) sps_subpic_ctu_top_left_y [ i ] if( i < sps_num_subpics_minus1 &&
sps_pic_width_max_in_luma_samples > CtbSizeY ) sps_subpic_width_minus1 [ i ] if( i < sps_num_subpics_minus1 &&
sps_pic_height_max_in_luma_samples > CtbSizeY ) sps_subpic_height_minus1 [ i ] } if( !sps_independent_subpics_flag) { sps_subpic_treated_as_pic_flag [ i ] sps_loop_filter_across_subpic_enabled_flag [ i ] } } sps_subpic_id_len_minus1 sps_subpic_id_mapping_explicitly_signalled_flag if( sps_subpic_id_mapping_explicitly_signalled_flag ) { sps_subpic_id_mapping_present_flag if( sps_subpic_id_mapping_present_flag ) for( i = 0; i <= sps_num_subpics_minus1; i++ ) sps_subpic_id [ i ] } }

sps_subpic_info_present_flag may indicate a flag indicating whether a subpicture exists in a corresponding sequence.

sps_num_subpics_minus1 may represent a value obtained by subtracting 1 from the number of subpictures present in one image.

sps_independent_subpics_flag may indicate a flag indicating whether all subpictures are independently coded (here, it means independence for inter prediction and in-loop filter)

sps_subpic_same_size_flag may indicate a flag indicating that all subpictures are the same size.

sps_subpic_ctu_top_left_x & sps_subpic_ctu_top_left_y may indicate the (x, y) value of the CTU located at the top left in one subpicture (CTU index value on the horizontal axis and CTU index value on the vertical axis)

sps_subpic_width_minus1 & sps_subpic_height_minus1 may indicate the width and height values of a subpicture based on the upper leftmost CTU in a subpicture (expressed in the number of CTUs)

sps_subpic_treated_as_pic_flag is a flag indicating whether to treat subpicture boundaries like picture boundaries (e.g., during inter prediction, instructs to treat subpicture boundaries like picture boundaries)

sps_loop_filter_across_subpic_enabled_flag is a flag indicating whether to perform in-loop filtering at the boundary between subpictures.

sps_subpic_id_len_minus1 : A value representing the length of subpicture id.

sps_subpic_id_mapping_explicitly_signalled_flag : A flag indicating whether to explicitly transmit the id of a subpicture.

sps_subpic_id_mapping_present_flag is a flag indicating whether to map the id of a subpicture.

sps_subpic_id : Indicates the id of a subpicture. (transmit id as many as the number of subpictures)

Corresponding to each syntax as described above, when applied to FIG. 7(B), each value may be mapped as follows.

sps_subpic_info_present_flag True sps_num_subpics_minus1 5 (6-1) sps_independent_subpics_flag True sps_subpic_same_size_flag True sps_subpic_ctu_top_left_x & sps_subpic_ctu_top_left_y - sps_subpic_width_minus1 9 (10-1) sps_subpic_height_minus1 8 (9-1) sps_subpic_treated_as_pic_flag - sps_loop_filter_across_subpic_enabled_flag - sps_subpic_id_len_minus1 2 (3-1) sps_subpic_id_mapping_explicitly_signalled_flag One sps_subpic_id_mapping_present_flag One sps_subpic_id 1, 2, 3, 4, 5, 6

Accordingly, the decoding apparatus 20 may obtain the corresponding syntax information from the sequence parameter set and check the number of subpictures, mapping id, independent filtering, horizontal length, vertical length, whether or not they are processed as independent pictures, and the like. And, the decoding apparatus 20 may identify the location and shape of individual subpictures based on the subpicture signaling information on the SPS (Sequence Parameter Set).

Meanwhile, variable resolution information according to an embodiment of the present invention may be included in separate flag information and a picture parameter set (PPS) in the SPS. This will be illustrated in more detail through Tables 3 and 4. As another example, the variable resolution information included in the picture parameter set (PPS) may be included in a picture header (PH).

seq_parameter_set_rbsp(　) { sps_ref_pic_resampling_enabled_flag if( sps_ref_pic_resampling_enabled_flag ) sps_res_change_in_clvs_allowed_flag sps_pic_width_max_in_luma_samples sps_pic_height_max_in_luma_samples sps_conformance_window_flag if( sps_conformance_window_flag ) { sps_conf_win_left_offset sps_conf_win_right_offset sps_conf_win_top_offset sps_conf_win_bottom_offset }

sps_ref_pic_resampling_enabled_flag : A flag that allows resampling of the reference picture (due to adaptive resolution, if the resolution of the picture to be decoded and the corresponding reference picture are different, resampling of the reference picture is required)

sps_res_change_in_clvs_allowed_flag : Flag allowing change of resolution of decoded picture

sps_pic_width_max_in_luma_samples : width value of original resolution of decoded picture (based on luma)

sps_pic_height_max_in_luma_samples : height value of original resolution of decoded picture (based on luma)

sps_conformance_window_flag : A flag indicating that the size of the output picture is to be set.

sps_conf_win_left_offset : left offset value for output picture

sps_conf_win_right_offset : right offset value for output picture

sps_conf_win_top_offset : Top offset value for output picture

sps_conf_win_bottom_offset : Bottom offset value for output picture

Here, the conformance window flag information (sps_conformance_window_flag) indicating that the size of the output picture is set within the sequence unit is cropped due to a difference in resolution between the picture reconstructed in the decoding device 20 or the encoding device 10 and the picture to be output. It can be used to determine if cropping is necessary. The decoding device 20 or the coding device 10 crops a picture reconstructed from a decoded picture buffer (DPB) using conformance window flag information, and the cropped picture is output can be processed. If there is no designation of separate window information in the corresponding flag, it may mean that the reconstructed picture and the output picture have the same size.

pic_parameter_set_rbsp(　) { pps_pic_width_in_luma_samples pps_pic_height_in_luma_samples pps_conformance_window_flag if( pps_conformance_window_flag ) { pps_conf_win_left_offset pps_conf_win_right_offset pps_conf_win_top_offset pps_conf_win_bottom_offset } pps_scaling_window_explicit_signalling_flag if( pps_scaling_window_explicit_signalling_flag ) { pps_scaling_win_left_offset pps_scaling_win_right_offset pps_scaling_win_top_offset pps_scaling_win_bottom_offset }

pps_pic_width_in_luma_samples : width value of resolution of decoded picture/subpicture (based on luma)

pps_pic_height_in_luma_samples : height value of resolution of decoded picture/sub picture (based on luma)

pps_conformance_window_flag : A flag indicating that the size of the output picture/subpicture is set.

pps_conf_win_left_offset : left offset value for output picture/sub picture

pps_conf_win_right_offset : right offset value for output picture/sub picture

pps_conf_win_top_offset : Top offset value for output picture/sub picture

pps_conf_win_bottom_offset : Bottom offset value for output picture/sub picture

Here, the conformance window is the same as the SPS content, but is information about a sub-picture rather than a picture.

pps_scaling_window_explicit_signaling_flag : Flag notifying that the size of the reference picture/sub picture is set

pps_scaling_win_left_offset : left offset value for reference picture/sub picture

pps_scaling_win_right_offset : Right offset value for reference picture/sub picture

pps_scaling_win_top_offset : Top offset value for reference picture/sub picture

pps_scaling_win_bottom_offset : Lower offset value for reference picture/sub picture

Here, the conformance window flag information (pps_conformance_window_flag) indicating that the size of an output picture/subpicture is set within a picture unit is a picture/subpicture reconstructed in the decoding device 20 or the coding device 10 and a picture/subpicture to be output. It can be used to determine whether cropping is necessary because resolutions of subpictures are different. The decoding device 20 or the coding device 10 crops a picture/subpicture reconstructed from a decoded picture buffer (DPB) using conformance window flag information, and , the cropped picture/subpicture can be output processed. If there is no separate window information designation in the corresponding flag, it may mean that the reconstructed picture/subpicture and the output picture/subpicture have the same size.

8 is an exemplary diagram illustrating a relationship between resolution adjustment for each (sub) picture and GOP units according to an embodiment of the present invention.

As described in Table 4, the size of the horizontal axis and the vertical axis of the scaling window to be encoded or decoded is the width and height of a specific sub-picture in the sub-picture information of SPS and PPS. It can be obtained by subtracting left and right and top and bottom offset values from the length. Alternatively, the size of the horizontal axis and vertical axis of the scaling window to be encoded or decoded is obtained by subtracting the left and right and top and bottom offset values of the scaling window from the width and height length of the decoded subpicture resolution of PPS or PH. can be obtained

Then, the encoding device 10 or the decoding device 20 compares the scaling window value of the reconstructed subpicture with the corresponding scaling window value of the reference subpicture to determine a scaling ratio. can

For example, the encoding device 10 may determine the scaling ratio as “1” if the resolution of an image is not changed within one group of pictures (GOP). When the resolution of an image is changed in units of GOP, the resolution of a decoded picture and a reference picture may be set to be the same.

Here, the GOP can be applied only when it is in a closed GOP. This is because the Closed GOP is set to have a reference relationship only within the same GOP, and the open GOP is a structure that can refer to pictures in other GOPs, so it is not applied. don't

Subpicture information corresponding to the example in FIG. 7(B) described above may be composed of two different PPS information.

More specifically, the subpicture information includes first subpicture information (PPS #1) for 640x575 subpictures (Nos. 1 and 2) and second subpicture information (Nos. 3 to 6) for 320x288 subpictures. It may be composed of picture information (PPS #2).

9 is an exemplary diagram illustrating a relationship between resolution adjustment for each subpicture and a NAL packet structure according to an embodiment of the present invention.

9 illustrates and describes the structure, signaling order, and reference relationship of a network abstraction layer (NAL) packet including first subpicture information (PPS #1) and second subpicture information (PPS #2). The slice NAL packets for subpictures #1 and #2 refer to the first subpicture information (PPS #1) via PH (picture header), and the slice NAL packets corresponding to subpictures #3 to #6 also refer to PH. Through this, it is possible to refer to the second sub-picture information (PPS #2). Here, a slice may be a data packing unit generating one independent network abstraction layer (NAL) bit-stream, and one subpicture may include one or more slices.

As another example, the above-described resolution (width, height) information, conformance information, and scaling window information of the PPS may be included in the picture header PH, in which case the decoding apparatus 20 ) or the encoding apparatus 10 may configure header information to refer to a picture header corresponding to a slice unit corresponding to a subpicture. Accordingly, the decoding apparatus 20 identifies subpicture resolution and window information included in picture-level header information such as PPS or PH, and changes the resolution and window information of the subpicture corresponding thereto to obtain a subpicture image. It is possible to adaptively change the resolution of .

As another example, only subpicture position and size information may be included in the picture level header. In this case, the decoding apparatus 20 may include at least one of the number, position, or size information of subpictures signaled in the SPS, and the A resolution of a subpicture may be adaptively determined based on picture level header information.

Table 5 below is a diagram illustrating syntax values of the aforementioned first sub-picture information (PPS #1) and second sub-picture information (PPS #2).

PPS #1 PPS#2 pps_pic_width_in_luma_samples 640 320 pps_pic_height_in_luma_samples 576 288 pps_conformance_window_flag - - pps_conf_win_left_offset - - pps_conf_win_right_offset - - pps_conf_win_top_offset - - pps_conf_win_bottom_offset - - pps_scaling_window_explicit_signaling_flag - - pps_signaling_win_left_offset - - pps_signaling_win_right_offset - - pps_signaling_win_top_offset - - pps_siganling_win_bottom_offset - -

As shown in Table 5, in the current embodiment of FIG. 7(B), offset information for the conformance cropping window or offset information for the scaling window is not required. Also, if the resolution of a sub-picture is changed in units of GOP, the resolution of the decoded sub-picture and the reference sub-picture are the same, and the scaling window is the same, so the ratio can be determined to be “1”.

If the resolution of a sub-picture within one GOP is changed from 640x576 to 320x288, the resolution of the current decoded sub-picture and the scaling window become 320x288, and the already reconstructed reference sub-picture (reference sub-picture) picture) and the scaling window is 640x576, so the scaling ratio is 2 (or 1/2) instead of "1" for both the horizontal axis and the vertical axis. It becomes possible to create a reduced or enlarged prediction block.

In addition, according to an embodiment of the present invention, when the size of a face in a cube map structure of an image format is not a multiple of 8 or a multiple of the minimum coding block size (minCB), a conformance window (conformance window) ) information or additional offset information corresponding to scaling window information may be further included in the PPS.

10 is a diagram for explaining a reduction in the number of tiles according to resolution adjustment for each subpicture according to an embodiment of the present invention.

As shown in FIG. 10(A), in the conventional method, an image is divided into tiles and encoded by adjusting bit-rates, and a total of 24 tiles must be encoded.

On the other hand, as shown in FIG. 10(B), in the case of using the method according to the embodiment of the present invention, only 12 tiles in total need to be coded, so it is intuitive that the compression performance is improved by approximately two times based on the number of tiles. You can check.

According to such a configuration, the encoding device 10 according to an embodiment of the present invention can distinguish an important region from an other region in an image, and an important region or a region of interest (ROI) corresponding to a viewport, etc. The amount of compressed data can be minimized by processing to be coded in high resolution for regions and low resolution for other regions.

In addition, the coding apparatus 10 and the decoding apparatus 20 according to an embodiment of the present invention preset signaling information to process such sub-picture adaptive variable resolution coding, and adaptively change the resolution corresponding thereto. By performing the processing, it is possible to provide a process that efficiently and quickly streams or restores images.

The method according to the present invention described above may be produced as a program to be executed on a computer and stored in a computer-readable recording medium. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, and magnetic tape. , floppy disks, and optical data storage devices.

The computer-readable recording medium is distributed to computer systems connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the method can be easily inferred by programmers in the technical field to which the present invention belongs.

In addition, although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those skilled in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

Claims (7)

In the video decoding method,
Receiving a bitstream including image information;
decoding subpicture signaling information from the bitstream;
identifying resolution adjustment information for each subpicture from the subpicture signaling information;
decoding one or more subpictures using the resolution adjustment information for each subpicture; and
Reconstructing a restored image using the decoded subpicture
Video decoding method. According to claim 1,
The step of identifying the resolution adjustment information for each subpicture,
obtaining one or more flags identifying the subpicture from a sequence parameter set of the bitstream; and
Acquiring resolution change information corresponding to the identified subpicture
Video decoding method. According to claim 2,
The resolution change information includes at least one of a resampling permission flag, a resolution change permission flag, and original resolution size information included in the sequence parameter set.
Video decoding method. According to claim 2,
The resolution change information includes resolution compatibility window information of an output picture included in picture level header information corresponding to the identified subpicture.
Video decoding method. According to claim 2,
The resolution change information includes resolution scaling window information of a reconstructed picture included in picture level header information corresponding to the identified subpicture.
Video decoding method. According to claim 1,
At least some of the resolution change information is included in a picture parameter set of a network abstraction layer (NAL), and the resolution change information is determined based on picture header information referenced by the at least some of the information.
Video decoding method. In the video encoding method,
Dividing an original image into sub-pictures;
determining a coding resolution for each divided sub-picture;
encoding an original image according to the encoding resolution for each sub-picture;
generating the subpicture signaling information including resolution adjustment information for each subpicture; and
Constructing a bitstream including an encoded image and the subpicture signaling information
Video encoding method.

Images