KR20230168602A - Method and apparatus for encoding/decoding image and recording medium for storing bitstream - Google Patents

Abstract

In the present invention, padding a motion compensation padding area within a first padding distance from the border of the current picture according to the motion vector of a border block adjacent to the border of the current picture, and padding a motion compensation padding area within a second padding distance from the border of the motion compensation padding area. An image decoding method comprising the steps of padding a repetitive padding area according to pixel values adjacent to the boundary of the motion compensation padding area, and storing an extended picture consisting of the current picture, the motion compensation padding area, and the repetitive padding area in a memory. provided.

Description

Video encoding/decoding method, device, and recording medium storing bitstream {METHOD AND APPARATUS FOR ENCODING/DECODING IMAGE AND RECORDING MEDIUM FOR STORING BITSTREAM}

The present invention relates to a video encoding/decoding method, device, and recording medium storing bitstreams. Specifically, the present invention relates to a video encoding/decoding method and device using an inter-screen prediction method, and a recording medium storing a bitstream.

Recently, demand for high-resolution, high-quality images such as UHD (Ultra High Definition) images is increasing in various application fields. As video data becomes higher resolution and higher quality, the amount of data increases relative to existing video data. Therefore, when video data is transmitted using media such as existing wired or wireless broadband lines or stored using existing storage media, transmission costs and Storage costs increase. In order to solve these problems that arise as image data becomes higher resolution and higher quality, highly efficient image encoding/decoding technology for images with higher resolution and quality is required.

In inter-prediction of a picture, the current picture is predicted with reference to a reference picture. At this time, when the current picture refers to outside the boundary of the reference picture, various methods are being discussed to improve the prediction accuracy of the current picture. For example, a method of extending a reference picture by padding the area outside the reference picture boundary is being discussed. And in bidirectional inter-screen prediction, when the current block refers to the padding area of the extended reference picture, various methods are being discussed to increase prediction accuracy.

The purpose of the present invention is to provide a video encoding/decoding method and device with improved encoding/decoding efficiency.

Another object of the present invention is to provide a recording medium that stores a bitstream generated by the video decoding method or device provided by the present invention.

An image decoding method according to an embodiment of the present invention includes padding a motion compensation padding area within a first padding distance from the border of the current picture according to a motion vector of a boundary block adjacent to the border of the current picture, the motion compensation padding padding a repetitive padding area within a second padding distance from the boundary of the area according to a pixel value adjacent to the boundary of the motion compensation padding area, and an expansion consisting of the current picture, the motion compensation padding area, and the repetitive padding area. It may include storing the picture in memory.

According to one embodiment, the step of padding the motion compensation padding area includes extracting a motion vector from the boundary block, and based on the motion vector, extracting a reference block from a motion compensation padding reference picture that is referenced to the motion compensation padding. It may include determining, determining a motion compensation padding reference block adjacent to the reference block in a padding direction, and padding the motion compensation padding area based on the motion compensation padding reference block.

According to one embodiment, in determining the motion compensation padding reference block, the motion compensation padding reference block is determined to include adjacent pixels within the first padding distance from a boundary of the reference block in the padding direction. It can be characterized.

According to one embodiment, in determining the motion compensation padding reference block, if the motion compensation padding reference possible distance between the boundary of the reference block and the boundary of the motion compensation padding reference picture is smaller than the first padding distance, The motion compensation padding reference block may be determined to include adjacent pixels within the motion compensation padding reference distance from the boundary of the reference block in the padding direction.

According to one embodiment, in the step of padding the motion compensation padding area based on the motion compensation padding reference block, the motion compensation padding reference possible distance between the boundary of the reference block and the boundary of the motion compensation padding reference picture is If it is smaller than the first padding distance, a motion compensation padding area within the motion compensation padding reference distance from the boundary of the current picture among the motion compensation padding areas is padded by the motion compensation padding reference block, and the motion compensation padding area is Among the padding areas, an area that is not a motion compensation padding area may be padded based on the pixel value of the motion compensation padding area.

According to one embodiment, the image decoding method may be characterized in that, when a motion vector cannot be extracted from the boundary block, the motion compensation padding area is padded according to pixel values adjacent to the boundary of the current picture. .

According to one embodiment, the step of extracting a motion vector from the border block includes, when a motion vector cannot be extracted from the border block, a temporal neighborhood corresponding to the position of the border block from the temporal corresponding reference picture of the current picture. It may be characterized by including determining a block, and extracting a motion vector from the temporal neighboring block.

According to one embodiment, the first padding distance may be determined based on at least one of the maximum size of the coding unit, the size of the current picture, and the size of the boundary block.

According to one embodiment, the first padding distance may be determined as one of 2, 4, 8, 16, 32, 64, 128, and 256.

According to one embodiment, the second padding distance may be determined based on at least one of the maximum size of the coding unit, the size of the current picture, and the size of the boundary block.

According to one embodiment, the size of the extended picture may be determined based on the size of the current picture, regardless of the value of the first padding distance.

An image decoding method according to another embodiment of the present invention includes determining a first reference picture and a second reference picture referenced by the current picture in order to predict the current block of the current picture, and performing a first motion of the current block. determining a first reference block and a second reference block, respectively, from the first reference picture and the second reference picture according to the vector and the second motion vector, and all pixels of the first reference block are The current block is selected using at least one of the first reference block and the second reference block depending on whether all pixels of the second reference block are included in the reference picture. It may include a prediction step.

According to one embodiment, in predicting the current block, some or all of the pixels of the first reference block are not included in the first reference picture, and pixels of the second reference block are When fully included in a reference picture, the current block may be predicted based on the second reference block.

According to one embodiment, when some of the pixels of the first reference block are not included in the first reference picture and all pixels of the second reference block are included in the second reference picture, the first reference The first area of the current block corresponding to the position of pixels included in the first reference picture in the block is determined as a weighted average of the pixels of the first reference block and the second reference block, and in the first reference block The second area of the current block corresponding to the positions of pixels not included in the first reference picture may be predicted based only on the second reference block.

According to one embodiment, in determining whether all pixels of the first reference block are entirely included in the first reference picture and whether all pixels of the second reference block are entirely included in the second reference picture, The first motion compensation padding area of the first reference picture may be regarded as the first reference picture, and the second motion compensation padding area of the second reference picture may be regarded as the second reference picture.

An image encoding method according to an embodiment of the present invention includes padding a motion compensation padding area within a first padding distance from the border of the current picture according to a motion vector of a boundary block adjacent to the border of the current picture, the motion compensation padding padding a repetitive padding area within a second padding distance from the boundary of the area according to a pixel value adjacent to the boundary of the motion compensation padding area, and an expansion consisting of the current picture, the motion compensation padding area, and the repetitive padding area. A step may include storing the picture in memory.

An image encoding method according to another embodiment of the present invention includes determining a first reference picture and a second reference picture referenced by the current picture in order to predict the current block of the current picture, and performing a first motion of the current block. determining a first reference block and a second reference block, respectively, from the first reference picture and the second reference picture according to the vector and the second motion vector, and all pixels of the first reference block are The current block is selected using at least one of the first reference block and the second reference block depending on whether all pixels of the second reference block are included in the reference picture. It may include a prediction step.

A non-transitory computer-readable recording medium according to an embodiment of the present invention stores a bitstream generated by the image encoding method.

The transmission method according to an embodiment of the present invention transmits a bitstream generated by the video encoding method.

The features briefly summarized above with respect to the present invention are merely exemplary aspects of the detailed description of the present invention that follows, and do not limit the scope of the present invention.

The present invention proposes various embodiments of a method for generating an extended picture including a motion compensation padding area to increase the prediction accuracy of inter-screen prediction.

Additionally, in order to increase the prediction accuracy of inter-screen prediction, the present invention proposes various embodiments of an efficient bi-directional inter-prediction method when all or part of a reference block is located in a repetitive padding area or a motion compensation padding area.

In the present invention, as the accuracy of inter-screen prediction improves, overall coding efficiency can be improved.

1 is a block diagram showing the configuration of an encoding device to which the present invention is applied according to an embodiment.
Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment to which the present invention is applied.
Figure 3 is a diagram schematically showing a video coding system to which the present invention can be applied.
Figure 4 shows an example of an extended picture generated according to an iterative padding method.
Figure 5 shows an example of an extended picture generated according to repetitive padding and motion compensation padding.
Figure 6 shows an example of a method for determining a motion compensation padding area.
Figure 7 shows an example of an extended picture in which an extended picture area is determined regardless of the size of the motion compensation padding area.
Figure 8 shows an example of a method for performing motion compensation padding based on temporal neighboring blocks.
Figure 9 shows an example of a block prediction method using bidirectional motion prediction.
10 and 11 show an example of a block prediction method using bidirectional motion prediction when a part of a reference block is located outside the reference picture.
Figure 12 shows an example of a block prediction method using bidirectional motion prediction when an extended picture has a motion compensation padding area.
Figure 13 shows an example of an image decoding method according to the present invention.
Figure 14 shows an example of an image encoding method according to the present invention.
Figure 15 exemplarily shows a content streaming system to which an embodiment according to the present invention can be applied.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. Similar reference numbers in the drawings refer to identical or similar functions across various aspects. The shapes and sizes of elements in the drawings may be provided as examples for clearer explanation. For a detailed description of the exemplary embodiments described below, refer to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description that follows is not to be taken in a limiting sense, and the scope of the exemplary embodiments is limited only by the appended claims, together with all equivalents to what those claims assert if properly described.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

The components appearing in the embodiments of the present invention are shown independently to represent different characteristic functions, and do not mean that each component is comprised of separate hardware or a single software component. That is, each component is listed and included as a separate component for convenience of explanation, and at least two of each component can be combined to form one component, or one component can be divided into a plurality of components to perform a function, and each of these components can perform a function. Integrated embodiments and separate embodiments of the constituent parts are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

The terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. Additionally, some of the components of the present invention may not be essential components that perform essential functions in the present invention, but may be merely optional components to improve performance. The present invention can be implemented by including only essential components for implementing the essence of the present invention excluding components used only to improve performance, and a structure including only essential components excluding optional components used only to improve performance. is also included in the scope of rights of the present invention.

In embodiments, the term “at least one” may mean one of numbers greater than 1, such as 1, 2, 3, and 4. In embodiments, the term “a plurality of” may mean one of two or more numbers, such as 2, 3, and 4.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing the embodiments of the present specification, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description will be omitted, and the same reference numerals will be used for the same components in the drawings. Redundant descriptions of the same components are omitted.

Glossary of Terms

Hereinafter, “video” may refer to a single picture that constitutes a video, or may refer to the video itself. For example, “encoding and/or decoding of a video” may mean “encoding and/or decoding of a video,” or “encoding and/or decoding of one of the videos that make up a video.” It may be possible.

Hereinafter, “movie” and “video” may be used with the same meaning and may be used interchangeably. Additionally, the target image may be an encoding target image that is the target of encoding and/or a decoding target image that is the target of decoding. Additionally, the target image may be an input image input to an encoding device or may be an input image input to a decoding device. Here, the target image may have the same meaning as the current image.

Hereinafter, “image,” “picture,” “frame,” and “screen” may be used with the same meaning and may be used interchangeably.

Hereinafter, the “target block” may be an encoding target block that is the target of encoding and/or a decoding target block that is the target of decoding. Additionally, the target block may be a current block that is currently the target of encoding and/or decoding. For example, “target block” and “current block” may be used with the same meaning and may be used interchangeably.

Hereinafter, “block” and “unit” may be used with the same meaning and may be used interchangeably. Additionally, “unit” may mean including a luminance (Luma) component block and a corresponding chroma component block in order to refer to it separately from a block. As an example, a Coding Tree Unit (CTU) may be composed of two chrominance component (Cb, Cr) coding tree blocks related to one luminance component (Y) coding tree block (CTB). .

Hereinafter, “sample,” “pixel,” and “pixel” may be used with the same meaning and may be used interchangeably. Here, the sample may represent the basic unit constituting the block.

Hereinafter, “inter” and “between screens” may be used with the same meaning and may be used interchangeably.

Hereinafter, “intra” and “within the screen” may be used with the same meaning and may be used interchangeably.

1 is a block diagram showing the configuration of an encoding device to which the present invention is applied according to an embodiment.

The encoding device 100 may be an encoder, a video encoding device, or an image encoding device. A video may contain one or more images. The encoding device 100 can sequentially encode one or more images.

Referring to FIG. 1, the encoding device 100 includes an image segmentation unit 110, an intra prediction unit 120, a motion prediction unit 121, a motion compensation unit 122, a switch 115, a subtractor 113, A transform unit 130, a quantization unit 140, an entropy encoding unit 150, an inverse quantization unit 160, an inverse transform unit 170, an adder 117, a filter unit 180, and a reference picture buffer 190. It can be included.

Additionally, the encoding device 100 can generate a bitstream including encoded information through encoding of an input image and output the generated bitstream. The generated bitstream can be stored in a computer-readable recording medium or streamed through wired/wireless transmission media.

The image segmentation unit 110 may divide the input image into various forms to increase the efficiency of video encoding/decoding. In other words, the input video consists of multiple pictures, and one picture can be hierarchically divided and processed for compression efficiency, parallel processing, etc. For example, one picture can be divided into one or multiple tiles or slices and further divided into multiple CTUs (Coding Tree Units). In another method, one picture may first be divided into a plurality of sub-pictures defined as a group of rectangular slices, and each sub-picture may be divided into the tiles/slices. Here, subpictures can be used to support the function of partially independently encoding/decoding and transmitting a picture. Since multiple subpictures can be restored individually, it has the advantage of being easy to edit in applications where multi-channel input is composed of one picture. Additionally, bricks can be created by dividing tiles horizontally. Here, a brick can be used as a basic unit of intra-picture parallel processing. Additionally, one CTU can be recursively divided into a quad tree (QT: Quadtree), and the end node of the division can be defined as a CU (Coding Unit). CU can be divided into PU (Prediction Unit), which is a prediction unit, and TU (Transform Unit), which is a transformation unit, and prediction and division can be performed. Meanwhile, CUs can be used as prediction units and/or transformation units themselves. Here, for flexible partitioning, each CTU may be recursively partitioned into not only a quad tree (QT) but also a multi-type tree (MTT). CTU can begin to be divided into a multi-type tree from the end node of QT, and MTT can be composed of BT (Binary Tree) and TT (Triple Tree). For example, the MTT structure can be divided into vertical binary split mode (SPLIT_BT_VER), horizontal binary split mode (SPLIT_BT_HOR), vertical ternary split mode (SPLIT_TT_VER), and horizontal ternary split mode (SPLIT_TT_HOR). In addition, when dividing, the minimum block size (MinQTSize) of the quad tree of the luminance block can be set to 16x16, the maximum block size (MaxBtSize) of the binary tree can be set to 128x128, and the maximum block size (MaxTtSize) of the triple tree can be set to 64x64. Additionally, the minimum block size (MinBtSize) of the binary tree and the minimum block size (MinTtSize) of the triple tree can be set to 4x4, and the maximum depth (MaxMttDepth) of the multi-type tree can be set to 4. Additionally, in order to increase the coding efficiency of the I slice, a dual tree that uses different CTU division structures for the luminance and chrominance components can be applied. On the other hand, in P and B slices, the luminance and chrominance CTB (Coding Tree Blocks) within the CTU can be divided into a single tree that shares the coding tree structure.

The encoding device 100 may perform encoding on an input image in intra mode and/or inter mode. Alternatively, the encoding device 100 may perform encoding on the input image in a third mode (eg, IBC mode, Palette mode, etc.) other than the intra mode and inter mode. However, if the third mode has similar functional characteristics to intra mode or inter mode, it may be classified as intra mode or inter mode for convenience of explanation. In the present invention, the third mode will be classified and described separately only when a detailed explanation is needed.

When the intra mode is used as the prediction mode, the switch 115 may be switched to the intra mode, and when the inter mode is used as the prediction mode, the switch 115 may be switched to the inter mode. Here, intra mode may mean intra prediction mode, and inter mode may mean inter-screen prediction mode. The encoding device 100 may generate a prediction block for an input block of an input image. Additionally, after the prediction block is generated, the encoding device 100 may encode the residual block using the residual of the input block and the prediction block. The input image may be referred to as the current image that is currently the target of encoding. The input block may be referred to as the current block that is currently the target of encoding or the encoding target block.

When the prediction mode is intra mode, the intra prediction unit 120 may use samples of blocks that have already been encoded/decoded around the current block as reference samples. The intra prediction unit 120 may perform spatial prediction for the current block using a reference sample and generate prediction samples for the input block through spatial prediction. Here, intra prediction may mean prediction within the screen.

As an intra prediction method, non-directional prediction modes such as DC mode and Planar mode and directional prediction modes (e.g., 65 directions) can be applied. Here, the intra prediction method may be expressed as an intra prediction mode or an intra prediction mode.

When the prediction mode is inter mode, the motion prediction unit 121 can search for the area that best matches the input block from the reference image during the motion prediction process and derive a motion vector using the searched area. . At this time, the search area can be used as the area. The reference image may be stored in the reference picture buffer 190. Here, when encoding/decoding of the reference image is processed, it may be stored in the reference picture buffer 190.

The motion compensation unit 122 may generate a prediction block for the current block by performing motion compensation using a motion vector. Here, inter prediction may mean inter-screen prediction or motion compensation.

When the motion vector value does not have an integer value, the motion prediction unit 121 and the motion compensation unit 122 can generate a prediction block by applying an interpolation filter to some areas in the reference image. . To perform inter-screen prediction or motion compensation, the motion prediction and motion compensation methods of the prediction unit included in the coding unit based on the coding unit include skip mode, merge mode, and improved motion vector prediction ( It is possible to determine whether it is in Advanced Motion Vector Prediction (AMVP) mode or Intra Block Copy (IBC) mode, and inter-screen prediction or motion compensation can be performed depending on each mode.

In addition, based on the inter-screen prediction method, AFFINE mode of sub-PU-based prediction, Subblock-based Temporal Motion Vector Prediction (SbTMVP) mode, and Merge with MVD (MMVD) mode of PU-based prediction, Geometric Partitioning Mode (GPM) ) mode can also be applied. In addition, to improve the performance of each mode, HMVP (History based MVP), PAMVP (Pairwise Average MVP), CIIP (Combined Intra/Inter Prediction), AMVR (Adaptive Motion Vector Resolution), BDOF (Bi-Directional Optical-Flow), Bi-predictive with CU Weights (BCW), Local Illumination Compensation (LIC), Template Matching (TM), and Overlapped Block Motion Compensation (OBMC) can also be applied.

The subtractor 113 may generate a residual block using the difference between the input block and the prediction block. The residual block may also be referred to as a residual signal. The residual signal may refer to the difference between the original signal and the predicted signal. Alternatively, the residual signal may be a signal generated by transforming, quantizing, or transforming and quantizing the difference between the original signal and the predicted signal. The remaining block may be a residual signal in block units.

The transform unit 130 may generate a transform coefficient by performing transformation on the remaining block and output the generated transform coefficient. Here, the transformation coefficient may be a coefficient value generated by performing transformation on the remaining block. When the transform skip mode is applied, the transform unit 130 may skip transforming the remaining blocks.

Quantized levels can be generated by applying quantization to the transform coefficients or residual signals. Hereinafter, in embodiments, the quantized level may also be referred to as a transform coefficient.

As an example, the 4x4 luminance residual block generated through intra-screen prediction is transformed using a DST (Discrete Sine Transform)-based basis vector, and the remaining residual blocks are transformed using a DCT (Discrete Cosine Transform)-based basis vector. can do. In addition, through RQT (Residual Quad Tree) technology, the transform block for one block is divided into a quad tree form, and after performing transformation and quantization on each transform block divided through RQT, when all coefficients become 0, To increase coding efficiency, cbf (coded block flag) can be transmitted.

As another alternative, MTS (Multiple Transform Selection) technology, which performs transformation by selectively using multiple transformation bases, can be applied. In other words, instead of dividing CUs into TUs through RQT, a similar function to TU division can be performed through SBT (Sub-block Transform) technology. Specifically, SBT is applied only to inter-screen prediction blocks, and unlike RQT, it can divide the current block into ½ or ¼ sizes vertically or horizontally and then perform transformation on only one of the blocks. For example, when split vertically, transformation can be performed on the leftmost or rightmost block, and when divided horizontally, transformation can be performed on the top or bottom block.

In addition, LFNST (Low Frequency Non-Separable Transform), a secondary transform technology that further transforms the residual signal converted to the frequency domain through DCT or DST, can be applied. LFNST additionally performs transformation on the 4x4 or 8x8 low-frequency area in the upper left corner, allowing the residual coefficients to be concentrated in the upper left corner.

The quantization unit 140 may generate a quantized level by quantizing a transform coefficient or a residual signal according to a quantization parameter (QP), and output the generated quantized level. At this time, the quantization unit 140 may quantize the transform coefficient using a quantization matrix.

As an example, a quantizer using QP values of 0 to 51 can be used. Alternatively, if the image size is larger and high coding efficiency is required, 0 to 63 QP can be used. Additionally, a DQ (Dependent Quantization) method that uses two quantizers instead of one quantizer can be applied. DQ performs quantization using two quantizers (e.g., Q0, Q1), but even without signaling information about the use of a specific quantizer, the quantizer to be used for the next transformation coefficient is determined based on the current state through a state transition model. It can be applied to be selected.

The entropy encoding unit 150 can generate a bitstream by performing entropy encoding according to a probability distribution on the values calculated by the quantization unit 140 or the coding parameter values calculated during the encoding process. and bitstream can be output. The entropy encoding unit 150 may perform entropy encoding on information about image samples and information for decoding the image. For example, information for decoding an image may include syntax elements, etc.

When entropy coding is applied, a small number of bits are allocated to symbols with a high probability of occurrence and a large number of bits are allocated to symbols with a low probability of occurrence to represent symbols, so that the bits for the symbols to be encoded are expressed. The size of the column may be reduced. The entropy encoding unit 150 may use encoding methods such as exponential Golomb, CAVLC (Context-Adaptive Variable Length Coding), and CABAC (Context-Adaptive Binary Arithmetic Coding) for entropy encoding. For example, the entropy encoding unit 150 may perform entropy encoding using a Variable Length Coding/Code (VLC) table. In addition, the entropy encoding unit 150 derives a binarization method of the target symbol and a probability model of the target symbol/bin, and then uses the derived binarization method, probability model, and context model. Arithmetic coding can also be performed using .

Relatedly, when applying CABAC, in order to reduce the size of the probability table stored in the decoding device, the table probability update method may be changed to a table update method using a simple formula. Additionally, two different probability models can be used to obtain more accurate symbol probability values.

The entropy encoder 150 can change a two-dimensional block form coefficient into a one-dimensional vector form through a transform coefficient scanning method to encode the transform coefficient level (quantized level).

Coding parameters include information (flags, indexes, etc.) encoded in the encoding device 100 and signaled to the decoding device 200, such as syntax elements, as well as information derived from the encoding or decoding process. It may include and may mean information needed when encoding or decoding an image.

Here, signaling a flag or index may mean that the encoder entropy encodes the flag or index and includes it in the bitstream, and the decoder may include the flag or index from the bitstream. This may mean entropy decoding.

The encoded current image can be used as a reference image for other images to be processed later. Accordingly, the encoding device 100 can restore or decode the current encoded image, and store the restored or decoded image as a reference image in the reference picture buffer 190.

The quantized level may be dequantized in the dequantization unit 160. It may be inverse transformed in the inverse transform unit 170. The inverse-quantized and/or inverse-transformed coefficients may be combined with the prediction block through the adder 117. A reconstructed block may be generated by combining the inverse-quantized and/or inverse-transformed coefficients with the prediction block. Here, the inverse-quantized and/or inverse-transformed coefficient refers to a coefficient on which at least one of inverse-quantization and inverse-transformation has been performed, and may refer to a restored residual block. The inverse quantization unit 160 and the inverse transform unit 170 may be performed as reverse processes of the quantization unit 140 and the transform unit 130.

The restored block may pass through the filter unit 180. The filter unit 180 includes a deblocking filter, a sample adaptive offset (SAO), an adaptive loop filter (ALF), a bilateral filter (BIF), and an LMCS (Luma). Mapping with Chroma Scaling) can be applied to restored samples, restored blocks, or restored images as all or part of the filtering techniques. The filter unit 180 may also be referred to as an in-loop filter. At this time, in-loop filter is also used as a name excluding LMCS.

The deblocking filter can remove block distortion occurring at the boundaries between blocks. To determine whether to perform a deblocking filter, it is possible to determine whether to apply a deblocking filter to the current block based on the samples included in a few columns or rows included in the block. When applying a deblocking filter to a block, different filters can be applied depending on the required deblocking filtering strength.

Using sample adaptive offset, an appropriate offset value can be added to the sample value to compensate for the encoding error. Sample adaptive offset can correct the offset of the deblocked image with the original image on a sample basis. You can use a method of dividing the samples included in the image into a certain number of regions, then determining the region to perform offset and applying the offset to that region, or a method of applying the offset by considering the edge information of each sample.

Bilateral filter (BIF) can also correct the offset from the original image on a sample basis for the deblocked image.

The adaptive loop filter can perform filtering based on a comparison value between the restored image and the original image. After dividing the samples included in the video into predetermined groups, filtering can be performed differentially for each group by determining the filter to be applied to that group. Information related to whether to apply an adaptive loop filter may be signaled for each coding unit (CU), and the shape and filter coefficients of the adaptive loop filter to be applied may vary for each block.

In LMCS (Luma Mapping with Chroma Scaling), luma-mapping (LM) refers to remapping luminance values through a piece-wise linear model, and chroma scaling (CS) refers to the average of the predicted signal. This refers to a technology that scales the residual value of the color difference component according to the luminance value. In particular, LMCS can be used as an HDR correction technology that reflects the characteristics of HDR (High Dynamic Range) images.

The reconstructed block or reconstructed image that has passed through the filter unit 180 may be stored in the reference picture buffer 190. The restored block that has passed through the filter unit 180 may be part of a reference image. In other words, the reference image may be a reconstructed image composed of reconstructed blocks that have passed through the filter unit 180. The stored reference image can then be used for inter-screen prediction or motion compensation.

Figure 2 is a block diagram showing the configuration of a decoding device according to an embodiment to which the present invention is applied.

The decoding device 200 may be a decoder, a video decoding device, or an image decoding device.

Referring to FIG. 2, the decoding device 200 includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra prediction unit 240, a motion compensation unit 250, and an adder 201. , it may include a switch 203, a filter unit 260, and a reference picture buffer 270.

The decoding device 200 may receive the bitstream output from the encoding device 100. The decoding device 200 may receive a bitstream stored in a computer-readable recording medium or receive a bitstream streamed through a wired/wireless transmission medium. The decoding device 200 may perform decoding on a bitstream in intra mode or inter mode. Additionally, the decoding device 200 can generate a restored image or a decoded image through decoding, and output the restored image or a decoded image.

If the prediction mode used for decoding is intra mode, the switch 203 may be switched to intra mode. If the prediction mode used for decoding is the inter mode, the switch 203 may be switched to inter.

The decoding device 200 can decode the input bitstream to obtain a reconstructed residual block and generate a prediction block. When the reconstructed residual block and the prediction block are obtained, the decoding device 200 may generate a restored block to be decoded by adding the restored residual block and the prediction block. The block to be decrypted may be referred to as the current block.

The entropy decoding unit 210 may generate symbols by performing entropy decoding according to a probability distribution for the bitstream. The generated symbols may include symbols in the form of quantized levels. Here, the entropy decoding method may be the reverse process of the entropy encoding method described above.

The entropy decoder 210 can change one-dimensional vector form coefficients into two-dimensional block form through a transform coefficient scanning method in order to decode the transform coefficient level (quantized level).

The quantized level may be inversely quantized in the inverse quantization unit 220 and inversely transformed in the inverse transformation unit 230. The quantized level may be generated as a restored residual block as a result of performing inverse quantization and/or inverse transformation. At this time, the inverse quantization unit 220 may apply the quantization matrix to the quantized level. The inverse quantization unit 220 and the inverse transform unit 230 applied to the decoding device may use the same technology as the inverse quantization unit 160 and the inverse transform section 170 applied to the above-described encoding device.

When the intra mode is used, the intra prediction unit 240 may generate a prediction block by performing spatial prediction on the current block using sample values of already decoded blocks surrounding the decoding target block. The intra prediction unit 240 applied to the decoding device may use the same technology as the intra prediction unit 120 applied to the above-described encoding device.

When inter mode is used, the motion compensation unit 250 may generate a prediction block by performing motion compensation on the current block using a motion vector and a reference image stored in the reference picture buffer 270. When the motion vector value does not have an integer value, the motion compensator 250 may generate a prediction block by applying an interpolation filter to a partial area in the reference image. To perform motion compensation, based on the coding unit, it can be determined whether the motion compensation method of the prediction unit included in the coding unit is skip mode, merge mode, AMVP mode, or current picture reference mode, and each mode Motion compensation can be performed according to . The motion compensation unit 250 applied to the decoding device may use the same technology as the motion compensation unit 122 applied to the above-described encoding device.

The adder 201 may generate a restored block by adding the restored residual block and the prediction block. The filter unit 260 may apply at least one of inverse-LMCS, deblocking filter, sample adaptive offset, and adaptive loop filter to the reconstructed block or reconstructed image. The filter unit 260 applied to the decoding device may apply the same filtering technology as the filtering technology applied to the filter unit 180 applied to the above-described encoding device.

The filter unit 260 may output a restored image. The reconstructed block or reconstructed image may be stored in the reference picture buffer 270 and used for inter prediction. The restored block that has passed through the filter unit 260 may be part of a reference image. In other words, the reference image may be a reconstructed image composed of reconstructed blocks that have passed through the filter unit 260. The stored reference image can then be used for inter-screen prediction or motion compensation.

Figure 3 is a diagram schematically showing a video coding system to which the present invention can be applied.

A video coding system according to an embodiment may include an encoding device 10 and a decoding device 20. The encoding device 10 may transmit encoded video and/or image information or data in file or streaming form to the decoding device 20 through a digital storage medium or network.

The encoding device 10 according to an embodiment may include a video source generator 11, an encoder 12, and a transmitter 13. The decoding device 20 according to one embodiment may include a receiving unit 21, a decoding unit 22, and a rendering unit 23. The encoder 12 may be called a video/image encoder, and the decoder 22 may be called a video/image decoder. The transmission unit 13 may be included in the encoding unit 12. The receiving unit 21 may be included in the decoding unit 22. The rendering unit 23 may include a display unit, and the display unit may be composed of a separate device or external component.

The video source generator 11 may acquire video/image through a video/image capture, synthesis, or creation process. The video source generator 11 may include a video/image capture device and/or a video/image generation device. A video/image capture device may include, for example, one or more cameras, a video/image archive containing previously captured video/images, etc. Video/image generating devices may include, for example, computers, tablets, and smartphones, and are capable of generating video/images (electronically). For example, a virtual video/image may be created through a computer, etc., and in this case, the video/image capture process may be replaced by the process of generating related data.

The encoder 12 can encode the input video/image. The encoder 12 can perform a series of procedures such as prediction, transformation, and quantization for compression and encoding efficiency. The encoder 12 may output encoded data (encoded video/image information) in the form of a bitstream. The detailed configuration of the encoding unit 12 may be the same as that of the encoding device 100 of FIG. 1 described above.

The transmission unit 13 may transmit encoded video/image information or data output in the form of a bitstream to the reception unit 21 of the decoding device 20 through a digital storage medium or network in the form of a file or streaming. Digital storage media may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit 13 may include elements for creating a media file through a predetermined file format and may include elements for transmission through a broadcasting/communication network. The receiving unit 21 may extract/receive the bitstream from the storage medium or network and transmit it to the decoding unit 22.

The decoder 22 can decode the video/image by performing a series of procedures such as inverse quantization, inverse transformation, and prediction corresponding to the operations of the encoder 12. The detailed configuration of the decoding unit 22 may be the same as that of the decoding device 200 of FIG. 2 described above.

The rendering unit 23 may render the decrypted video/image. The rendered video/image may be displayed through the display unit.

In the present invention, a border area padding technology is presented that expands a reference picture using a padding method in the border area when compensating for motion through motion prediction, and then generates an inter-screen prediction block based on this. The border area padding method may place a burden on the decoder due to its high complexity and large memory usage. Therefore, a method for padding the border area with low decoder complexity and low memory usage is proposed. Additionally, in bidirectional motion compensation, various embodiments of a motion compensation method when a padded boundary area is referenced are proposed.

Hereinafter, Figure 4 explains a method of determining an extended picture according to an iterative padding method. And Figures 5 to 8 explain a method of determining an extended picture according to the motion compensation padding method and the iterative padding method.

Figure 4 shows an example of an extended picture generated according to an iterative padding method.

Figure 4 shows an extended picture 400 consisting of a reference picture 402 and an extended picture area 404. Here, the extended picture area 404 refers to an area extended by a predetermined padding distance from the border of the reference picture 402. The predetermined padding distance may be determined according to the maximum width of the coding unit and/or the maximum height of the coding unit. Alternatively, the predetermined padding distance may be determined to be a predetermined value larger than the maximum width of the coding unit or the maximum height of the coding unit. For example, as shown in FIG. 4, when the maximum width of the coding unit is maxCUwidth, the predetermined padding size may be (maxCUwidth + 16).

The extended picture area 404 in FIG. 4 consists only of a repetitive padding area. A repetitive padding area is created by repeatedly padding pixels located at the border of a reference picture. Accordingly, all pixels created by padding based on one pixel of the boundary of the reference picture 402 in the extended picture area 404 of FIG. 4 have the same value. For example, the values of pixels 408 of the extended picture area 404 generated from pixel 406 of the reference picture 402 may all be determined to be the same as the value of pixel 406 of the reference picture 402. there is.

Figure 5 shows an example of an extended picture generated according to repetitive padding and motion compensation padding.

The extended picture 500 in FIG. 5 is composed of a reference picture 502 and an extended picture area. And the extended picture area is composed of a repetitive padding area (504) and a motion compensation padding area (MC padding area) (506).

The pixels of the motion compensation padding area 506 are determined by motion compensation using the motion vector of the boundary block of the reference picture 502. The method for determining the motion compensation padding area 506 is described in detail in FIG. 6. The area of the motion compensation padding area 506 may be determined by the first padding distance applied to the motion compensation padding area 506. For example, the area of the motion compensation padding area 506 is the product of the width of the reference picture 502 (picture_width) and the first padding distance and the product of the height of the reference picture 502 (picture_height) and the first padding distance. It can be decided by

The first padding distance may be determined by the height or width of the reference picture 502. Alternatively, the first padding distance may be a fixed value. For example, as shown in FIG. 5, the predetermined value may be 64. Therefore, the size of the motion compensation padding area adjacent to the upper and lower boundaries of the reference picture may be (picture_width x 64), and the size of the motion compensation padding area adjacent to the left and right boundaries of the reference picture may be (64 x picture_height). there is. Depending on the embodiment, a first padding distance applied to the motion compensation padding areas 506 adjacent to the left and right sides of the reference picture 502 and a first padding distance applied to the motion compensation padding areas 506 adjacent to the upper and lower sides of the reference picture 502 The applied first padding distance may be different.

The repetitive padding area 504 is created by repeatedly padding pixels located on the border of the reference picture 502 or the motion compensation padding area 506 by a second padding distance. The second padding distance may be determined according to the maximum width of the coding unit and/or the maximum height of the coding unit. Alternatively, the second padding distance may be determined to be a predetermined value larger than the maximum width of the coding unit or the maximum height of the coding unit. For example, as shown in FIG. 5, when the maximum width of the coding unit is maxCUwidth, the predetermined second padding distance may be (maxCUwidth + 16).

The motion compensation padding area 506 can be created when certain conditions are met. For example, the motion compensation padding area 506 may be determined according to the slice type of the current picture 502. If the slice type of the current picture 502 is an I slice, the motion compensation padding area 506 may not be created, and if the slice type of the current picture 502 is a P or B slice, the motion compensation padding area 506 may be created.

Figure 6 shows an example of a method for determining a motion compensation padding area.

The motion compensation padding area 604 of the current picture 600 is determined using a motion vector derived from a boundary block 602 of a predetermined size located at the border of the current picture 600. Using the motion vector, a reference block 612 corresponding to the boundary block 602 is determined from the reference picture 610 referenced for reconstruction of the current picture. Once the reference block 612 is determined, the motion compensation padding block 608 is derived from the motion compensation padding reference block 614 to the left of the reference block 612.

In FIG. 6, in order to determine the left portion of the motion compensation padding area 604 of the current picture 600, the motion compensation padding reference block 614 to the left of the reference block 612 was referenced. However, when determination of the upper, lower, or right motion compensation padding area is required, the block above, lower, or right of the reference block of the reference picture 610 may be referenced.

The size of the border block 602 may be set differently depending on the embodiment. In Figure 6, the size of the border block 602 is set to 4x4. Also, depending on the embodiment, the size of the border block 602 may be set to the minimum size of the block determined by the encoder or decoder.

The size of the motion compensation padding block 608 is the same as the size of the motion compensation padding reference block 614 to the left of the reference block 612. In FIG. 6, only pixels within M pixel distance from the left of the reference block 612 are available, so the size of the motion compensation padding reference block 614 is set to Mx4. Accordingly, the size of the motion compensation padding reference block 614 may be determined depending on how far pixels from the left of the reference block 612 are available.

However, the width of the motion compensation padding block 608 cannot be larger than the width of the left portion of the motion compensation padding area 604. For example, when the motion compensation padding area is determined to be an area within a 64 pixel distance from the reference picture as shown in FIG. 5, the width of the motion compensation padding block 608 is determined to be equal to or smaller than 64.

As on the left, the motion compensation padding reference block in the motion compensation padding area on the right can be set to Mx4. And the motion compensation padding reference blocks in the upper and lower motion compensation padding areas can be set to 4xM.

If the border block 602 is restored according to bidirectional prediction, two motion vectors can be derived from the border block 602. Therefore, from two motion vectors, two reference blocks for motion compensation padding can be derived. And based on the two reference blocks, the motion compensation padding block 608 can be determined.

According to one embodiment, the larger motion compensation padding reference block among the two motion compensation padding reference blocks derived from two motion vectors may be determined as the motion compensation padding block 608. Alternatively, a motion compensation padding reference block of a reference block existing in a picture closer to the current picture 600 may be determined as the motion compensation padding block 608.

Alternatively, a block determined by averaging two motion compensation padding reference blocks may be determined as the motion compensation padding block 608. Alternatively, a block determined by performing a weighted average of two motion compensation padding reference blocks may be determined as the motion compensation padding block 608. At this time, the weight used in the weighted average may be determined according to the Picture Order Count (POC) distance between reference pictures.

According to FIG. 5, the sizes of the upper and lower motion compensation padding areas are respectively determined as (picture_width x 64), and the sizes of the left and right motion compensation padding areas are respectively determined as (64 x picture_height). According to FIG. 6, the size of the upper and lower motion compensation padding blocks is determined to be 4xM, and the size of the left and right motion compensation padding blocks is determined to be Mx4. Due to the first padding distance defined for the motion compensation padding block, the maximum value of M is determined by the first padding distance.

According to the embodiment, the first padding distance is indicated as 64 in FIG. 5 . However, the first padding distance may have values such as 8, 16, 32, 128, 256, etc., rather than 64. As the distance from the reference picture increases, the prediction accuracy of the motion vector extracted from the boundary block decreases. Accordingly, the first padding distance can be determined within a range in which the prediction accuracy of inter-screen prediction can be improved.

According to one embodiment, the first padding distance may be determined according to the maximum width or maximum height of the coding unit. For example, the first padding distance may be determined to be the same as the maximum width or maximum height of the coding unit. Alternatively, the first padding distance may be determined by multiplying or dividing a predetermined value from the maximum width or maximum height of the coding unit.

According to one embodiment, the first padding distance may be determined based on the size of the reference picture. For example, as the size of the reference picture increases, the first padding distance may increase. Conversely, as the size of the reference picture becomes smaller, the first padding distance may decrease.

According to one embodiment, the first padding distance may be determined according to the size of a boundary block that serves as a reference for motion compensation padding. According to FIG. 6, the size of the border block 602 is defined as 4x4, and the padding distance of the motion compensation padding area 604 is defined as 64. However, as the size of the border block increases, the padding distance of the motion compensation padding area may also increase.

According to one embodiment, the first padding distance may be determined from a slice header or a picture header. The slice header or picture header may include information indicating the first padding distance, or may refer to a parameter set containing the information. The parameter set may include a video parameter set, a sequence parameter set, and a picture parameter set.

If the value of M is smaller than the first padding distance, the motion compensation padding area from (M+1) to the first padding distance can be determined by repeatedly padding the pixel value located at the outermost position in the corresponding direction in the current picture. there is. For example, the motion compensation padding area 616 may be determined based on the current picture 600.

Alternatively, the motion compensation padding area may be determined by repeatedly padding the pixel value located at the outermost position in the corresponding direction in the motion compensation padding block. For example, the motion compensation padding area 616 may be determined by the motion compensation padding block 608 determined by the motion compensation padding.

If the border block 602 is encoded/decoded in intra-prediction mode or the motion vector information of the border block 602 is not available, the value of M is set to 0, and movement is performed in the method applied to the repetitive padding area. Padding of the compensation padding area 616 may be performed.

According to Figure 5, the size of the motion compensation padding area is (picture_width x 64) and (64 x picture_height). Depending on the embodiment, in order to reduce memory usage, the size of the motion compensation padding area may be determined according to a random integer N less than 64 (N ≤ 64). Therefore, the size of the motion compensation padding area can be set to (picture_width x N) and (N x picture_height). At this time, the size of N can be set to any integer less than or equal to 64 (e.g., 2, 4, 8, 16, 32, 64, etc.). Based on the determined size of N, the size of M may be determined in the Mx4 or 4xM motion compensation padding block. For example, if the size of N in the motion compensation padding area is determined to be 16, the size of M of the motion compensation padding block may be determined to be equal to or smaller than 16. That is, it can be set as M ≤ N.

Figure 7 shows an example of an extended picture in which an extended picture area is determined regardless of the size of the motion compensation padding area.

According to Figure 4, the extended picture area includes only a repetitive padding area. And, the extended picture area is created by extending the reference picture by pixels of the second padding distance (maxCUwidth + 16) in each direction from the border of the reference picture. On the other hand, according to FIG. 5, the extended picture area includes a repetitive padding area and a motion compensation padding area. And the extended picture area extends the reference picture by pixels with a distance (maxCUwidth + 16 + N) corresponding to the sum of the first padding distance (N) and the second padding distance (maxCUwidth + 16) in each direction from the boundary of the reference picture. It is created by doing.

Therefore, when determining the extended picture area according to FIG. 5, N pixels are further extended from the reference picture in each direction from the boundary of the reference picture than when determining the extended picture area according to the embodiment of FIG. 4, so memory usage is reduced. This increases. Therefore, in order to reduce memory usage, even when a motion compensation padding area is applied, the size of the extended picture area may be limited as shown in FIG. 4.

The size of the extended picture 700 according to FIG. 7 may be determined to be the same as the size of the extended picture 400 of FIG. 4. Accordingly, the size of the memory required to store the extended picture 400 of FIG. 4 and the memory required to store the extended picture 700 of FIG. 7 are determined to be the same.

In FIG. 5, the motion compensation padding area 506 is determined by a predetermined first padding distance, and the repetitive padding area 504 outside the motion compensation padding area 506 is determined by a predetermined second padding distance. Therefore, in the case of FIG. 5, the size of the extended picture 500 is determined by the sum of the first padding distance and the second padding distance. However, in the case of FIG. 7, regardless of the first padding distance, the size of the extended picture 700 may be determined by the second padding distance.

Specifically, the motion compensation padding area 706 in FIG. 7 is determined according to the first padding distance. However, since the extended picture 700 is determined as an area extended by the second padding distance from the reference picture 702, the repetitive padding area 704 is determined by the distance obtained by subtracting the first padding distance from the second padding distance.

According to Figure 7, the first padding distance is defined as N and the second padding distance is defined as (maxCUwidth + 16). The first padding distance is set to be smaller than the second padding distance, and N is set to be smaller than (maxCUwidth + 16). Here, maxCUwidth represents the maximum width of the coding unit. The second padding distance may be determined as the maximum width of the coding unit or the maximum height of the coding unit, or may be determined as a larger value.

In conclusion, the size of the extended picture 700 can be determined according to the second padding distance, regardless of the first padding distance applied to the motion compensation padding area 706. Therefore, regardless of whether the motion compensation padding area 706 is applied or not and the first padding distance, the size of the extended picture 700 and the size of the memory required for the extended picture 700 are fixed, so that the memory required for video encoding and decoding Resources can be managed efficiently.

According to another embodiment, the second padding distance is not defined, and the extended picture 700 may be extended only by the first padding distance (N) from the reference picture 702. In this case, only the motion compensation padding area 706 can be determined on the top, bottom, left, and right sides of the extended picture 700. And only the upper left, upper right, lower left, and lower right areas near the vertex of the extended picture 700 can be set as the repetitive padding area 704. Accordingly, the extended picture 700 may be determined to be an area extended from the reference picture 702 by the first padding distance.

In conclusion, the size of the extended picture 700 can be determined according to the first padding distance by omitting the creation of the repetitive padding area 704 according to the second padding distance. Therefore, in generating the extended picture 700, the second padding distance is not applied and only the first padding distance is applied, so that the size of the extended picture 700 and the size of the memory required for the extended picture 700 are fixed. You can. Because of this, memory resources required for video encoding and decoding can be efficiently managed.

Figure 8 shows an example of a method for performing motion compensation padding based on temporal neighboring blocks.

In the motion compensation padding method of FIGS. 5 to 7, if a border block adjacent to the border of the current picture is encoded/decoded in intra-prediction mode, or if the motion vector information of the border block is not available, padding using motion compensation is not possible. do. Therefore, according to one embodiment, in the above case, since a reference block to be referred to in determining the motion compensation padding block cannot be determined, the motion compensation padding area may be padded according to an iterative padding method.

In this case, since the motion compensation padding area is substantially determined according to a repetitive padding method, there is a possibility that the accuracy of the extended picture area is lowered, thereby lowering coding efficiency. Therefore, according to one embodiment, when a boundary block is encoded/decoded in intra-prediction mode, or when motion vector information of the boundary block is not available, a motion compensation padding block may be derived based on the temporal neighboring blocks of the boundary block. there is.

According to Figure 8, when the border block 802 adjacent to the border of the current picture 800 is encoded/decoded in intra-prediction mode, or when the motion vector information of the border block 802 is not available, the temporal corresponding reference picture ( At 820, a temporal neighboring block 822 of the same spatial location as the boundary block 802 is determined. And based on the motion vector of the temporal adjacent block 822, the corresponding reference block 842 in the reference picture 840 is determined. Then, a motion compensation padding reference block 844 is derived to the left of the reference block 842, and a motion compensation padding block 806 is determined based on the motion compensation padding reference block 844.

If the temporal adjacent block 822 is also encoded/decoded in intra-prediction mode, or the motion vector information of the temporal adjacent block 822 is not available, a reference block to be used in determining the motion compensation padding block 806 is determined. Since this cannot be done, the motion compensation padding area can be padded according to an iterative padding method. However, depending on the embodiment, if a motion vector cannot be derived from the temporal adjacent block 822, a new temporal adjacent block is derived from another temporally corresponding reference picture, and a motion compensation padding reference block 844 is derived from the new temporal adjacent block. This can be derived.

In FIG. 8, the padding distance applied to the motion compensation padding area 804 is N, which is a random integer, and the padding distance of the motion compensation padding block 806 is M, which is a random integer. Here M may be equal to or smaller than N. If M, the padding distance of the motion compensation padding block 806 determined by the motion compensation padding reference block 844, is smaller than N, the remaining blocks 808 that are not padded by the motion compensation padding method are the compensation padding block 806. ) can be padded using a repetitive padding method by referring to the pixels.

Hereinafter, Figures 9 to 11 show embodiments of a method for bidirectional motion prediction based on an extended picture determined based on an iterative padding method.

Figure 9 shows an example of a block prediction method using bidirectional motion prediction.

According to bidirectional motion prediction, a prediction signal of the current block 902 is generated using two reference blocks 924 and 944 derived from different reference pictures 922 and 942 referenced by the current picture 900. Specifically, the List0 reference block (924) derived from the List0 reference picture (922) and the List1 reference block derived from the List1 reference picture (942) A prediction signal for the current block can be generated using (List1 reference block) 944.

According to Figure 9, list 0 reference block 924 is partially out of boundary (OOB) of list 0 reference picture 922, while list 1 reference block 944 is completely list 1 reference picture 922. (942) Exists within (non out of boundary, non-OOB).

When determining the extended picture areas 920 and 940 using only the iterative padding method, when the reference block is partially or completely located outside the reference picture as shown in Figure 9, the prediction accuracy of the bidirectional motion prediction method may be lowered. . Therefore, in the above case, various embodiments are proposed to improve the prediction accuracy of the bidirectional motion prediction method.

Figure 10 shows an example of a block prediction method using bidirectional motion prediction when a part of a reference block is located outside the reference picture.

According to FIG. 10, when performing bidirectional motion prediction, the list1 reference block 1044 is completely within the list1 reference picture 1042, and the list0 reference block 1024 is partially within the list0 reference picture 1022. ) exists outside of. At this time, the prediction block of the current block 1002 in the current picture 1000 can be generated using only the list 1 reference block 1044 that exists completely in the reference picture. Conversely, if list0 reference block 1024 is completely within list0 reference picture 1022 and list1 reference block 1044 is partially outside list1 reference picture 1042, then list0 reference A prediction block of current block 1002 may be generated using only block 1024.

As shown in Figure 10, by using only reference blocks completely included in the reference picture, reference blocks including pixels in the repetitive padding area are excluded from motion prediction. Therefore, as the repetitive padding area is excluded from the prediction process, the prediction accuracy of the current block can be improved.

Figure 11 shows an example of a block prediction method using bidirectional motion prediction when a part of a reference block is located outside the reference picture.

According to FIG. 11, when performing bidirectional motion prediction, the list1 reference block 1144 is completely in the list1 reference picture 1142, and the list0 reference block 1124 is partially in the list0 reference picture 1122. ) exists outside of. At this time, all pixels of the list 1 reference block 1144 that completely exist in the list 1 reference picture 1142 can be referenced for generating the prediction block of the current block 1102. However, since part of the list 0 reference block 1124 exists outside the list 0 reference picture 1122, only pixels in the area 1126 included in the list 0 reference picture 1122 among the list 0 reference block 1124 are It may be referenced in the prediction block generation of the current block 1102. And among the list 0 reference blocks 1124, pixels in the area 1128 that are not included in the list 0 reference picture 1122 are not referred to in generating the prediction block of the current block 1102.

Specifically, the first area 1104 of the current block 1102 has corresponding areas 1126 and 1146 in the list 0 reference block 1124 and the list 1 reference block 1144. Accordingly, the first area 1104 may be determined based on the corresponding area 1126 of the List0 reference block 1124 and the corresponding area 1146 of the List1 reference block 1144. At this time, the first area 1104 may be determined as the average or weighted average of the two corresponding areas 1126 and 1146. The weight applied to the weighted average may be determined based on the adaptive weight value (bi-prediction with CU-level weight, BCW) of the coding unit (CU). Alternatively, the weight applied to the weighted average may be determined according to the temporal distance between the current picture 1100 and the list 0 reference picture 1122 and the temporal distance between the current picture 1100 and the list 1 reference picture 1142.

And, only the second area 1106 of the current block 1102 and the corresponding area 1148 of the list 1 reference block 1144 exist in the list 1 reference picture 1142. And the corresponding region 1128 of the list 0 reference block 1124 does not exist in the list 0 reference picture 1122. Therefore, the second area 1106 of the current block 1102 can be determined based only on the corresponding area 1148 of the List1 reference block 1144. And the corresponding area 1128 of the list 0 reference block 1124 is not referenced in determining the second area 1106 of the current block 1102.

As shown in Figure 11, in the reference block, only for the area where both pictures in both directions can be referenced, both pictures are referenced, and for the area where only one of the two pictures can be referenced, only the referenceable picture is referenced, thereby performing repetitive padding. Pixels in the area are excluded from motion prediction. Therefore, as in FIG. 10, as the repetitive padding area is excluded from the prediction process, the prediction accuracy of the current block can be improved.

Figure 12 shows an example of a block prediction method using bidirectional motion prediction when an extended picture has a motion compensation padding area.

In the case of FIGS. 9 to 11, since the reference picture is expanded according to the repetitive padding method, a bidirectional motion prediction method that avoids reference to the repetitive padding area has been proposed to increase prediction accuracy. However, when a reference picture is extended by a motion compensation padding method, even if part or all of the reference block is outside the reference picture, it can be referenced for prediction of the current block.

According to FIG. 12, motion compensation padding areas 1226 and 1246 are created according to a motion compensation method, and the current block 1202 can be predicted based on the motion compensation padding areas 1226 and 1246. According to one embodiment, the motion compensation padding areas 1226 and 1246 may be considered areas within each reference picture 1222 and 1242.

Therefore, although part of the list 0 reference block 1224 in FIG. 12 is outside the list 0 reference picture 1222, the list 0 motion compensation padding area 1226 is considered to be an area within the list 0 reference picture 1222. , all of the List0 reference blocks 1224 can be used for prediction of the current block 1202.

List1 reference block 1244 is entirely outside of List1 reference picture 1242. However, since the List1 motion compensation padding area 1246 is considered an area within the List1 reference picture 1242, depending on the embodiment, a part of the List1 reference block 1244 may be used for prediction of the current block 1202. there is.

In Figure 12, the motion compensation padding area is considered an area within the reference picture. Therefore, the embodiments of FIGS. 9 to 11 can be applied based on the motion compensation padding area being considered a reference picture. Therefore, when the motion compensation padding method is applied, the area referenced by prediction is expanded, and the efficiency of bidirectional prediction can be increased.

Figure 13 shows an example of a method for generating and storing an extended picture according to the present invention.

In step 1302, a motion compensation padding area within a first padding distance from the border of the current picture is padded according to the motion vector of a border block adjacent to the border of the current picture.

Step 1302 includes extracting a motion vector from a boundary block, based on the motion vector, determining a reference block from the motion compensation padding reference picture that is referenced to the motion compensation padding, and selecting a motion compensation padding reference block adjacent to the reference block in the padding direction. It may include determining and padding the motion compensation padding area based on a motion compensation padding reference block. The motion compensation padding reference picture refers to a reference picture referenced for reconstruction of a boundary block.

According to one embodiment, in determining the motion compensation padding reference block, the motion compensation padding reference block may be determined to include adjacent pixels within the first padding distance from the boundary of the reference block in the padding direction. .

According to one embodiment, in determining the motion compensation padding reference block, if the motion compensation padding reference possible distance between the boundary of the reference block and the boundary of the motion compensation padding reference picture is smaller than the first padding distance, The motion compensation padding reference block may be determined to include adjacent pixels within the motion compensation padding reference distance from the boundary of the reference block in the padding direction.

According to one embodiment, in the step of padding the motion compensation padding area based on the motion compensation padding reference block, the motion compensation padding reference possible distance between the boundary of the reference block and the boundary of the motion compensation padding reference picture is If it is smaller than the first padding distance, a motion compensation padding possible area among the motion compensation padding areas may be padded by the motion compensation padding reference block. The motion compensation padding possible area may be determined as an area within the motion compensation padding referenceable distance from the boundary of the current picture within the motion compensation padding area. Additionally, an area that is not a motion compensation padding area among the motion compensation padding areas may be padded based on the pixel value of the motion compensation padding area.

According to one embodiment, when a motion vector cannot be extracted from the boundary block, the motion compensation padding area is padded according to pixel values adjacent to the boundary of the current picture. For example, when the boundary block is predicted within the screen, a motion vector is not extracted from the boundary block.

According to one embodiment, the step of extracting a motion vector from the border block includes, when a motion vector cannot be extracted from the border block, a temporal neighborhood corresponding to the position of the border block from the temporal corresponding reference picture of the current picture. It may include determining a block and extracting a motion vector from the temporal neighboring block.

According to one embodiment, the first padding distance may be determined based on at least one of the maximum size of the coding unit, the size of the current picture, and the size of the boundary block. Alternatively, the first padding distance may be determined as one of 2, 4, 8, 16, 32, 64, 128, and 256.

In step 1304, a repetitive padding area within a second padding distance from the boundary of the motion compensation padding area is padded according to pixel values adjacent to the boundary of the motion compensation padding area.

According to one embodiment, the second padding distance may be determined based on at least one of the maximum size of the coding unit, the size of the current picture, and the size of the boundary block.

In step 1306, an extended picture consisting of the current picture, a motion compensation padding area, and a repetitive padding area is created. And the extended picture is stored in memory.

According to one embodiment, the size of the extended picture may be determined based on the size of the current picture, regardless of the value of the first padding distance.

The method for generating and storing the extended picture can be applied to an image decoding method and an image encoding method.

Figure 14 shows an example of a bidirectional inter-screen prediction method according to the present invention.

In step 1402, in order to predict the current block of the current picture, a first reference picture and a second reference picture referenced by the current picture are determined.

In step 1404, a first reference block and a second reference block are determined from the first reference picture and the second reference picture, respectively, according to the first motion vector and the second motion vector of the current block.

In step 1406, the first reference block and The current block is predicted using at least one of the second reference blocks.

According to one embodiment, when some or all of the pixels of the first reference block are not included in the first reference picture and all of the pixels of the second reference block are included in the second reference picture, the current A block may be predicted based on the second reference block.

According to one embodiment, when some of the pixels of the first reference block are not included in the first reference picture and all pixels of the second reference block are included in the second reference picture, the first reference The first area of the current block corresponding to the positions of pixels included in the first reference picture in the block may be determined as a weighted average of pixels of the first reference block and the second reference block. And the second area of the current block corresponding to the positions of pixels not included in the first reference picture in the first reference block may be predicted based only on the second reference block.

According to one embodiment, in determining whether all pixels of the first reference block are entirely included in the first reference picture and whether all pixels of the second reference block are entirely included in the second reference picture, The first motion compensation padding area of the first reference picture may be regarded as the first reference picture, and the second motion compensation padding area of the second reference picture may be regarded as the second reference picture.

The bidirectional inter-screen prediction method can be applied to an image decoding method and an image encoding method.

Figure 15 is a diagram illustrating a content streaming system to which an embodiment according to the present invention can be applied.

As shown in FIG. 15, a content streaming system to which an embodiment of the present invention is applied may largely include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

The encoding server compresses content input from multimedia input devices such as smartphones, cameras, CCTV, etc. into digital data, generates a bitstream, and transmits it to the streaming server. As another example, when multimedia input devices such as smartphones, cameras, CCTV, etc. directly generate bitstreams, the encoding server may be omitted.

The bitstream may be generated by an image encoding method and/or an image encoding device to which an embodiment of the present invention is applied, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

The streaming server transmits multimedia data to the user device based on a user request through a web server, and the web server can serve as a medium to inform the user of what services are available. When a user requests a desired service from the web server, the web server delivers it to a streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system may include a separate control server, and in this case, the control server may control commands/responses between each device in the content streaming system.

The streaming server may receive content from a media repository and/or encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server may store the bitstream for a certain period of time.

Examples of the user devices include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, slate PCs, Tablet PC, ultrabook, wearable device (e.g. smartwatch, smart glass, head mounted display), digital TV, desktop There may be computers, digital signage, etc.

Each server in the content streaming system may be operated as a distributed server, and in this case, data received from each server may be distributedly processed.

The above embodiments can be performed in the same or corresponding methods in the encoding device and the decoding device. Additionally, an image can be encoded/decoded using at least one or a combination of at least one of the above embodiments.

The order in which the above embodiments are applied may be different in the encoding device and the decoding device. Alternatively, the order in which the above embodiments are applied may be the same in the encoding device and the decoding device.

The above embodiments can be performed for each luminance and chrominance signal. Alternatively, the above embodiments for luminance and chrominance signals can be performed in the same way.

In the above embodiments, the methods are described based on flowcharts as a series of steps or units, but the present invention is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. . Additionally, a person of ordinary skill in the art will recognize that the steps shown in the flowchart are not exclusive and that other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You will understand.

The above embodiments may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field.

The bitstream generated by the encoding method according to the above embodiment may be stored in a non-transitory computer-readable recording medium. Additionally, the bitstream stored in the non-transitory computer-readable recording medium can be decoded using the decoding method according to the above embodiment.

Here, examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. -optical media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , a person skilled in the art to which the present invention pertains can make various modifications and variations from this description.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims fall within the scope of the spirit of the present invention. They will say they do it.

Claims (13)

In the video decoding method,
padding a motion compensation padding area within a first padding distance from the border of the current picture according to the motion vector of a border block adjacent to the border of the current picture;
padding a repetitive padding area within a second padding distance from the boundary of the motion compensation padding area according to a pixel value adjacent to the boundary of the motion compensation padding area; and
An image decoding method comprising storing an extended picture composed of the current picture, the motion compensation padding area, and the repetitive padding area in a memory.
According to paragraph 1,
The step of padding the motion compensation padding area is:
extracting a motion vector from the boundary block;
Based on the motion vector, determining a reference block from a motion compensation padding reference picture that is referenced to motion compensation padding;
determining a motion compensation padding reference block adjacent to the reference block in a padding direction; and
An image decoding method comprising padding the motion compensation padding area based on the motion compensation padding reference block.
According to paragraph 2,
In determining the motion compensation padding reference block,
The motion compensation padding reference block is determined to include adjacent pixels within the first padding distance from the boundary of the reference block in the padding direction.
According to paragraph 3,
In determining the motion compensation padding reference block,
If the motion compensation padding reference possible distance between the boundary of the reference block and the boundary of the motion compensation padding reference picture is smaller than the first padding distance, the motion compensation padding reference block moves in the padding direction from the boundary of the reference block. An image decoding method characterized in that the compensation padding is determined to include adjacent pixels within a referenceable distance.
According to clause 4,
In the step of padding the motion compensation padding area based on the motion compensation padding reference block,
If the motion compensation padding referenceable distance between the boundary of the reference block and the boundary of the motion compensation padding reference picture is smaller than the first padding distance, the motion compensation padding can be referenced from the boundary of the current picture in the motion compensation padding area. A motion compensation padding possible area within the distance is padded by the motion compensation padding reference block,
An image decoding method, wherein an area that is not a motion compensation padding area among the motion compensation padding areas is padded based on pixel values of the motion compensation padding area.
According to paragraph 2,
The video decoding method is,
When a motion vector cannot be extracted from the boundary block, the motion compensation padding area is padded according to pixel values adjacent to the boundary of the current picture.
According to paragraph 2,
The step of extracting a motion vector from the boundary block is:
If a motion vector cannot be extracted from the boundary block, determining a temporal neighboring block corresponding to the location of the boundary block from a temporally corresponding reference picture of the current picture; and
An image decoding method comprising extracting a motion vector from the temporal neighboring block.
According to paragraph 1,
The first padding distance is,
An image decoding method characterized in that it is determined based on at least one of the maximum size of the coding unit, the size of the current picture, and the size of the boundary block.
According to paragraph 1,
The first padding distance is,
A video decoding method characterized in that it is determined as one of 2, 4, 8, 16, 32, 64, 128, and 256.
According to paragraph 1,
The second padding distance is,
An image decoding method characterized in that it is determined based on at least one of the maximum size of the coding unit, the size of the current picture, and the size of the boundary block.
According to paragraph 1,
The size of the extended picture is,
An image decoding method characterized in that it is determined based on the size of the current picture, regardless of the value of the first padding distance.
In the video encoding method,
padding a motion compensation padding area within a first padding distance from the border of the current picture according to the motion vector of a border block adjacent to the border of the current picture;
padding a repetitive padding area within a second padding distance from the boundary of the motion compensation padding area according to a pixel value adjacent to the boundary of the motion compensation padding area; and
An image encoding method comprising storing an extended picture composed of the current picture, the motion compensation padding area, and the repetitive padding area in a memory.
In a computer-readable recording medium storing a bitstream generated by an image encoding method,
The video encoding method is,
padding a motion compensation padding area within a first padding distance from the border of the current picture according to the motion vector of a border block adjacent to the border of the current picture;
padding a repetitive padding area within a second padding distance from the boundary of the motion compensation padding area according to a pixel value adjacent to the boundary of the motion compensation padding area; and
A recording medium comprising the step of storing an extended picture composed of the current picture, the motion compensation padding area, and the repetitive padding area in a memory.

Images