HK40111368A - Image encoding/decoding method and data transmitting method - Google Patents

Description

Image encoding/decoding methods and data transmission methods

This application is a divisional application of the invention patent application filed on September 24, 2019, with application number 201980062809.8 and title "Image Encoding/Decoding Method and Apparatus".

Technical Field

This invention relates to image encoding/decoding methods and devices.

Background Technology

With the widespread use of the Internet and portable terminals, and the development of information and communication technologies, multimedia data is being used more and more. Therefore, in order to provide various services or perform various tasks through image prediction in various systems, there is an urgent need to improve the performance and efficiency of image processing systems. However, research and development achievements have not kept pace with this trend.

Therefore, existing methods and devices for encoding/decoding images need performance improvements in image processing, particularly in image encoding or image decoding.

Summary of the Invention

Technical issues

The purpose of this invention to solve the above problems is to provide an image encoding/decoding device that uses an adjustment offset to modify the motion vector prediction value.

Technical solutions

According to an embodiment of the present invention, a method for decoding an image to achieve the above objectives includes: constructing a motion information prediction candidate list for a target block, selecting a prediction candidate index, obtaining a predicted motion vector adjustment offset, and reconstructing the motion information of the target block.

Here, constructing a candidate list for motion information prediction may also include: when the already included candidates, candidates obtained based on offset information, new candidates, and candidates obtained based on offset information do not overlap, the new candidate is included in the candidate group.

Here, deriving the predicted motion vector adjustment offset may also include: deriving the predicted motion vector adjustment offset based on the offset application flag and/or offset selection information.

Beneficial effects

When using inter-frame prediction according to the present invention as described above, coding performance can be improved by efficiently obtaining the predicted motion vectors.

Attached Figure Description

Figure 1 is a block diagram illustrating an image encoding and decoding system according to an embodiment of the present invention.

Figure 2 is a block diagram illustrating an image encoding device according to an exemplary embodiment of the present invention.

Figure 3 illustrates an image decoding device according to an embodiment of the present invention.

Figure 4 illustrates a tree-based block partitioning method as an embodiment of the present invention.

Figure 5 is an example diagram illustrating various cases of obtaining prediction blocks through inter-frame prediction according to the present invention.

Figure 6 is an example diagram of constructing a list of reference pictures according to an embodiment of the present invention.

Figure 7 is a conceptual diagram illustrating a non-translational motion model according to an embodiment of the present invention.

Figure 8 is an example diagram illustrating motion estimation in units of sub-blocks according to an embodiment of the present invention.

Figure 9 is a flowchart illustrating the encoding of motion information according to an embodiment of the present invention.

Figure 10 is a layout diagram of the target block and its adjacent blocks according to an embodiment of the present invention.

Figure 11 shows an example diagram of statistical candidates according to an embodiment of the present invention.

Figure 12 is a conceptual diagram of statistical candidates based on a non-translational motion model according to an embodiment of the present invention.

Figure 13 is an example diagram of the construction of motion information of the position of each control point stored as a statistical candidate according to an embodiment of the present invention.

Figure 14 is a flowchart illustrating motion information encoding according to an embodiment of the present invention.

Figure 15 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 16 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 17 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 18 is an example diagram illustrating the arrangement of multiple motion vector prediction values according to an embodiment of the present invention.

Figure 19 is a flowchart illustrating motion information encoding under a merging mode according to an embodiment of the present invention.

The best mode of the invention

The image encoding/decoding method and apparatus of the present invention can construct a predicted motion candidate list of a target block, derive a predicted motion vector from the motion candidate list based on the predicted candidate index, reconstruct the predicted motion vector adjustment offset information, and reconstruct the motion vector of the target block based on the predicted motion vector and the predicted motion vector adjustment offset information.

In the image encoding/decoding method and apparatus of the present invention, the motion candidate list may include at least one of spatial candidates, temporal candidates, statistical candidates, or combined candidates.

In the image encoding/decoding method and apparatus of the present invention, the predicted motion vector adjustment offset can be determined based on at least one of an offset application flag or offset selection information.

In the image encoding/decoding method and apparatus of the present invention, information regarding whether the predicted motion vector adjustment offset information is supported may be included in at least one of the sequence, image, sub-image, slice, tile, or block.

In the image encoding/decoding method and apparatus of the present invention, when encoding a target block in a merge mode, the motion vector of the target block can be reconstructed by using a zero vector, and when encoding a target block in a contention mode, the motion vector of the target block can be reconstructed by using a motion vector difference.

The mode of the present invention

This disclosure can be modified in various ways and has various implementations. Specific implementations of this disclosure will be described with reference to the accompanying drawings. However, these implementations are not intended to limit the technical scope of this disclosure, and it should be understood that this disclosure covers various modifications, equivalents, and alternatives within the scope and concept of this disclosure.

The terms first, second, A, and B used in this disclosure may be used to describe various components, rather than to limit them. These expressions are used only to distinguish one component from another. For example, without departing from the scope of this disclosure, a first component may be referred to as a second component, and a second component may be referred to as a first component. The term and/or covers a combination of or any one of several related terms.

When a component is said to be "connected to" another component or "coupled to/coupled to" another component, it should be understood that a component is connected to the other component directly or through any other component. On the other hand, when a component is said to be "directly connected to" or "directly coupled to" another component, it should be understood that there are no other components between the components.

The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. Unless the context clearly specifies otherwise, the singular form includes the plural reference. In this disclosure, the terms “comprising” or “having” indicate the presence of a feature, number, step, operation, component, part, or combination thereof, but do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, terms including technical or scientific terms used in this disclosure may have the same meaning as commonly understood by those skilled in the art. Terms that are commonly defined in dictionaries may be interpreted as having the same or similar meaning as in the context of the relevant art. Unless otherwise defined, these terms should not be interpreted as having an ideal or overly formal meaning.

Typically, an image can include one or more color spaces depending on its color format. The image can include one or more images of the same size or one or more images of different sizes. For example, the YCbCr color configuration can support color formats such as 4:4:4, 4:2:2, 4:2:0, and monochrome (consisting only of Y). For example, YCbCr 4:2:0 can consist of one luminance component (Y in this example) and two chrominance components (Cb/Cr in this example). In this case, the configuration ratio of the chrominance and luminance components can have a width-to-height ratio of 1:2. For example, in the case of 4:4:4, it can have the same configuration ratio in terms of width and height. As in the examples above, when an image includes one or more color spaces, the image can be divided into color spaces.

Images can be classified into I, P, and B based on their image type (e.g., picture type, subpicture type, slice type, tile type, block type, etc.). I images can be images encoded without a reference image. P images can be images encoded using a reference image, allowing only forward prediction. B images can be images encoded using a reference image, allowing bidirectional prediction. However, depending on the encoding settings, some types (P and B) can be combined, or different compositions of image types can be supported.

The various encoded/decoded information fragments generated in this disclosure can be processed explicitly or implicitly. Explicit processing can be understood as the following: generating encoded/decoded information in sequences, images, sub-images, slices, tiles, blocks, chunks, or sub-blocks by an encoder, including selection information in the bitstream, and reconstructing the relevant information into decoded information by a decoder parsing the relevant information at the same unit level as in the encoder. Implicit processing can be understood as processing the encoded/decoded information using the same procedures, rules, etc., at both the encoder and decoder.

Figure 1 is a conceptual diagram illustrating an image encoding and decoding system according to an embodiment of the present disclosure.

Referring to FIG1, each of the image encoding device 105 and the image decoding device 100 may be a user terminal, such as a personal computer (PC), laptop computer, personal digital assistant (PDA), portable multimedia player (PMP), portable game console (PSP), wireless communication terminal, smartphone, or television (TV), or a server terminal such as an application server or service server. Each of the image encoding device 105 and the image decoding device 100 may be any of a variety of devices, each device including: a communication device such as a communication modem that communicates with various devices or wired/wireless communication networks; a memory 120 or 125 that stores various programs and data for inter-frame prediction or intra-frame prediction to encode or decode images; or a processor 110 or 115 that performs calculation and control operations by executing programs.

Furthermore, the image encoding device 105 can transmit images encoded as bitstreams to the image decoding device 100 in real time or non-real time via wired/wireless communication networks such as the Internet, short-range wireless communication networks, wireless local area networks (WLANs), wireless broadband (Wi-Bro) networks, or mobile communication networks, or via various communication interfaces such as cables or universal serial buses (USB). The image decoding device 100 can reconstruct the image from the received bitstream by decoding the bitstream and reproduce the image. Additionally, the image encoding device 105 can transmit images encoded as bitstreams to the image decoding device 100 via a computer-readable recording medium.

Although the aforementioned image encoding and image decoding devices can be separate devices, depending on the implementation, they can be incorporated into a single image encoding/decoding device. In this case, some components of the image encoding device can be substantially the same as their corresponding components of the image decoding device. Therefore, these components can be configured to include the same structure or perform at least the same function.

Therefore, in the following detailed description of the technical components and their operating principles, redundant descriptions of the corresponding technical components will be omitted. Furthermore, since the image decoding device is a computing device that applies the image encoding method executed in the image encoding device to decoding, the following description will focus on the image encoding device.

The computing device may include: a memory storing programs or software modules that perform image encoding and/or image decoding methods; and a processor connected to the memory and executing the programs. An image encoding device may be referred to as an encoder, and an image decoding device may be referred to as a decoder.

Figure 2 is a block diagram illustrating an image encoding apparatus according to an embodiment of the present disclosure.

Referring to FIG2, the image encoding device 20 may include a prediction unit 200, a subtraction unit 205, a transformation unit 210, a quantization unit 215, an inverse quantization unit 220, an inverse transformation unit 225, an addition unit 230, a filter unit 235, an encoded image buffer 240, and an entropy encoding unit 245.

The prediction unit 200 can be implemented using a prediction module as a software module, and generates prediction blocks for the blocks to be encoded via intra-frame prediction or inter-frame prediction. The prediction unit 200 can generate prediction blocks by predicting the target block to be encoded in the image. In other words, the prediction unit 200 can generate prediction blocks with predicted pixel values for each pixel by predicting the pixel values of pixels in the target block based on inter-frame prediction or intra-frame prediction. Furthermore, the prediction unit 200 can provide the encoding unit with information required to generate prediction blocks—such as information about prediction modes, like intra-frame prediction or inter-frame prediction modes—so that the encoding unit can encode the information about the prediction modes. The processing unit undergoing prediction, the prediction method, and specific details about the processing unit can be determined based on the encoding settings. For example, the prediction method and prediction mode can be determined based on the prediction unit, and prediction can be performed based on the transform unit. Additionally, when using a specific encoding mode, the original block can be encoded as is and sent to the decoder without generating prediction blocks through the prediction unit.

Intra-prediction units can have: directional prediction modes, such as horizontal modes, vertical modes, etc., used according to the prediction direction; and non-directional prediction modes, such as DC, plane, etc., using methods such as averaging and interpolating reference pixels. Intra-prediction mode candidate groups can be constructed using directional and non-directional modes, and one of various candidates, such as 35 prediction modes (33 directional + 2 non-directional), 67 prediction modes (65 directional + 2 non-directional), or 131 prediction modes (129 directional + 2 non-directional), can be used as candidate groups.

The intra-prediction unit may include a reference pixel construction unit, a reference pixel filter unit, a reference pixel interpolation unit, a prediction mode determination unit, a prediction block generation unit, and a prediction mode coding unit. The reference pixel construction unit can construct pixels belonging to and adjacent to the target block as reference pixels for intra-prediction. Depending on the coding settings, one adjacent reference pixel line can be constructed as a reference pixel, or another adjacent reference pixel line can be constructed as a reference pixel, or multiple reference pixel lines can be constructed as reference pixels. When some reference pixels are unavailable, available reference pixels can be used to generate a reference pixel. When all reference pixels are unavailable, a predetermined value (e.g., the median of the pixel value range represented by the bit depth) can be used to generate a reference pixel.

To reduce degradation remaining after coding processing, the reference pixel filter unit of the intra-prediction unit can perform filtering on the reference pixel. In this case, the filter used can be a low-pass filter, such as a 3-tap filter [1/4, 1/2, 1/4], a 5-tap filter [2/16, 3/16, 6/16, 3/16, 2/16], etc. Whether to apply filtering and the type of filtering can be determined based on coding information (e.g., block size, shape, prediction mode, etc.).

The reference pixel interpolation unit of the intra-prediction unit can generate fractional-unit pixels based on the prediction mode through linear interpolation of the reference pixels, and can determine the interpolation filter to be applied based on the coding information. In this case, the interpolation filter used may include a 4-tap cubic filter, a 4-tap Gaussian filter, a 6-tap Wiener filter, an 8-tap Kalman filter, etc. Typically, interpolation and low-pass filtering are performed separately, but the filtering process can be performed by combining the filters applied to both processes into one.

The prediction mode determination unit of the intra-frame prediction unit can select at least one optimal prediction mode from the prediction mode candidates, taking coding cost into account, and the prediction block generation unit can use the corresponding prediction mode to generate a prediction block. The prediction mode coding unit can encode the optimal prediction mode based on the prediction value. In this case, the prediction information can be adaptively encoded according to whether the prediction value is correct or not.

In an intra-frame prediction unit, the predicted value is called the most probable mode (MPM), and some modes belonging to the prediction mode candidate group can be constructed into an MPM candidate group. The MPM candidate group can include predicted modes of a predetermined prediction mode (e.g., DC mode, planar mode, vertical mode, horizontal mode, diagonal mode, etc.) or predicted modes of spatially adjacent blocks (e.g., left block, top block, top-left block, top-right block, bottom-left block, etc.). Additionally, modes derived from modes previously included in the MPM candidate group (the difference between +1 and -1 in the case of directional mode) can also be constructed into an MPM candidate group.

There may be a priority for the prediction patterns used to construct MPM candidate groups. The order in which patterns are included in the MPM candidate groups can be determined based on the priority, and the construction of MPM candidate groups can be completed by filling the number of MPM candidate groups according to the priority (determined by the number of prediction pattern candidate groups). In this case, the priority can be determined according to the order of prediction patterns of spatially adjacent blocks, predetermined prediction patterns, and patterns derived from prediction patterns previously included in the MPM candidate groups, but other modifications are also possible.

For example, spatially adjacent blocks can be included in the candidate group in the order of left block, top block, bottom left block, top right block, top left block, etc., and predetermined prediction patterns can be included in the candidate group in the order of DC pattern, planar pattern, vertical pattern, and horizontal pattern. A total of six patterns can be constructed into candidate groups by including patterns obtained by adding +1, -1, etc., to the candidate groups from already included patterns. Alternatively, a total of seven patterns can be constructed into candidate groups by including a priority such as left, top, DC, planar, bottom left, top right, top left, (left+1), (left-1), (top+1).

Subtraction unit 205 can generate residual blocks by subtracting prediction blocks from target blocks. In other words, subtraction unit 205 can calculate the difference between the pixel value of each pixel in the target block to be encoded and the predicted pixel value of the corresponding pixel in the prediction block generated by the prediction unit, to generate a residual signal in the form of a block, i.e., a residual block. Furthermore, subtraction unit 205 can generate residual blocks using units other than those obtained by the block partitioning unit described later.

Transformation unit 210 can transform a spatial signal into a frequency signal. The signal obtained through the transformation process is called the transform coefficient. For example, a residual block having a residual signal received from a subtraction unit can be transformed into a transform block with transform coefficients, and the input signal is determined according to the encoding configuration, which is not limited to the residual signal.

Transformation units can transform residual blocks using transformation schemes such as, but not limited to, Hadamard transform, discrete sine transform (DST)-based transform, or DCT-based transform. These transformation schemes can be changed and modified in various ways.

It can support at least one of the transformation schemes, and can support at least one sub-transformation scheme for each transformation scheme. A sub-transformation scheme can be obtained by modifying a portion of the fundamental vectors in the transformation scheme.

For example, in the case of DCT, one or more sub-transformation schemes DCT-1 to DCT-8 can be supported, and in the case of DST, one or more sub-transformation schemes DST-1 to DST-8 can be supported. A transformation scheme candidate group can be configured with a portion of the sub-transformation schemes. For example, DCT-2, DCT-8, and DST-7 can be grouped into a candidate group for transform.

Transformations can be performed in both the horizontal and vertical directions. For example, a one-dimensional transformation can be performed in the horizontal direction using DCT-2, and a one-dimensional transformation can be performed in the vertical direction using DST-7. Two-dimensional transformations can be used to transform pixel values from the spatial domain to the frequency domain.

A fixed transform scheme can be used, or the transform scheme can be adaptively selected based on the coding configuration. In the latter case, the transform scheme can be selected explicitly or implicitly. When the transform scheme is selected explicitly, information about the transform scheme or set of transform schemes applied in each of the horizontal and vertical directions can be generated, for example, at the block level. When the transform scheme is selected implicitly, the coding configuration can be limited based on image type (I/P/B), color components, block size, block shape, block position, intra-frame prediction mode, etc., and a predetermined transform scheme can be selected based on the coding settings.

In addition, some transformations can be skipped depending on the encoding settings. That is, one or more horizontal and vertical units can be explicitly or implicitly omitted.

Furthermore, the transform unit can send the information needed to generate the transform block to the encoding unit, which encodes the information, includes the encoded information in the bitstream, and sends the bitstream to the decoder. Therefore, the decoder's decoding unit can parse the information from the bitstream for inverse transform.

The quantization unit 215 can quantize the input signal. The signal obtained from the quantization is called the quantization coefficient. For example, the quantization unit 215 can obtain a quantized block with quantization coefficients by quantizing a residual block having residual transform coefficients received from the transform unit, and can determine the input signal according to the encoding settings, which are not limited to the residual transform coefficients.

The quantization unit can quantize the residual block of the transformation using quantization schemes such as, but not limited to, dead-zone uniform boundary value quantization and quantization weighted matrix. These quantization schemes can be changed and modified in various ways.

Quantization can be skipped based on encoding settings. For example, quantization (and inverse quantization) can be skipped based on encoding settings (e.g., a quantization parameter of 0, i.e., a lossless compression environment). In another example, quantization can be omitted when quantization-based compression performance is not performed due to image characteristics. Quantization can be skipped in the entirety or a portion of a quantization block (M×N) (M/2×N/2, M×N/2, or M/2×N), and quantization skip selection information can be set explicitly or implicitly.

The quantization unit can send the information needed to generate the quantization block to the encoding unit, which then encodes the information, includes the encoded information in the bitstream, and sends the bitstream to the decoder. Therefore, the decoder's decoding unit can parse the information from the bitstream for inverse quantization.

Although the above example was described assuming that the residual block is transformed and quantized via a transform unit and a quantization unit, a residual block with transform coefficients can be generated by transforming the residual signal without quantization. The residual block can undergo quantization only without transformation. Alternatively, the residual block can undergo both transformation and quantization. These operations can be determined based on the encoding settings.

The inverse quantization unit 220 performs inverse quantization on the residual block quantized by the quantization unit 215. That is, the inverse quantization unit 220 generates a residual block with frequency coefficients by performing inverse quantization on the quantized frequency coefficient sequence.

The inverse transform unit 225 performs an inverse transform on the residual block dequantized by the inverse quantization unit 220. That is, the inverse transform unit 225 performs an inverse transform on the frequency coefficients of the dequantized residual block to generate a residual block with pixel values, i.e., a reconstructed residual block. The inverse transform unit 225 can perform the inverse transform by reversing the transform scheme used by the transform unit 210.

Addition unit 230 reconstructs the target block by adding the predicted block predicted by prediction unit 200 to the residual block recovered by inverse transform unit 225. The reconstructed target block is stored as a reference picture (or reference block) in encoded picture buffer 240 for use as a reference picture when encoding the next block, another block, or another picture of the target block later.

Filter unit 235 may include one or more post-processing filters, such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF). The deblocking filter removes block distortion occurring at boundaries between blocks in the reconstructed image. The ALF performs filtering based on values obtained by comparing the reconstructed image after filtering the blocks with the deblocking filter with the original image. The SAO reconstructs the pixel-level offset difference between the original image and the residual blocks to which the deblocking filter was applied. These post-processing filters can be applied to the reconstructed image or blocks.

The encoded image buffer 240 can store blocks or images reconstructed by the filter unit 235. The reconstructed blocks or images stored in the encoded image buffer 240 can be provided to the prediction unit 200 that performs intra-frame prediction or inter-frame prediction.

Entropy coding unit 245 can generate a quantization coefficient sequence, a transform coefficient sequence, or a signal sequence by scanning the quantization coefficients, transform coefficients, or residual signals of the generated residual block according to at least one scanning order (e.g., zigzag scanning, vertical scanning, horizontal scanning, etc.). Entropy coding unit 245 can encode the quantization coefficient sequence, transform coefficient sequence, or signal sequence using at least one entropy coding technique. In this case, information about the scanning order can be determined according to coding settings (e.g., image type, coding mode, prediction mode, transform type, etc.), and the relevant information can be implicitly determined or explicitly generated.

Additionally, encoded data, including the encoded information sent from each component, can be generated and output as a bitstream, which can be implemented by a multiplexer (MUX). In this case, encoding can be performed using methods such as exponential Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC) as encoding techniques, but is not limited to these, and various encoding techniques derived from and modified from them can be used.

When entropy encoding (assuming CABAC in this example) is performed on syntax elements such as residual block data and information generated during encoding/decoding processes, an entropy encoding device may include a binarization unit (binarizer), a context modeler, and a binary arithmetic encoding unit (binary arithmetic encoder). In this case, the binary arithmetic encoding unit may include a regular encoding engine and a bypass encoding engine.

Since the syntax elements input to the entropy coding device may not be binary values, if the syntax element is not a binary value, the binarization unit can binarize the syntax element and output a binary string consisting of 0s or 1s. In this case, the binary (bin) can represent bits consisting of 0s or 1s and can be encoded by the binary arithmetic coding unit. In this case, one of the regular coding unit or the bypass coding unit can be selected based on the probability of occurrence of 0s and 1s, and this can be determined according to the encoding/decoding settings. If the syntax element is data with the same frequency of 0s and 1s, the bypass coding unit can be used; otherwise, the regular coding unit can be used.

Various methods can be used when binarizing grammatical elements. For example, fixed-length binarization, unary binarization, truncated Ricean binarization, and K-order exponential Golomb binarization can be used. Additionally, signed or unsigned binarization can be performed depending on the range of values for the grammatical element. The binarization processing of grammatical elements described in this invention can be performed not only on the binarization methods mentioned in the examples above, but also on other additional binarization methods.

Figure 3 is a block diagram illustrating an image decoding apparatus according to an embodiment of the present disclosure.

Referring to Figure 3, the image decoding device 30 can be configured to include an entropy decoder 305, a prediction unit 310, an inverse quantization unit 315, an inverse transform unit 320, an addition/subtraction unit 325, a filter 330, and a decoded image buffer 335.

Furthermore, the prediction unit 310 can be configured to include an intra-frame prediction module and an inter-frame prediction module.

When an image bitstream is received from the image encoding device 20, it can be sent to the entropy decoder 305.

The entropy decoder 305 can decode the bitstream into decoded data including quantization coefficients and decoded information to be sent to each component.

Prediction unit 310 can generate prediction blocks based on data received from entropy decoder 305. A list of reference images can be generated using a default configuration scheme based on reference images stored in decoded image buffer 335.

The intra-frame prediction unit may include a reference sample construction unit, a reference sample filter unit, a reference sample interpolation unit, a prediction block generation unit, and a prediction mode decoding unit. Some of these units may perform the same processing as in the encoder, while others may perform the encoder's processing in reverse.

The inverse quantization unit 315 can inverse quantize the quantized transform coefficients provided in the bitstream and decoded by the entropy decoder 305.

The inverse transform unit 320 can generate residual blocks by applying inverse DCT, inverse integer transform, or similar inverse transform techniques to the transform coefficients.

The inverse quantization unit 315 and the inverse transform unit 320 can perform the processing of the transform unit 210 and the quantization unit 215 of the image encoding device 20 in reverse, and can be implemented in various ways. For example, the inverse quantization unit 315 and the inverse transform unit 320 can use the same processing and inverse transform shared with the transform unit 210 and the quantization unit 215, and can perform the transform and quantization in reverse using information received from the image encoding device 20 about the transform and quantization processing (e.g., transform size, transform shape, quantization type, etc.).

The residual block, which has already undergone inverse quantization and inverse transform, can be added to the prediction block obtained by prediction unit 310 to generate a reconstructed image block. This addition can be performed by addition/subtraction unit 325.

Regarding filter 330, a deblocking filter can be applied when needed to remove blocking artifacts from the reconstructed image blocks. Additional loop filters can be used to improve video quality before and after decoding.

The reconstructed and filtered image blocks can be stored in the decoded image buffer 335.

Although not shown in the accompanying drawings, the image encoding/decoding device may also include image partitioning units and block partitioning units.

Image partitioning units can divide or partition an image into at least one region based on predetermined partitioning units. Here, partitioning units can include sub-images, slices, tiles, blocks, blocks (e.g., maximum coding units), etc.

An image can be divided into one or more rows or columns of tiles. In this case, a tile can be a block-based unit that includes a predetermined non-square area of the image. Alternatively, a tile can be divided into one or more blocks, and a block can consist of blocks arranged in rows or columns.

Slices can be set using one or more configurations, one of which can be composed of bundles (e.g., blocks, regions, tiles, etc.) depending on the scan order, and one of them can be composed of shapes that include non-square regions, and other additional definitions are also possible.

Regarding the definition of slice configuration, relevant information can be generated explicitly or implicitly. Multiple definitions can be set for the configuration of each partitioning unit and slice, and selection information can be generated accordingly.

Slices can be configured in non-square shapes such as blocks, regions, and tiles, and slice position and size information can be represented based on position information for the dividing unit (e.g., top left, bottom right, etc.).

In this invention, the description assumes that an image can be composed of one or more sub-images, a sub-image can be composed of one or more slices, tiles, or blocks, a slice can be composed of one or more tiles or blocks, and a tile can be composed of one or more blocks. However, the invention is not limited thereto.

A partition unit can consist of an integer number of blocks, but is not limited to this, and can consist of decimal numbers instead of integers. That is, when it is not composed of an integer number of blocks, at least one partition unit can consist of sub-blocks.

In addition to non-square slices, there can be partitioning units such as sub-images or tiles, and the position and size information of the units can be represented based on various methods.

For example, the position and size information of non-square cells can be represented based on information about the number of non-square cells, information about the number of columns or rows of non-square cells, information about whether they are evenly divided into columns or rows in non-square cells, information about the width or height of column or row cells in non-square cells, and index information of non-square cells.

In the case of sub-images and tiles, the position and size information of each unit can be represented based on all or part of the information, and based on this, it can be divided or partitioned into one or more units.

Simultaneously, blocks can be divided into various units and sizes using block partitioning units. A basic coding unit (or maximum coding unit, coding tree unit, CTU) can refer to the basic (or initial) unit used for prediction, transformation, quantization, etc., in image coding processing. In this case, a basic coding unit can consist of one luminance basic coding block (maximum coding block or CTB) and two basic chrominance coding blocks, depending on the color format (YCbCr in this example), and the size of each block can be determined according to the color format. Coding blocks (CBs) can be obtained according to the partitioning process. A CB can be understood as a unit that is no longer further subdivided due to certain limitations and can be set as the starting unit for partitioning into subunits. In this disclosure, blocks conceptually include various shapes such as triangles, circles, etc., and are not limited to squares.

While the following description is given in the context of a single color component, it is also applicable, proportionally to the ratios according to the color format, to other color components with some modifications (e.g., in the case of YCbCr 4:2:0, the width-to-height length ratio of the luminance component to the chrominance component is 2:1). Furthermore, although block partitioning depending on other color components (e.g., depending on the block partitioning result of Y in Cb/Cr) is feasible, it should be understood that independent block partitioning for each color component is also feasible. Moreover, while a common block partitioning configuration can be used (considering proportionality to the length ratio), it is also necessary to consider and understand the use of separate block partitioning configurations based on the color components.

In a block partitioning unit, a block can be represented as M×N, and the maximum and minimum values of each block can be obtained within this range. For example, if the maximum and minimum values of a block are 256×256 and 4×4 respectively, then blocks of size ^2m × ²ⁿ (where m and n are integers from 2 to 8 in this example), blocks of size 2m×2m (where m and n are integers from 2 to 128 in this example), or blocks of size m×m (where m and n are integers from 4 to 256 in this example) can be obtained. In this paper, m and n can be equal or different, and one or more ranges, such as maximum and minimum values, can be generated to support the blocks.

For example, information about the maximum and minimum size of a block can be generated, and information about the maximum and minimum size of a block can be generated in some partitioning configurations. In the former case, the information can be about the range of maximum and minimum sizes that can be generated in the image, while in the latter case, the information can be about the maximum and minimum sizes that can be generated based on some partitioning configuration. The partitioning configuration can be defined by image type (I/P/B), color components (YCbCr, etc.), block type (encoding/prediction/transformation/quantization), partition type (index or type), and partitioning scheme (quadtree QT, binary tree BT, and ternary tree TT as tree methods, and SI2, SI3, and SI4 as type methods).

Furthermore, constraints can exist on the aspect ratio (block shape) that can be used for blocks, and at this point, boundary values can be set. Only blocks less than or equal to/less than the boundary value (k) can be supported, where k can be limited according to the aspect ratio A/B (A is the longer or equal value between the width and height, and B is another value). k can be a real number equal to or greater than 1, such as 1.5, 2, 3, 4, etc. As in the example above, constraints on the shape of a block in an image can be supported, or one or more constraints can be supported depending on the partitioning configuration.

In summary, whether block partitioning is supported can be determined based on the above scope and constraints, as well as the partitioning configuration described later. For example, partitioning can be supported when a candidate (child block) split from a block (parent block) meets the block support conditions; otherwise, partitioning may not be supported.

The block partitioning unit can be configured with respect to each component of the image encoding and image decoding devices, and the size and shape of the block can be determined in the process. Different blocks can be configured according to the component. The blocks may include prediction blocks for prediction units, transform blocks for transform units, and quantization blocks for quantization units. However, this disclosure is not limited to this, and block units may be defined separately for other components. Although the shape of each of the inputs and outputs in each component is described as non-square in this disclosure, the inputs and outputs of some components may have any other shape (e.g., non-square triangles).

The size and shape of the initial (or starting) block in the block partitioning unit can be determined from the higher unit. The initial block can be divided into smaller blocks. Once the optimal size and shape are determined based on the block partitioning, that block can be designated as the initial block for the lower unit. The higher unit can be a coding block, and the lower unit can be a prediction block or a transform block, and this disclosure is not limited thereto. Instead, various modifications are possible. Once the initial block of the lower unit is determined as in the example above, a partitioning process can be performed to detect blocks of optimal size and shape, similar to the higher unit.

In summary, a block partitioning unit can divide a basic coding block (or the largest coding block) into at least one coding block, and a coding block can be divided into at least one prediction block/transform block/quantization block. Furthermore, a prediction block can be divided into at least one transform block/quantization block, and a transform block can be divided into at least one quantization block. Some blocks may have dependencies on other blocks (i.e., defined by higher and lower units) or may be independent of other blocks. For example, a prediction block may be a higher unit above a transform block, or it may be a unit independent of the transform block. Various relationships can be established based on the type of block.

Depending on the encoding settings, it can be determined whether higher and lower units are combined. Combining units means that blocks of higher units undergo encoding processing by lower units (e.g., in prediction units, transform units, inverse transform units, etc.) without being split into lower units. In other words, this may mean sharing the partitioning process among multiple units and generating partitioning information in one unit (e.g., a higher unit).

For example, (when a coded block is combined with a prediction block or a transform block), the coded block can undergo prediction, transform, and inverse transform.

For example, (when a coded block is combined with a prediction block), the coded block can undergo prediction, and a transform block that is equal to or smaller in size than the coded block can undergo both transformation and inverse transformation.

For example, (when a coded block is combined with a transform block), a prediction block that is equal to or smaller in size than the coded block can be predicted, and the coded block can be transformed and inverse transformed.

For example, (when a prediction block is combined with a transform block), a prediction block that is equal to or smaller in size than a coding block can undergo prediction, transform, and inverse transform.

For example, (when there is no block combination), a prediction block that is equal to or smaller than the coded block in size can be predicted, and a transform block that is equal to or smaller than the coded block in size can be transformed and inverse transformed.

Although various cases of coded blocks, prediction blocks, and transform blocks have been described in the examples above, this disclosure is not limited thereto.

For the combination of units, a fixed configuration can be supported in the image, or an adaptive configuration can be supported considering various coding factors. Coding factors include image type, color components, coding mode (intra-frame/inter-frame), partitioning configuration, block size/shape/position, aspect ratio, prediction-related information (e.g., intra-frame prediction mode, inter-frame prediction mode, etc.), transform-related information (e.g., transform scheme selection information, etc.), quantization-related information (e.g., quantization region selection information and quantization transform coefficient coding information), etc.

Once a block of optimal size and shape has been detected as described above, pattern information (e.g., partitioning information) for that block can be generated. This pattern information, along with information generated from the components to which the block belongs (e.g., prediction-related information and transformation-related information), can be included in the bitstream and sent to the decoder, where it can be parsed at the same unit level for use in the video decoding process.

The partitioning scheme will now be described. Although for ease of description, it is assumed that the initial blocks are shaped as squares, this disclosure is not limited thereto, and the description can be applied in the same or similar manner to cases where the initial blocks are not squares.

The block partitioning unit can support various types of partitions. For example, it can support tree-based partitioning or index-based partitioning, and other methods as well. In tree-based partitioning, the partition type can be determined based on various types of information (e.g., information indicating whether to perform partitioning, tree type, partition direction, etc.), while in index-based partitioning, specific index information can be used to determine the partition type.

Figure 4 is an example diagram showing the various partition types that can be obtained in the block partitioning unit of this disclosure.

In this example, it is assumed that the partition type shown in Figure 4 was obtained through a single partition operation (or procedure), which should not be construed as limiting the scope of this disclosure. The partition type may also be obtained through multiple partition operations. Furthermore, other partition types not shown in Figure 4 may also be available.

(Tree-based partitioning)

In the tree-based partitioning of this disclosure, QT, BT, and TT can be supported. If one tree method is supported, this can be called single-tree partitioning, while if two or more tree methods are supported, this can be called multi-tree partitioning.

In QT, a block is divided into two partitions (i.e., 4 partitions) in each of the horizontal and vertical directions (n), while in BT, a block is divided into two partitions (b to g) in either the horizontal or vertical direction. In TT, a block is divided into three partitions (h to m) in either the horizontal or vertical direction.

In Qt, a block can be divided into four partitions (o and p) by restricting the partitioning direction to either the horizontal or vertical direction. Furthermore, in BT, it supports partitioning the block into only equal-sized partitions (b and c), partitioning the block into only partitions of different sizes (d to g), or both of these partitioning types. Additionally, in TT, it supports partitioning the block into partitions focused only on a specific direction (1:1:2 or 2:1:1 in the left-to-right or top-to-bottom direction) (h, j, k, and m), partitioning the block into partitions focused on the center (1:2:1) (i and l), or both of these partitioning types. Furthermore, it supports partitioning the block into four partitions in each of the horizontal and vertical directions (i.e., a total of 16 partitions) (q).

The tree method supports partitioning a block into z partitions (b, d, e, h, i, j, o) only horizontally, or into z partitions (c, f, g, k, l, m, p) only vertically, or both partition types. In this paper, z can be an integer equal to or greater than 2, for example, 2, 3, or 4.

In this disclosure, it is assumed that partition type n is supported as QT, partition types b and c are supported as BT, and partition types i and l are supported as TT.

Depending on the encoding settings, one or more tree partitioning schemes can be supported. For example, QT, QT/BT, or QT/BT/TT can be supported.

In the examples above, the basic tree partitioning scheme is QT, and BT and TT are included as additional partitioning schemes depending on whether other trees are supported. However, various modifications can be made. Information indicating whether other trees are supported (bt_enabled_flag, tt_enabled_flag, and bt_tt_enabled_flag, where 0 indicates no support and 1 indicates support) can be implicitly determined based on encoding settings or explicitly determined in units such as sequences, images, sub-images, slices, tiles, blocks, etc.

Partition information may include indications of whether partitioning is performed (tree_part_flag or qt_part_flag, bt_part_flag, tt_part_flag, and bt_tt_part_flag, which can have values of 0 or 1, where 0 indicates no partitioning and 1 indicates partitioning). Additionally, depending on the partitioning scheme (BT and TT), information about the partitioning direction can be added (dir_part_flag, or bt_dir_part_flag, tt_dir_part_flag, and bt_tt_dir_part_flag, which have values of 0 or 1, where 0 indicates <width/horizontal> and 1 indicates <height/vertical>). This information can be generated during the partitioning process.

When multi-tree partitioning is supported, various partition information fragments can be configured. For ease of description, the following description provides an example of how to configure partition information at a depth level (that is, recursive partitioning can be performed by setting one or more supported partition depths).

In example (1), check the information indicating whether to perform partitioning. If partitioning is not performed, the partitioning process ends.

If partitioning is performed, the selection information regarding the partition type is checked (e.g., tree_idx, 0 for QT, 1 for BT, and 2 for TT). The partition direction information is additionally checked based on the selected partition type, and the process proceeds to the next step (if additional partitioning is feasible due to reasons such as the partition depth not yet reaching its maximum value, the process restarts from the beginning; if additional partitioning is not possible, the partitioning process ends).

In Example (2), the system checks whether the partitioning is performed using a certain tree scheme (QT), and the process proceeds to the next step. If the partitioning is not performed using a tree scheme (QT), the system checks whether the partitioning is performed using another tree scheme (BT). In this case, if the partitioning is not performed using that tree scheme, the system checks whether the partitioning is performed using a third tree scheme (TT). If the partitioning is not performed using a third tree scheme (TT), the partitioning process ends.

If partitioning is performed using a tree-based scheme (QT), the process proceeds to the next step. Alternatively, if partitioning is performed using a second-tree scheme (BT), the partition orientation information is checked, and the process proceeds to the next step. If partitioning is performed using a third-tree scheme (TT), the partition orientation information is checked, and the process proceeds to the next step.

In Example (3), the system checks whether to perform partitioning using a tree scheme (QT). If partitioning is not performed using a tree scheme (QT), the system checks whether to perform partitioning using another tree scheme (BT and TT). If partitioning is not performed, the partitioning process ends.

If partitioning is performed using a tree scheme (QT), the process proceeds to the next step. Alternatively, if partitioning is performed using other tree schemes (BT and TT), the partition orientation information is checked, and the process proceeds to the next step.

While the examples above either prioritize the tree partitioning scheme (Examples 2 and 3) or do not prioritize it (Example 1), various modifications are also possible. Furthermore, in the examples above, the partitioning in the current step is independent of the partitioning result of the previous step. However, it is also possible to configure the partitioning in the current step to depend on the partitioning result of the previous step.

In Examples 1 through 3, if some tree partitioning scheme (QT) was executed in a previous step, and the process has thus reached the current step, the same tree partitioning scheme (QT) can also be supported in the current step.

On the other hand, if a certain tree partitioning scheme (QT) was not executed in a previous step and therefore another tree partitioning scheme (BT or TT) was executed, and then the process proceeds to the current step, it can be configured to support other tree partitioning schemes (BT and TT) besides a certain tree partitioning scheme (QT) in the current step and subsequent steps.

In the above cases, the tree configuration supporting block partitioning can be adaptive, and therefore the partitioning information described above can also be configured differently. (Assuming the example described below is Example 3). That is, if partitioning was not performed using a certain tree scheme (QT) in the previous steps, the partitioning process can be performed without considering the tree scheme (QT) in the current step. Furthermore, partitioning information associated with a particular tree scheme (e.g., information indicating whether to perform partitioning, information about the partitioning direction, etc. In this example <QT>, information indicating whether to perform partitioning) can be removed.

The examples above illustrate adaptive partitioning information configuration for cases where block partitioning is allowed (e.g., block size is within the range of maximum and minimum values, partition depth of each tree scheme has not reached the maximum depth (allowed depth), etc.). Even when block partitioning is restricted (e.g., block size does not exist within the range of maximum and minimum values, partition depth of each tree scheme has reached the maximum depth, etc.), partitioning information can still be configured adaptively.

As already mentioned, tree-based partitioning can be performed recursively in this disclosure. For example, if the partition flag of a coded block with partition depth k is set to 0, then block encoding is performed in the coded block with partition depth k. If the partition flag of a coded block with partition depth k is set to 1, then block encoding is performed in N sub-coded blocks with partition depth k+1 according to the partitioning scheme (where N is an integer equal to or greater than 2, such as 2, 3, and 4).

In the above process, a sub-coded block can be set as a coded block (k+1) and partitioned into sub-coded blocks (k+2). This hierarchical partitioning scheme can be determined based on partitioning configurations such as partition range and allowed partition depth.

In this case, the bitstream structure representing the partition information can be selected from one or more scanning methods. For example, the bitstream of partition information can be configured based on the order of partition depth or based on whether partitioning was performed.

For example, in the case of partition depth order, partition information is obtained at the current depth level based on the initial block, and then at the next depth level. In the case of whether partitioning is performed, additional partition information is first obtained from the blocks split from the initial block, and other additional scanning methods can be considered.

Regardless of the tree type (or all trees), the maximum block size and minimum block size can have common settings, separate settings for each tree, or common settings for two or more trees. In this case, the maximum block size can be set to be equal to or less than the maximum encoded block size. If the maximum block size of the first tree is not the same as the maximum block size, partitioning is implicitly performed using a predetermined second-tree method until the maximum block size of the first tree is reached.

Furthermore, regardless of the tree type, a common split depth can be supported; a separate split depth can be supported for each tree; or a common split depth can be supported for two or more trees. Alternatively, split depth can be supported for some trees, while not for others.

It supports explicit syntax elements for setting information, and can also implicitly determine some setting information.

(Index-based partitioning)

In the index-based partitioning of this disclosure, both Constant Partition Index (CSI) and Variable Partition Index (VSI) schemes can be supported.

In the CSI scheme, k sub-blocks can be obtained by partitioning in a predetermined direction, where k can be an integer equal to or greater than 2, such as 2, 3, or 4. Specifically, the size and shape of the sub-blocks can be determined based on k, regardless of the size and shape of the blocks. The predetermined direction can be one or a combination of two or more of the following: horizontal, vertical, and diagonal (top left to bottom right or bottom left to top right).

In the index-based CSI partitioning scheme of this disclosure, z candidates can be obtained by partitioning in a horizontal or vertical direction. In this case, z can be an integer equal to or greater than 2, such as 2, 3, or 4, and the sub-blocks can be equal in width and height, and equal or different in the other of width and height. The width or height ratio of the sub-blocks is _A1 : _A2 : ... : _AZ , and each of _A1 to _AZ can be an integer equal to or greater than 1, such as 1, 2, or 3.

Furthermore, candidates can be obtained by partitioning the blocks horizontally and vertically into x and y partitions, respectively. Each of x and y can be an integer equal to or greater than 1, such as 1, 2, 3, or 4. However, candidates where both x and y are 1 may be restricted (because a already exists). Although Figure 4 shows a case where the sub-blocks have the same width or height ratio, candidates with different width or height ratios can also be included.

Furthermore, candidates can be divided into w partitions along one of the diagonal directions—top left to bottom right and bottom left to top right. In this paper, w can be an integer equal to or greater than 2, such as 2 or 3.

Referring to Figure 4, partition types can be classified into symmetric partition types (b) and asymmetric partition types (d and e) based on the length ratio of each sub-block. Furthermore, partition types can be classified into partition types concentrated in a specific direction (k and m) and central partition types (k). Partition types can be defined by various coding factors, including sub-block shape and sub-block length ratio, and the supported partition types can be implicitly or explicitly determined based on coding settings. Therefore, candidate groups can be determined based on the partition types supported in an index-based partitioning scheme.

In the VSI scheme, with a fixed width (w) or height (h) for each sub-block, one or more sub-blocks can be obtained through partitioning in a predetermined direction. In this document, each of w and h can be an integer equal to or greater than 1, such as 1, 2, 4, or 8. Specifically, the number of sub-blocks can be determined based on the size and shape of the block and the w or h value.

In the index-based VSI partitioning scheme of this disclosure, candidates can be partitioned into sub-blocks, each of which has a fixed width and length. Alternatively, candidates can be partitioned into sub-blocks, each of which has a fixed width and length. Since the width or height of the sub-blocks is fixed, equal partitions in either the horizontal or vertical direction are allowed. However, this disclosure is not limited thereto.

If the size of the block before partitioning is M×N, and the width (w) of each sub-block is fixed, the height (h) of each sub-block is fixed, or both the width (w) and height (h) of each sub-block are fixed, then the number of sub-blocks obtained can be (M*N)/w, (M*N)/h, or (M*N)/w/h.

Depending on the coding settings, only one or both of the CSI and VSI schemes may be supported, and information about the supported schemes may be determined implicitly or explicitly.

This disclosure will be described in the context of the supported CSI schemes.

Candidate groups can be configured to include two or more candidates from an index-based partitioning scheme, depending on the encoding settings.

For example, candidate groups such as {a, b, c}, {a, b, c, n}, or {a to g and n} can be formed. Candidate groups can include examples of block types predicted to occur multiple times based on general statistical characteristics, such as blocks divided into two partitions in the horizontal or vertical direction, or in each of the horizontal and vertical directions.

Alternatively, candidate groups such as {a, b}, {a, o}, or {a, b, o} can be configured, or candidate groups such as {a, c}, {a, p}, or {a, c, p}. A candidate group can be an example that includes candidates in which each candidate is partitioned into two partitions horizontally and four partitions vertically. This could be an example of configuring a candidate group with block types predicted to be primarily partitioned in a specific direction.

Alternatively, candidate groups such as {a, o, p} or {a, n, q} can be configured. This could be an example of configuring candidate groups to include block types that are predicted to be partitioned into many smaller partitions than the blocks before partitioning.

Alternatively, candidate groups such as {a, r, s} can be configured, and can be, for example, determining the best partitioning result of a non-square shape that can be obtained from the block before splitting by other methods (tree methods), and configuring the non-square shape as a candidate group.

As noted in the examples above, various candidate group configurations can be available, and one or more candidate group configurations can be supported, taking into account various encoding/decoding factors.

Once the candidate groups are fully configured, various partition information configurations can be made available.

For example, index selection information can be generated for a candidate group that includes unpartitioned candidates (a) and partitioned candidates (b to s).

Alternatively, information indicating whether to perform partitioning can be generated (information indicating whether the partition type is 'a'). If partitioning is performed (if the partition type is not 'a'), index selection information about the candidate group (b to s) including the candidates to be partitioned can be generated.

Partition information can be configured in many other ways than those described above. In addition to information indicating whether to perform a partition, binary bits can be assigned to the index of each candidate in the candidate group in various ways, such as fixed-length binarization, variable-length binarization, etc. If the number of candidates is 2, one bit can be assigned to the index selection information, and if the number of candidates is 3, one or more bits can be assigned to the index selection information.

Compared to tree-based partitioning schemes, partition types that are predicted to occur multiple times can be included in the candidate group in index-based partitioning schemes.

Since the number of bits used to represent index information can increase based on the number of supported candidate groups, this scheme is suitable for single-level partitioning (e.g., partition depth is limited to 0) rather than tree-based hierarchical partitioning (recursive partitioning). That is, single partitioning operations can be supported without further subdividing the sub-blocks obtained through index-based partitioning.

This may mean that further partitioning into smaller blocks of the same type is not possible (e.g., a coded block obtained through index-based partitioning cannot be further subdivided into coded blocks), and it also means that further partitioning into blocks of different types may also be impossible (e.g., it is not possible to partition a coded block into prediction blocks and coded blocks). Clearly, this disclosure is not limited to the above examples, and other modifications are also possible.

Now, a description will be given of determining the block partitioning configuration primarily based on the block type among the coding factors.

First, encoded blocks can be obtained during the partitioning process. A tree-based partitioning scheme can be used for the partitioning process, and partition types such as a (no partitioning), n (QT), b, c (BT), i, or l (TT) as shown in Figure 4 can be generated based on the tree type. Depending on the encoding configuration, various combinations of tree types, such as QT/QT+BT/QT+BT+TT, may be available.

The following example illustrates the process of dividing the coded blocks obtained in the above steps into prediction blocks and transform blocks. It assumes that prediction, transform, and inverse transform are performed based on the size of each partition.

In example (1), prediction can be performed by setting the size of the prediction block to be equal to the size of the coding block, and transformation and inverse transformation can be performed by setting the size of the transform block to be equal to the size of the coding block (or prediction block).

In example (2), prediction can be performed by setting the size of the prediction block to be equal to the size of the coding block. The transform block can be obtained by partitioning the coding block (or prediction block), and the transform and inverse transform can be performed based on the size of the obtained transform block.

Here, a tree-based partitioning scheme can be used for the partitioning process, and partition types such as a (no partitioning), n (QT), b, c (BT), i, or l (TT) as shown in Figure 4 can be obtained based on the tree type. Depending on the encoding configuration, various combinations of tree types, such as QT/QT+BT/QT+BT+TT, may be available.

Here, the partitioning process can be an index-based partitioning scheme. The partition type can be obtained based on the index type, for example, a (no partition), b, c, or d in Figure 4. Depending on the encoding configuration, various candidate groups such as {a, b, c} and {a, b, c, d} can be configured.

In example (3), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. For the transform block, its size is set to the size of the coded block, and a transform and inverse transform can be performed on the transform block. In this example, the prediction block and the transform block can be independent of each other.

Index-based partitioning schemes can be used in the partitioning process, and the partition type can be obtained based on the index type, such as a (no partition), b to g, n, r, or s in Figure 4. Various candidate groups can be configured according to the encoding configuration, such as {a, b, c, n}, {a to g, n}, and {a, r, s}.

In example (4), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. For the transform block, its size is set to the size of the prediction block, and a transform and inverse transform can be performed on the transform block. In this example, the transform block can have a size equal to the size of the obtained prediction block, or the obtained prediction block can have a size equal to the size of the transform block (setting the size of the transform block to the size of the prediction block).

Tree-based partitioning schemes can be used in the partitioning process, and partition types can be generated based on the tree type, such as a (no partition), b, c (BT), or n (QT) in Figure 4. Depending on the encoding configuration, various combinations of tree types, such as QT/BT/QT+BT, may be available.

Here, an index-based partitioning scheme can be used in the partitioning process, and the partition type can be obtained based on the index type, such as a (no partition), b, c, n, o, or p in Figure 4. Various candidate groups can be configured according to the encoding configuration, such as {a, b}, {a, c}, {a, n}, {a, o}, {a, p}, {a, b, c}, {a, o, p}, {a, b, c, n}, and {a, b, c, n, p}. Furthermore, candidate groups can be configured in a standalone VSI scheme or in a combination of CSI and VSI schemes as an index-based partitioning scheme.

In Example (5), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. Alternatively, a transform block can be obtained by partitioning the coded block, and the transform block undergoes transform and inverse transform based on the size of the obtained transform block. In this example, each of the prediction block and the transform block can be obtained by partitioning the coded block.

Here, tree-based partitioning schemes and index-based partitioning schemes can be used in the partitioning process, and candidate groups can be configured in the same way or in a similar manner as in Example 4.

In this context, the above examples represent possible scenarios depending on whether the process for partitioning each block type is shared, and should not be construed as limiting the scope of this disclosure. Various modifications are also possible. Furthermore, various encoding factors and block types can be considered to determine the block partitioning configuration.

Coding factors may include image type (I/P/B), color components (YCbCr), block size/shape/position, block aspect ratio, block type (coded block, prediction block, transform block, or quantization block), partition status, coding mode (intra-frame/inter-frame), prediction-related information (intra-frame prediction mode or inter-frame prediction mode), transform-related information (transform scheme selection information), and quantization-related information (quantization region selection information and quantization transform coefficient coding information).

In the image coding method according to an embodiment of the present invention, inter-frame prediction can be configured as follows. The inter-frame prediction of the prediction unit may include a reference image construction step, a motion estimation step, a motion compensation step, a motion information determination step, and a motion information coding step. Additionally, the video coding apparatus may include a reference image construction unit, a motion estimation unit, a motion compensation unit, a motion information determination unit, and a motion information coding unit that implement the reference image construction step, the motion estimation step, the motion compensation step, the motion information determination step, and the motion information coding step. Some of the above processes may be omitted, or other processes may be added, and the order may be changed from the above sequence.

In the video decoding method according to an embodiment of the present invention, inter-frame prediction can be configured as follows. The inter-frame prediction of the prediction unit may include a motion information decoding step, a reference image construction step, and a motion compensation step. Furthermore, the image decoding device may include a motion information decoding unit, a reference image construction unit, and a motion compensation unit that implement the motion information decoding step, the reference image construction step, and the motion compensation step. Some of the above processes may be omitted, or other processes may be added, and the order may be changed from the above order.

Since the reference image construction unit and motion compensation unit of the image decoding device perform the same functions as their counterparts in the image encoding device, detailed descriptions are omitted. Furthermore, the motion information decoding unit can be reverse-engineered using the methods employed in the motion information encoding unit. Here, the prediction block generated by the motion compensation unit can be sent to the addition unit.

Figure 5 is an example diagram illustrating various cases of obtaining prediction blocks through inter-frame prediction according to the present invention.

Referring to Figure 5, in unidirectional prediction, prediction block A can be obtained from previously encoded reference images (T-1, T-2) (forward prediction), or prediction block B can be obtained from later encoded reference images (T+1, T+2) (backward prediction). In bidirectional prediction, prediction blocks (C and D) can be generated from multiple previously encoded reference images (T-2 to T+2). Typically, P-image type supports unidirectional prediction, and B-image type supports bidirectional prediction.

As in the example above, the images referenced for the encoding of the current image can be obtained from memory, and a list of reference images can be constructed based on the current image (T) in chronological or display order, including reference images before and after the current image.

Inter-frame prediction (E) can be performed on the current image and on previous or subsequent images based on the current image. Performing inter-frame prediction on the current image can be referred to as non-directional prediction. This can be supported by either I-image type or P/B-image type, and the supported image type can be determined based on the coding settings. Performing inter-frame prediction in the current image uses spatial correlation to generate prediction blocks. Performing inter-frame prediction in other images for the purpose of using temporal correlation is different, but the prediction method (e.g., reference image, motion vector, etc.) may be the same.

Here, it is assumed that P-image and B-image are image types capable of performing inter-frame prediction, but this can also be applied to various image types that are added or replaced. For example, a particular image type may not support intra-frame prediction, but may only support inter-frame prediction, or may only support inter-frame prediction in a predetermined direction (backward direction), or may only support inter-frame prediction in a predetermined direction.

The reference image construction unit can construct and manage reference images for encoding the current image using a reference image list. At least one reference image list can be constructed based on encoding settings (e.g., image type, prediction orientation, etc.), and prediction blocks can be generated from the reference images included in the reference image list.

In the case of unidirectional prediction, inter-frame prediction can be performed on at least one reference image included in reference image list 0 (L0) or reference image list 1 (L1). Additionally, in the case of bidirectional prediction, inter-frame prediction can be performed on at least one reference image included in a combined list (LC) generated by combining L0 and L1.

For example, one-way prediction can be classified as forward prediction (Pred_L0) using a forward reference image list (L0) and backward prediction (Pred_L1) using a backward reference image list (L1). Two-way prediction (Pred_BI) can use both a forward reference image list (L0) and a backward reference image list (L1).

Alternatively, two or more forward predictions performed by copying the forward reference image list (L0) to the backward reference image list (L1) can be included in the bidirectional prediction, and two or more backward predictions performed by copying the backward reference image list (L0) to the forward reference image list (L1) can also be included in the bidirectional prediction.

The prediction direction can be indicated by flag information indicating the corresponding direction (e.g., inter_pred_idc, assuming this value can be adjusted by predFlagL0, predFlagL1, and predFlagBI). predFlagL0 indicates whether forward prediction is performed, and predFlagL1 indicates whether backward prediction is performed. Bidirectional prediction can be indicated by whether the prediction is performed via predFlagBI or by activating both predFlagL0 and predFlagL1 simultaneously (e.g., when each flag is 1).

This invention primarily describes the case of omnidirectional prediction using a list of omnidirectional reference images, but the same or modified application can be applied to other situations.

Typically, the following approach can be used: determine the optimal reference image for the image to be encoded by the encoder, and explicitly send information about the reference image to the decoder. To this end, the reference image construction unit can manage the list of images referenced for inter-frame prediction of the current image, and can set rules for reference image management considering limited memory size.

The transmitted information can be defined as a Reference Image Set (RPS), where images selected in the RPS are classified as reference images and stored in memory (or a Data Point Block (DPB), while images not selected in the RPS are classified as non-reference images. Images may be deleted from memory after one hour. A predetermined number of images (e.g., 14, 15, 16, or more) can be stored in memory, and the size of the memory can be set according to the image level and resolution.

Figure 6 is an example diagram of constructing a list of reference pictures according to an embodiment of the present invention.

Referring to Figure 6, typically, reference images (T-1, T-2) existing before the current image can be assigned to L0 and managed, and reference images (T+1, T+2) existing after the current image can be assigned to L1 and managed. When constructing L0, if the number of reference images for L0 is insufficient, reference images from L1 can be assigned. Similarly, when constructing L1, if the number of reference images for L1 is insufficient, reference images from L0 can be assigned.

Furthermore, the current image can be included in at least one list of reference images. For example, L0 or L1 can include the current image, and L0 can be constructed by adding a reference image (or the current image) with a chronological order of T to a list of reference images preceding the current image, while L1 can be constructed by adding a reference image with a chronological order of T to a list of reference images following the current image.

The construction of the reference image list can be determined based on the encoding settings.

The current image may not be included in the reference image list, but may be managed through a separate memory separate from the reference image list, or the current image may be included in at least one reference image list and managed thereafter.

For example, this can be determined using a signal (curr_pic_ref_enabled_flag) that indicates whether the current image is included in the list of reference images. Here, the signal can be implicitly determined or explicitly generated.

Specifically, when the signal is deactivated (e.g., curr_pic_ref_enabled_flag = 0), the current image may not be included as a reference image in any list of reference images. When the signal is activated (e.g., curr_pic_ref_enabled_flag = 1), whether to include the current image in a predetermined list of reference images can be implicitly determined (e.g., added only to L0, added only to L1, or added to both L0 and L1) or the relevant signal can be explicitly generated and determined (e.g., curr_pic_ref_from_l0_flag, curr_pic_ref_from_l1_flag). This signal can be supported in units such as sequences, images, sub-images, slices, tiles, and blocks.

Here, the current image can be positioned either first or last in the list of reference images shown in Figure 6, and the arrangement order in the list can be determined according to encoding settings (e.g., image type information). For example, type I can be positioned first, and type P/B can be positioned last, but the invention is not limited thereto, and other modified examples are also possible.

Alternatively, a separate reference image store can be supported based on a signal (ibc_enabled_flag) indicating whether block matching (or template matching) is supported in the current image. Here, the signal can be implicitly determined information or explicitly generated information.

Specifically, when the signal is deactivated (e.g., ibc_enabled_flag = 0), this may mean that block matching is not supported in the current image, and when the signal is activated (e.g., ibc_enabled_flag = 1), block matching can be supported in the current image, and reference image memory for this purpose can be supported. In this example, it is assumed that additional memory is provided, but it is also possible to configure block matching to be supported directly from existing memory supported by the current image without providing additional memory.

The reference image construction unit may include a reference image interpolation unit, and whether to perform interpolation processing on fractional-unit pixels can be determined based on the interpolation precision of the inter-frame prediction. For example, when the interpolation precision is in integer units, reference image interpolation processing can be omitted, and when the interpolation precision is in fractional units, reference image interpolation processing can be performed.

The interpolation filters used in the reference image interpolation process can be implicitly determined based on the encoding settings, or explicitly determined from among multiple interpolation filters. Depending on the encoding settings, the configuration of multiple interpolation filters can support fixed candidates or adaptive candidates, and the number of candidates can be an integer of 2, 3, 4 or more. The explicitly determined unit can be determined from a sequence, image, sub-image, slice, tile, block, or chunk.

Here, the encoding settings can be determined based on image type, color components, target block state information (e.g., block size, shape, horizontal/vertical length ratio, etc.), and inter-frame prediction settings (e.g., motion information encoding mode, motion model selection information, motion vector precision selection information, reference image, reference orientation, etc.). Motion vector precision can mean the precision of the motion vectors (i.e., pmv + mvd), but it can be replaced by the precision of the motion vector prediction values (pmv) or the motion vector difference (mvd).

Here, the interpolation filter can have a filter length with k taps, and k can be an integer of 2, 3, 4, 5, 6, 7, 8, or more. The filter coefficients can be derived from equations with various coefficient characteristics, such as Wiener and Kalman filters. The filter information used for interpolation (e.g., filter coefficients, tap information, etc.) can be implicitly determined or derived, or the relevant information can be explicitly generated. In this case, the filter coefficients can be configured to include 0.

Interpolation filters can be applied to a predetermined pixel unit, and the predetermined pixel unit can be restricted to an integer or fractional unit, or can be applied to both integer and fractional units.

For example, an interpolation filter can be applied to k integer unit pixels that are adjacent to the interpolation target pixel (i.e., fractional unit pixels) in the horizontal or vertical direction.

Alternatively, the interpolation filter can be applied to p integer units and q fractional units (p+q=k) adjacent to the interpolation target pixel in the horizontal or vertical direction. In this case, the precision referred to as the fractional unit used for interpolation (e.g., in 1/4 units) can be expressed as the same as or lower than the interpolation target pixel (e.g., 2/4 -> 1/2).

When only one of the x-component and y-component of the interpolated target pixel lies within a fractional unit, interpolation can be performed based on k pixels adjacent in the direction of the fractional component (e.g., the x-axis is horizontal and the y-axis is vertical). When both the x-component and y-component of the interpolated target pixel lie within a fractional unit, a first interpolation can be performed based on x pixels adjacent in either the horizontal or vertical direction, and interpolation can be performed based on y pixels adjacent in the other direction. In this case, the case where x and y are the same as k will be described, but the case is not limited to this, and it is also possible for x and y to be different.

The interpolated target pixel can be obtained through interpolation precision (1/m), where m can be an integer of 1, 2, 4, 8, 16, 32, or higher. The interpolation precision can be implicitly determined based on encoding settings, or it can be explicitly generated based on relevant information. Encoding settings can be limited based on image type, color components, reference image, motion information encoding mode, motion model selection information, etc. The explicitly determined unit can be determined from a sequence, sub-image, slice, tile, or block.

Through the above processing, the interpolation filter settings for the target block can be obtained, and reference image interpolation can be performed based on them. Furthermore, detailed interpolation filter settings based on these settings can be obtained, and reference image interpolation can be performed based on them. That is, while it's possible to obtain the interpolation filter settings for the target block using a fixed candidate, it's also possible to use multiple candidates. In the example described later, it's assumed that the interpolation precision is 1/16 (e.g., 15 pixels to be interpolated).

As an example of setting up detailed interpolation filters, one of a plurality of candidates can be adaptively used for the first pixel unit at a predetermined position, and a predetermined interpolation filter can be used for the second pixel unit at the predetermined position.

The first pixel unit can be determined among all fractional units of pixels supported by interpolation precision (1/16 to 15/16 in this example), and the number of pixels included in the first pixel unit can be a, and a can be determined between 0, 1, 2, ..., (m-1).

The second pixel unit may include pixels excluded from the first pixel unit from the total number of pixels in the fractional units, and the number of pixels included in the second pixel unit can be obtained by subtracting the number of pixels in the first pixel unit from the total number of pixels to be interpolated. In this example, an example of dividing the pixel unit into two will be described, but the invention is not limited thereto and may divide it into three or more.

For example, the first pixel unit can be a unit represented by multiple units such as 1/2, 1/4, 1/8. For example, in the case of a 1/2 unit, it can be a pixel located at 8/16, and in the case of a 1/4 unit, it can be a pixel located at {4/16, 8/16, 12/16}, and in the case of a 1/8 unit, it can be a pixel located at {2/16, 4/16, 6/16, 8/16, 10/16, 12/16, 14/16}.

Detailed interpolation filter settings can be determined based on the coding settings, which can be limited by image type, color components, target block state information, inter-frame prediction settings, etc. Below are examples of detailed interpolation filter settings based on various coding elements. For ease of explanation, assume a is 0 and c is or greater (c is an integer greater than or equal to 1).

For example, in the case of color components (luminance, chrominance), a can be (1,0), (0,1), (1,1). Alternatively, in the case of motion information encoding mode (merging mode, competing mode), a can be (1,0), (0,1), (1,1), (1,3). Or, in the case of motion model selection information (translational motion, non-translational motion A, non-translational motion B), a can be (0,1,1), (1,0,0), (1,1,1), (1,3,7). Alternatively, in the case of motion vector precision (1/2, 1/4, 1/8), a can be (0,0,1), (0,1,0), (1,0,0), (0,1,1), (1,0,1), (1,1,0), (1,1,1). Alternatively, in the case of reference images (current image, other images), a can be (0,1), (1,0), (1,1).

As described above, interpolation can be performed by selecting one of several interpolation precisions, and when interpolation based on adaptive interpolation precision is supported (e.g., adaptive_ref_resolution_enabled_flag. If it is 0, a predetermined interpolation precision is used, and if it is 1, one of several interpolation precisions is used), precision selection information (e.g., ref_resolution_idx) can be generated.

Motion estimation and compensation can be performed based on interpolation accuracy, and the representation and storage units for motion vectors can also be determined based on interpolation accuracy.

For example, when the interpolation precision is 1/2 unit, motion estimation and compensation processing can be performed at 1/2 unit, and motion vectors can be represented at 1/2 unit and used in the encoding process. Furthermore, motion vectors can be stored at 1/2 unit and can be referenced in the encoding process of another block of motion information.

Alternatively, when the interpolation precision is 1/8, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/8 units and used in the encoding process and stored in 1/8 units.

Additionally, motion estimation and compensation processing, as well as motion vectors, can be performed, represented, and stored in units different from interpolation precision, such as integers, 1/2 units, and 1/4 units. These units can be adaptively determined based on inter-frame prediction methods/settings (e.g., motion estimation/compensation methods, motion model selection information, motion information coding modes, etc.).

As an example, assuming an interpolation precision of 1/8, in the case of a translational motion model, motion estimation and compensation can be performed in 1/4 units, and motion vectors can be represented in 1/4 units (assuming units in the encoding process in this example), and stored in 1/8 units. In the case of a non-translational motion model, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/4 units, and stored in 1/8 units.

For example, assuming an interpolation precision of 1/8 unit, in the case of block matching, motion estimation and compensation can be performed at 1/4 unit, and motion vectors can be represented at 1/4 unit and stored at 1/8 unit. In the case of template matching, motion estimation and compensation processing can be performed at 1/8 unit, and motion vectors can be represented at 1/8 unit and stored at 1/8 unit.

For example, assuming an interpolation precision of 1/16, in race mode, motion estimation and compensation processing can be performed in 1/4 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units. In merge mode, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units. In skip mode, motion estimation and compensation can be performed in 1/16 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units.

In summary, the units for motion estimation, motion compensation, motion vector representation, and storage can be adaptively determined based on the inter-frame prediction method or settings and the interpolation accuracy. Specifically, the units for motion estimation, motion compensation, and motion vector representation can be adaptively determined based on the inter-frame prediction method or settings, and the storage unit for motion vectors is typically determined based on the interpolation accuracy, but is not limited to this, and various modified examples are feasible. Furthermore, the examples above have illustrated an example based on one category (e.g., motion model selection information, motion estimation/compensation method, etc.), but settings can also be determined when two or more categories are mixed.

Furthermore, as mentioned above, the interpolation accuracy information can have a predetermined value or can be selected as one of multiple accuracies, and the reference image interpolation accuracy is determined based on the motion estimation and compensation settings supported by the inter-frame prediction method or settings. For example, when the translational motion model supports up to 1/8 unit and the non-translational motion model supports up to 1/16 unit, interpolation can be performed based on the accuracy unit of the non-translational motion model with the highest accuracy.

In other words, reference image interpolation can be performed based on settings for the accuracy information used to support it—such as translational motion model, non-translational motion model, competition mode, merging mode, and skip mode. In this case, the accuracy information can be determined implicitly or explicitly, and when the relevant information is explicitly generated, it can be included in units such as sequences, images, sub-images, slices, tiles, and blocks.

A motion estimation unit refers to the process of estimating (or searching) whether a target block has a high correlation with a predetermined block of a predetermined reference image. The size and shape (M×N) of the target block—on which prediction is performed—can be obtained from the block partitioning unit. As an example, the target block can be determined in the range of 4×4 to 128×128. Typically, inter-frame prediction can be performed on a block-by-block basis, but it can be performed on a block-by-block basis, such as a coded block or a transform block, depending on the block partitioning unit settings. Estimation can be performed within an estimable range of the reference region, and at least one motion estimation method can be used. In the motion estimation method, the estimation order and conditions can be defined for each pixel.

Motion estimation can be performed based on motion estimation methods. For example, in the case of block matching, the region to be compared for motion estimation processing can be the target block, and in the case of template matching, it can be a predetermined region (template) set around the target block. In the former case, the block with the highest relevance can be found within the estimable range of the target block and the reference region, and in the latter case, the region with the highest relevance can be found within the estimable range of the template defined according to the encoding settings and the reference region.

Motion estimation can be performed based on motion models. Additional motion models, besides translational motion models that only consider parallel motion, can be used to perform motion estimation and compensation. For example, motion models that consider motions such as rotation, perspective, zoom in/out, and parallel motion can be used to perform motion estimation and compensation. This can be supported to improve coding performance by generating prediction blocks that reflect the various types of motions occurring based on the regional features of the image.

Figure 7 is a conceptual diagram illustrating a non-translational motion model according to an embodiment of the present invention.

Referring to Figure 7, as an example of an affine model, an example is shown where motion information is represented based on motion vectors _V0 and _V1 at predetermined positions. Since motion can be represented based on multiple motion vectors, accurate motion estimation and compensation are possible.

As in the example above, inter-frame prediction is performed based on a predefined motion model, but inter-frame prediction based on an additional motion model can also be supported. Here, it is assumed that the predefined motion model is a translational motion model and the additional motion model is an affine model, but the invention is not limited to this, and various modifications are possible.

In the case of a translational motion model, motion information (assuming unidirectional prediction) can be represented based on a motion vector, and the control point (reference point) used to indicate the motion information is assumed to be the upper left coordinate, but is not limited to this.

In the case of a non-translational motion model, it can be represented as motion information for various configurations. In this example, it is assumed that the configuration is represented as additional information in a motion vector (relative to the top-left corner coordinates). Some motion estimation and compensation mentioned in the examples described later can be performed not on a block-by-block basis, but on a predetermined sub-block basis. In this case, the size and position of the predetermined sub-blocks can be determined based on each motion model.

Figure 8 is an example diagram illustrating motion estimation on a sub-block basis according to an embodiment of the present invention. Specifically, it illustrates motion estimation on a sub-block basis based on an affine model (two motion vectors).

In the case of a translational motion model, the motion vectors of the pixel units included in the target block can be the same. That is, the motion vectors can be applied to the pixel units together, and a single motion vector ( _V0 ) can be used to perform motion estimation and compensation.

In the case of a non-translational motion model (affine model), the motion vectors of the pixel units included in the target block may not be the same, and a separate motion vector may be required for each pixel. In this case, the motion vectors of the pixel units or sub-block units can be derived based on the motion vectors ( _V0 , _V1 ) at the positions of predetermined control points of the target block, and the derived motion vectors can be used to perform motion estimation and compensation.

For example, the motion vector of a sub-block or pixel unit within the target block (e.g., ( _Vx , _Vy )) can be derived from the equations _Vx = ( _V1x - _V0x )×x/M - ( _V1y - _V0y )×y/N + _V0x , _Vy = ( _V1y - _V0y )×x/M + ( _V1x - _V0x )×y/N + _V0y . In these equations, _V0 (in this example, ( _V0x , _V0y )) refers to the motion vector at the top left of the target block, and _V1 (in this example, ( _V1x , _V1y )) refers to the motion vector at the top right of the target block. Considering the complexity, motion estimation and compensation for a non-translational motion model can be performed on a sub-block basis.

Here, the size of the sub-block (M×N) can be determined according to the encoding settings and can have a fixed size or can be set to an adaptive size. M and N can be integers of 2, 4, 8, 16, or more, and M and N can be the same or different. The size of the sub-block can be explicitly generated in units such as sequences, pictures, sub-pictures, slices, tiles, and blocks. Alternatively, it can be implicitly determined by a mutual commitment between the encoder and decoder, or it can be determined by the encoding settings.

Here, the encoding settings can be defined by one or more elements, including state information, image type, color components, and inter-frame prediction settings for the target block (motion information encoding mode, reference image information, interpolation accuracy, motion model selection information, etc.).

In the examples above, the process of deriving the size of the sub-blocks based on a predetermined non-translational motion model and performing motion estimation and compensation accordingly has been described. As in the examples above, motion estimation and compensation can be performed on a sub-block or pixel-by-pixel basis according to the motion model, and a detailed description of this will be omitted.

The following are various examples of motion information constructed based on motion models.

For example, in the case of a motion model representing rotational motion, the translational motion of a block can be represented by a motion vector, and the rotational motion can be represented by rotation angle information. The rotation angle information can be measured relative to a predetermined position (e.g., the top-left coordinate) and can be represented as k candidates (k is an integer of 1, 2, 3 or more) with predetermined intervals (e.g., angle differences of 0 degrees, 11.25 degrees, 22.25 degrees, etc.) within a predetermined angle range (e.g., between -90 degrees and 90 degrees).

Here, the rotation angle information can be encoded as is during motion information encoding processing, or it can be encoded based on the motion information of adjacent blocks (e.g., motion vectors, rotation angle information) (e.g., prediction + difference information).

Alternatively, the translational motion of a block can be represented by a single motion vector, and the rotational motion of a block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vector can be encoded as is during motion information encoding processing, or encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational motion models or non-translational motion models) (e.g., prediction + difference information), or encoded based on another motion vector in a block representing rotational motion (e.g., prediction + difference information).

For example, in the case of a motion model representing sizing or scaling movements such as zooming in/out, the translational motion of a block can be represented by a single motion vector, and the scaling motion can be represented by scaling information. Scaling information can be represented as scaling information indicating expansion or reduction in the horizontal or vertical direction based on a predetermined position (e.g., top-left coordinate).

Here, scaling can be applied in at least one of the horizontal or vertical directions. Alternatively, it can support individual scaling information applied in the horizontal or vertical directions, or scaling information applied in a common direction. The width and height of the scaling block can be added to a predetermined position (top-left coordinate) to determine the location used for motion estimation and compensation.

Here, scaling information can be encoded as is during motion information encoding processing, or it can be encoded based on motion information of neighboring blocks (e.g., motion vectors, scaling information) (e.g., prediction + difference information).

Alternatively, the translation of a block can be represented by a single motion vector, and the size adjustment of the block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vectors can be encoded as is during motion information encoding processing, or they can be encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational motion models or non-translational motion models) (e.g., prediction + difference information), or they can be encoded based on predetermined coordinates within the block (e.g., lower right coordinates) (e.g., prediction + difference).

In the examples above, the representation of some motions has been described, and it can be represented as motion information for multiple motions.

For example, in the case of motion models representing various or complex motions, the translational motion of a block can be represented by a motion vector, the rotational motion can be represented by rotation angle information, and the sizing adjustment can be represented by scaling information. Since the description of each motion can be derived from the above examples, detailed descriptions will be omitted.

Alternatively, the translational motion of a block can be represented by a single motion vector, and other motions of the block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vectors can be encoded as is during motion information encoding processing, or they can be encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational or non-translational motion models) (e.g., prediction + difference information), or they can be encoded based on other motion vectors within blocks representing various motions (e.g., prediction + difference information).

The above description may pertain to affine models and will be based on the presence of one or two additional motion vectors. In summary, it is assumed that the number of motion vectors used according to the motion model can be 1, 2, or 3, and that a single motion model can be considered based on the number of motion vectors used to represent motion information. Furthermore, when only one motion vector exists, it is assumed to be a predefined motion model.

Multiple motion models can be supported for inter-frame prediction, and this can be determined by a signal indicating support for additional motion models (e.g., `adaptive_motion_mode_enabled_flag`). Here, a signal of 0 indicates support for predefined motion models, and a signal of 1 indicates support for multiple motion models. This signal can be generated in units such as sequences, images, sub-images, slices, tiles, blocks, chunks, etc., but if it is not possible to check them individually, the signal value can be assigned according to predefined settings. Alternatively, whether support is implicit can be determined based on encoding settings. Alternatively, the implicit or explicit case can be determined based on encoding settings. Here, encoding settings can be limited by one or more elements of image type, image class (e.g., a normal image if 0, a 360-degree image if 1), or color components.

Whether multiple motion models are supported can be determined through the process described above. The following assumes that two or more additional motion models are supported, and that multiple motion models are supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., but some configurations may be excluded. In the example described later, it is assumed that motion models A, B, and C are supported, where A is the fundamentally supported motion model, and B and C are additionally supported motion models.

Configuration information about supported motion models can be generated within the unit. That is, configurations for supported motion models such as {A, B}, {A, C}, and {A, B, C} are feasible.

For example, indices (0 to 2) can be assigned to candidates for the above configurations and selected. If index 2 is selected, a motion model configuration supporting {A, C} can be determined, and if index 3 is selected, a motion model configuration supporting {A, B, C} can be determined.

Alternatively, information indicating whether a predetermined motion model is supported can be supported individually. That is, a flag indicating whether B is supported or a flag indicating whether C is supported can be generated. If both flags are 0, then only A is supported. This example could be an instance where information indicating whether multiple motion models are supported is not generated and processed.

When candidate groups of supported motion models are configured as in the example above, the motion model of one of the candidate groups can be explicitly identified and used on a block-by-block basis, or it can be used implicitly.

Typically, a motion estimation unit can be a component present in the encoding device, but it can also be a component that can be included in the decoding device depending on the prediction method (e.g., template matching). For example, in the case of template matching, the decoder can obtain the motion information of the target block by performing motion estimation via a template adjacent to the target block. In this case, motion estimation related information (e.g., motion estimation range, motion estimation method <scan order>, etc.) can be implicitly determined or explicitly generated and included in units such as sequences, pictures, sub-pictures, slices, tiles, blocks, etc.

The motion compensation unit refers to the process of obtaining data of some blocks of a predetermined reference image determined by motion estimation processing as predicted blocks of a target block. Specifically, the predicted blocks of the target block can be generated from at least one region (or block) of at least one reference image based on motion information (e.g., reference image information, motion vector information, etc.) obtained through motion estimation processing.

Motion compensation can be performed based on the following motion compensation methods.

In the case of block matching, the predicted block of the target block can be compensated with data of the regions corresponding to the right M and down N relative to (P _{x + V x} , P _y + V _y ), which are explicitly obtained from the reference image and are obtained through the motion vector (V _x , V _y ) of the target block (M _{× N} ) and the top-left coordinates (P _x , P _y ) of the target block.

In the case of template matching, the predicted block of the target block can be compensated with data of the regions corresponding to the right M and down N relative to (P _{x + V x} , P _y + V _y ), which are implicitly obtained from the reference image and are obtained through the motion vector (V _x , V _y ) of the target block (M _{× N} ) and the top-left coordinates (P _x , P _y ) of the target block.

Alternatively, motion compensation can be performed based on the following motion model.

In the case of the translational motion model, the predicted block of the target block can be compensated with data of the regions corresponding to the _{right M} and down N relative to ( _Px + _Vx , _Py + _Vy ), which are explicitly obtained from the reference image and are obtained by a motion vector ( _Vx , _Vy ) of the target block (M×N ₎ and the upper left coordinates ( _Px , _{Py )} of the target block.

In the case of a non-translational motion model, the predicted sub-blocks can be compensated using data from regions M to the right and N downwards relative to ( _Pmx + _Vnx , _Pmy + _Vny ). This ( _Pmx + _Vnx , _Pmy + _Vny ) is obtained by implicitly acquiring the motion vectors (Vmx, Vny) of m×n sub-blocks via multiple motion vectors ( _V0x , _V0y ) and ( _V1x , _V1y ) of the target block (M×N) explicitly obtained from the reference image _{, along with the top-left coordinates (Pmx , Pny} ) of each sub-block. In other words, the predicted sub-blocks can be collected and compensated for by _the predicted sub-blocks of the target block.

The motion information determination unit can perform processing for selecting the optimal motion information for the target block. Typically, the optimal motion information can be determined based on coding cost and rate distortion techniques using block distortion (e.g., distortion of the target block and the reconstructed block, such as SAD (sum of absolute differences), SSD (sum of squared differences), etc.), where the rate distortion technique considers the bit amount generated based on the corresponding motion information. The predicted block generated based on the motion information determined through the above processing can be sent to the subtraction and addition units. Alternatively, it can be configured according to some prediction methods (e.g., template matching, etc.) that can be included in the decoding device, and in this case, it can be determined based on the block distortion.

In the motion information determination unit, settings related to inter-frame prediction can be considered, such as motion compensation methods and motion models. For example, when multiple motion compensation methods are supported, the motion compensation method selection information, the corresponding motion vectors, and reference image information can be the optimal motion information. Alternatively, when multiple motion models are supported, the motion model selection information, its corresponding motion vectors, and reference image information can be the optimal motion information.

The motion information encoding unit can encode the motion information of the target block obtained through motion information determination processing. In this case, the motion information can consist of information about the region and image referenced for the prediction of the target block. Specifically, it can consist of information about the reference image (e.g., reference image information) and information about the reference region (e.g., motion vector information).

Additionally, settings related to inter-frame prediction (or selection information, such as motion estimation/compensation methods, motion model selection information, etc.) can be included in the motion information of the target block. Information about the reference image and reference region (e.g., the number of motion vectors, etc.) can be configured based on settings related to inter-frame prediction.

Motion information can be encoded by configuring a reference image and information about the reference region into a combination, and the combination of the reference image and information about the reference region can be configured as a motion information encoding mode.

Here, information about the reference image and reference region can be obtained based on adjacent blocks or predefined information (e.g., images encoded before or after the current image, zero motion vectors, etc.).

Neighboring blocks can be classified into blocks <inter_blk_A> that belong to the same space as the target block and are closest to the target block, blocks <inter_blk_B> that belong to the same space and are far from the target block, and blocks <inter_blk_C> that belong to a different space from the target block. Information about the reference image and reference region can be obtained based on neighboring blocks (candidate blocks) that belong to at least one of the categories.

For example, the motion information of the target block can be encoded based on the motion information of the candidate block or the reference image information, and the motion information of the target block can also be encoded based on the information derived from the motion information of the candidate block or the reference image information (or information median, transformation processing, etc.). In other words, the motion information of the target block can be predicted from the candidate block, and information about the motion information can be encoded.

In this invention, the motion information of the target block can be encoded based on one or more motion information encoding modes. Here, the motion information encoding modes can be defined in various ways and can include one or more of skip modes, merge modes, and competition modes.

Based on the template matching (tmp) described above, it can be combined with motion information coding modes, or it can be supported as a standalone motion information coding mode, or it can be included in the detailed configuration of all or some motion information coding modes. This presupposes that template matching is supported at a higher unit (e.g., picture, sub-picture, slice, etc.), but the flags regarding support can be considered as part of the inter-frame prediction settings.

Based on the method described above for performing block matching (ibc) within the current image, it can be combined with motion information coding modes, or it can be supported as a standalone motion information coding mode, or it can be included in the detailed configuration of all or some motion information coding modes. This presupposes that block matching is supported in a higher unit within the current image, but the flag regarding support can be considered part of the inter-frame prediction settings.

Based on the aforementioned motion model (affine), motion information coding modes can be combined with other motion information coding modes, or they can be supported as standalone motion information coding modes, or they can be included in the detailed configuration of all or some motion information coding modes. This presupposes that higher units support non-translational motion models, but the flags regarding support can be considered part of the inter-frame prediction settings.

For example, single motion information encoding patterns such as temp_inter, temp_tmp, temp_ibc, and temp_affine can be supported. Alternatively, combined motion information encoding patterns such as temp_inter_tmp, temp_inter_ibc, temp_inter_affine, and temp_inter_tmp_ibc can be supported. Alternatively, the motion information prediction candidate group for constructing temp can include template-based candidates, candidates based on methods for performing block matching in the current image, and affine-based candidates.

Here, temp can refer to skip mode, merge mode, and comp mode. For example, it is the same as the motion information encoding modes such as skip_inter, skip_tmp, skip_ibc, and skip_affine in skip mode; merge_inter, merge_tmp, merge_ibc, and merge_affine in merge mode; and comp_inter, comp_tmp, comp_ibc, and comp_affine in comp mode.

When skip mode, merge mode, and competition mode are supported, and candidates considering the above factors are included in the motion information prediction candidate group for each mode, a mode can be selected by distinguishing between the flags for skip mode, merge mode, and competition mode. For example, a flag indicates whether skip mode is supported; if the value is 1, skip mode is selected. A flag indicates whether merge mode is supported; if the value is 1, merge mode is selected, and if the value is 0, competition mode can be selected. Additionally, candidates based on inter, tmp, ibc, and affine can be included in the motion information prediction candidate group for each mode.

Alternatively, when multiple motion information encoding modes are supported under a common mode, in addition to the flags used to select one of the skip mode, merge mode, and competition mode, additional flags can be supported to distinguish the detailed mode of the selected mode. For example, when the merge mode is selected, this means additionally supporting flags to select a detailed mode among merge_inter, merge_tmp, merge_ibc, merge_affine, etc.—which are detailed modes associated with the merge mode. Alternatively, flags indicating whether it is merge_inter can be supported, and if it is not merge_inter, additional flags for selection among merge_tmp, merge_ibc, merge_affine, etc. can be supported.

The encoding settings can support all or part of the motion information encoding mode candidates. Here, the encoding settings can be limited by more than one of the following elements: target block state information, image type, image category, color components, and inter-frame prediction support settings (e.g., whether template matching is supported in the current image, non-translation motion model support elements, etc., and whether block matching is supported).

For example, the supported motion information encoding patterns can be determined based on the block size. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H of the block's width (W) and height (H). With the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. With the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by either the first or second threshold size, or both.

In this scenario, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this case, a first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and a second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

For example, supported motion information encoding modes can be determined based on the image type. In this case, the image type can include at least one of skip mode, merge mode, and competition mode. In this case, a method for performing block matching (or template matching) in the current image, a single motion information encoding mode for an affine model (hereinafter referred to as an element) can be supported, or motion information encoding modes can be supported by combining two or more elements. Alternatively, elements other than the predetermined motion information encoding modes can be configured as motion information prediction candidate groups.

P/B image types may include at least one of skip mode, merge mode, and competition mode. In this case, methods for performing general inter-frame prediction, template matching, block matching on the current image can be supported, as well as a single motion information coding mode for an affine model (hereinafter referred to as an element), or motion information coding modes can be supported by combining two or more elements. Alternatively, elements other than the predetermined motion information coding mode can be configured as motion information prediction candidate groups.

Figure 9 is a flowchart illustrating the encoding of motion information according to an embodiment of the present invention.

Referring to Figure 9, the motion information encoding mode S900 of the target block can be checked.

Motion information coding modes can be defined by combining and setting predetermined information (e.g., motion information) used for inter-frame prediction. Predetermined information may include one or more of the following: predicted motion vector, motion vector difference, motion vector difference accuracy, reference image information, reference orientation, motion model information, information about the presence or absence of residual components, etc. Motion information coding modes may include at least one of skip mode, merge mode, and contention mode, and may include other additional modes.

The configuration and settings of explicitly generated information and implicitly determined information can be determined based on the motion information encoding pattern. In this case, in the explicit case, each piece of information can be generated individually or in combination (e.g., in the form of an index).

For example, in skip or merge modes, the predicted motion vector, reference image information, and/or reference orientation can be defined based on (a) predetermined index information. In this case, the index may include one or more candidates, and each candidate can be set based on the motion information of a predetermined block (e.g., the predicted motion vector, reference image information, and reference orientation of the corresponding block are composed of a combination). Additionally, motion vector differences (e.g., zero vectors) can be implicitly handled.

For example, in competition mode, the predicted motion vector, reference image information, and/or reference orientation can be defined based on (one or more) predetermined index information. In this case, the index can support each information segment individually, or a combination of two or more information segments can be supported. In this case, the index can include one or more candidates, and each candidate can be set based on the motion information of a predetermined block (e.g., predicted motion vector, etc.), or can be configured with predetermined values (e.g., the interval between current images is processed as 1, 2, 3, etc., and the reference orientation information is processed as L0, L1, etc.) (e.g., reference image information, reference orientation information, etc.). In addition, motion vector differences can be explicitly generated, and motion vector difference accuracy information can be additionally generated based on the motion vector difference (e.g., if it is not a zero vector).

As an example, in skip mode, information about the presence or absence of residual components can be implicitly processed (e.g., processed as <cbf_flag=0>), and the motion model can be implicitly processed as a predetermined value (e.g., supporting parallel motion models).

For example, in merge or competition modes, information about the presence or absence of residual components can be explicitly generated, and the motion model can be explicitly selected. In this case, information for classifying the motion model can be generated under a single motion information encoding mode, or separate motion information encoding modes can be supported.

In the above description, when supporting one index, index selection information may not be generated, while when supporting two or more indexes, index selection information may be generated.

In this invention, a candidate group comprising two or more indices is referred to as a motion information prediction candidate group. Furthermore, existing motion information encoding patterns can be modified or new motion information encoding patterns can be supported through various predetermined information and settings.

Referring to Figure 9, the predicted motion vector S910 of the target block can be obtained.

The predicted motion vector can be determined as a predetermined value, or a candidate set of predicted motion vectors can be constructed and selected from the candidate set. When the predicted motion vector can be determined as a predetermined value, it implies an implicit determination; and when a candidate set of predicted motion vectors can be constructed and selected from the candidate set, index information for selecting the predicted motion vector can be explicitly generated.

Here, the predetermined value can be set based on the motion vector of a block at a predetermined position (e.g., left, top, top left, top right, bottom left, etc.), or it can be set based on the motion vectors of two or more blocks, or it can be set to a default value (e.g., zero vector).

Depending on the motion information encoding mode, the predicted motion vectors can have the same or different candidate group construction settings. For example, skip mode/merge mode/competition mode can support a, b, and c prediction candidates respectively, and the number of each candidate can be the same or different. In this case, a, b, and c can include integers greater than or equal to 1, such as 2, 3, 5, 6, 7, etc. The order, settings, etc., of constructing candidate groups will be described through other implementations. In the examples described later, the construction of the skip mode candidate group will be described based on the same assumptions as the merge mode, but it is not limited to this, and some can have different constructions.

The predicted motion vectors obtained based on the index information can be used as is to reconstruct the motion vectors of the target block, or they can be adjusted based on the distance between reference images and the reference orientation.

For example, in competitive mode, reference image information can be generated separately. When the distance between the current image and the reference image of the target block differs from the distance between the image containing the predicted motion vector candidate block and the reference image of the corresponding block, adjustments can be made based on the distance between the current image and the reference image of the target block.

Furthermore, the number of motion vectors derived from the motion model can be different. That is, in addition to the motion vector at the upper left control point, motion vectors for the upper right and lower left control points can also be derived from the motion model.

Referring to Figure 9, the motion vector difference S920 of the target block can be reconstructed.

In skip mode and merge mode, the motion vector difference can be obtained as a zero vector value, and in competition mode, the difference information of each component can be reconstructed.

Motion vectors can be represented according to a predetermined precision (e.g., based on interpolation precision settings or motion model selection information settings). Alternatively, they can be represented based on one of several precisions, and predetermined information for this purpose can be generated. In this case, the predetermined information can relate to the selection of motion vector precision.

For example, when one component of the motion vector is 32/16, motion component data for 32 can be obtained based on an implicit precision setting in units of 1/16 pixels. Alternatively, the motion component can be converted to 2/1 based on an explicit precision setting in units of 2 pixels, and motion component data for 2 can be obtained.

The above configuration can be a method for accurately processing motion vectors with the same precision, or it can be obtained by adding the motion vector difference to a predicted motion vector obtained through the same precision transformation process.

Alternatively, the motion vector difference can be represented with a predetermined precision, or it can be represented with one of multiple precisions. In the case where the motion vector difference can be represented with one of multiple precisions, precision selection information can be generated.

Precision can include at least one of 1/16, 1/8, 1/4, 1/2, 1, 2, 4 pixel units, and the number of candidates can be an integer greater than or equal to 1, 2, 3, 4, 5. Here, the supported precision candidate group configuration (e.g., division by number, candidate, etc.) can be explicitly determined in units such as sequences, pictures, sub-pictures, slices, tiles, and blocks. Alternatively, it can be implicitly determined based on encoding settings, which can be defined by at least one or more elements of image type, reference image, color components, and motion model selection information.

The following describes a case where precision candidates are formed based on various coding elements and precision is described by assuming a difference in motion vectors, which can be similar to or applicable to motion vectors.

For example, in the case of a translational motion model, candidate groups such as {1/4, 1}, {1/4, 1/2}, {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} can be configured, and in the case of a non-translational motion model, precision candidate groups such as {1/16, 1/4, 1}, {1/16, 1/8, 1}, {1/16, 1, 4}, {1/16, 1/4, 1, 2}, and {1/16, 1/4, 1, 4} can be configured. This assumes that the minimum precision supported in the case of the translational motion model is 1/4 pixel unit, while the minimum precision supported in the case of the non-translational motion model is 1/16 pixel unit, and additional precision beyond the minimum precision can be included in the candidate groups.

Alternatively, when the reference image is a different image, candidate groups such as {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} can be configured. Additionally, when the reference image is the current image, candidate groups such as {1, 2}, {1, 4}, and {1, 2, 4} can be configured. In this case, the configuration for the reference image being the current image can be as follows: for block matching, interpolation is not performed in fractional units, and if interpolation is performed in fractional units, it can be configured to include fractional units in the precision candidates.

Alternatively, when the color component is a luminance component, candidate groups such as {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} are feasible. And when the color component is a chrominance component, candidate groups such as {1/8, 1/4}, {1/8, 1/2}, {1/8, 1}, {1/8, 1/4, 1/2}, {1/8, 1/2, 2}, and {1/8, 1/4, 1/2, 2} are feasible. In this case, when the color component is a chrominance component, candidate groups proportional to the luminance component can be formed based on the color component composition ratio (e.g., 4:2:0, 4:2:2, 4:4:4, etc.), or a single candidate group can be configured.

Although the above examples describe the case where there are multiple precision candidates for each coded element, it is possible for there to be a single candidate configuration (i.e., represented with a predetermined minimum precision).

In summary, the motion vector difference with minimum accuracy can be reconstructed based on the accuracy of the motion vector difference in the competition mode, and the motion vector difference with zero vector value can be obtained in the skip mode and the merge mode.

Here, the number of motion vector differences reconstructed from the motion model can be different. That is, in addition to the motion vector difference of the position of the upper left control point, the motion vector differences of the positions of the upper right and lower left control points can also be derived from the motion model.

Alternatively, a single motion vector difference precision can be applied to multiple motion vector differences, or a single motion vector difference precision can be applied to each motion vector difference, depending on the encoding settings.

Additionally, information about the accuracy of the motion vector difference can be generated when at least one motion vector difference is not zero, and otherwise it can be omitted, but is not limited to this.

Referring to Figure 9, the motion vector S930 of the target block can be reconstructed.

The motion vector of the target block can be reconstructed by obtaining the predicted motion vector and the difference between the motion vectors—which are obtained through processing prior to this step—and summing them. In this case, when reconstructing by selecting one of multiple accuracies, accuracy unification processing can be performed.

For example, a motion vector obtained by adding the predicted motion vector (related to the accuracy of the motion vector) and the motion vector difference can be reconstructed based on accuracy selection information. Alternatively, the motion vector difference (related to the accuracy of the differential motion vector) can be reconstructed based on accuracy selection information, and the motion vector can be obtained by adding the reconstructed motion vector difference to the predicted motion vector.

Referring to Figure 9, motion compensation S940 can be performed based on the motion information of the target block.

In addition to the predicted motion vector and motion vector difference mentioned above, motion information can also include reference image information, reference orientation, motion model information, etc. In skip mode and merge mode, some information (e.g., reference image information, reference orientation, etc.) can be implicitly determined, while in competition mode, relevant information can be explicitly processed.

The predicted block can be obtained by performing motion compensation based on the motion information obtained through the above processing.

Referring to Figure 9, decoding S950 can be performed on the residual components of the target block.

The target block can be reconstructed by adding the residual components obtained through the above processing to the predicted block. In this case, explicit or implicit processing can be performed on the information regarding the presence or absence of the residual components, depending on the motion information encoding pattern.

Figure 10 is a layout diagram of the target block and its adjacent blocks according to an embodiment of the present invention.

Referring to Figure 10, blocks <inter_blk_A> adjacent to the target block in the left, top, top left, top right, and bottom left directions, and blocks <inter_blk_B> adjacent to the block corresponding to the target block in the spatial Col_Pic that are different in time in the center, left, right, top, bottom, top left, top right, bottom left, and bottom right directions, can be configured as candidate blocks for predicting the motion information (e.g., motion vectors) of the target block.

In the case of inter_blk_A, the orientation of adjacent blocks can be determined based on the encoding order such as raster scan or z scan, and existing orientations can be removed, or adjacent blocks in the right, bottom, and lower right directions can additionally include candidate blocks.

Referring to Figure 10, Col_Pic can be an adjacent image before or after the current image (e.g., when the interval between images is 1), and the corresponding block in the image can have the same position as the target block.

Alternatively, Col_Pic can be an image in which the interval between images is predefined relative to the current image (e.g., the interval between images is z, where z is an integer of 1, 2, or 3), and the corresponding block can be set to a position moved from the predetermined coordinates (e.g., top left) of the target block by a predetermined disparity vector, and the disparity vector can be set to a predefined value.

Alternatively, Col_Pic can be set based on motion information of adjacent blocks (e.g., a reference image), and disparity vectors can be set based on motion information of adjacent blocks (e.g., motion vectors) to determine the position of the corresponding block.

In this case, k neighboring blocks can be referenced, and k can be an integer of 1, 2, or greater. If k is 2 or greater, Col_Pic and the disparity vector can be obtained based on calculations such as maximum, minimum, median, or weighted average of motion information of neighboring blocks (e.g., reference images or motion vectors). For example, the disparity vector can be set as the motion vector of the left or top block, and can be set as the median or average of the motion vectors of the left and bottom blocks.

The settings for time candidates can be determined based on motion information configuration settings. For example, the position of Col_Pic and the position of the corresponding block can be determined based on whether motion information is configured on a block-by-block or sub-block-by-sub-block basis. For this motion information, motion information configured on a block-by-block or sub-block-by-sub-block basis should be included in the motion information prediction candidate group. For example, when obtaining motion information on a sub-block basis, the block at the position where the predetermined disparity vector has been moved can be set as the position of the corresponding block.

The above examples illustrate the following: information about the location of the block corresponding to Col_Pic is implicitly determined, and related information can also be explicitly generated in units such as sequences, pictures, slices, tile groups, tiles, and blocks.

On the other hand, when partitioning units such as sub-images, slices, and tiles are not limited to a single image (e.g., the current image) and are shared between images, and if the position of the block corresponding to Col_Pic differs from the partitioning unit to which the target block belongs (i.e., if it extends or deviates from the boundary of the partitioning unit to which the target block belongs), it can be determined based on motion information of subsequent candidates according to a predetermined priority. Alternatively, it can be set to the same position in the image as the target block.

The motion information of spatially adjacent blocks and temporally adjacent blocks can be included in the motion information prediction candidate group of the target block, which is called the spatial candidate and the temporal candidate. A_a spatial candidates and b_b temporal candidates can be supported, and a and b can be integers from 1 to 6 or greater. In this case, a can be greater than or equal to b.

Based on the configuration of the predicted candidates according to the previous motion information, a fixed number of spatial and temporal candidates can be supported, or a variable number (e.g., including 0) can be supported.

Additionally, blocks <inter_blk_C> that are not adjacent to the target block can be considered during the candidate group construction.

For example, motion information of blocks located relative to a target block (e.g., top-left coordinates) at predetermined intervals can be the target to be included in the candidate group. The intervals can be p and q (depending on each component of the motion vector), where p and q can be integers of 2 or greater, such as 0, 2, 4, 8, 16, 32, and p and q can be the same or different. That is, assuming the top-left coordinates of the target block are (m, n), motion information of blocks at positions (m ± p, n ± q) can be included in the candidate group.

For example, motion information of blocks that have been encoded before the target block can be used as targets to be included in the candidate group. In this case, based on a predetermined encoding order relative to the target block (e.g., raster scan, Z-scan, etc.), a predetermined number of recently encoded blocks can be considered as candidates, and since encoding is performed using a first-in-first-out method such as FIFO, blocks with older encoding orders can be removed from the candidates.

Because the above example involves a case where motion information of a non-nearest neighbor block that has already been encoded relative to the target block is included in the candidate group, this is called a statistical candidate.

In this case, the reference block used to obtain statistical candidates can be set as the target block, but a higher block of the target block can be set (e.g., if the target block is in coded block <1>/predicted block <2>, the higher block is the maximum coded block <1>/coded block <2>, etc.). k statistical candidates can be supported, and k can be an integer from 1 to 6 or greater.

Motion information obtained by combining spatial, temporal, and statistical candidates can be included in a candidate group. For example, candidates can be obtained by applying a weighted average to each component of multiple (2, 3, or more) motion information previously included in the candidate group, and candidates can be obtained by processing each component of multiple motion information using median, maximum, minimum, etc. This is called a combined candidate, and r combined candidates can be supported, where r can be an integer of 1, 2, 3, or greater.

When the candidates used for combination do not have the same information about the reference image, they can be set as the reference image information for the combined candidates based on the reference image information about one of the candidates, or they can be set to a predefined value.

Based on the configuration of the predicted candidates according to the previous motion information, it can support a fixed number of statistical candidates or combined candidates, or it can support a variable number.

Additionally, default candidates with predetermined values such as (s, t) can be included in the candidate group, and the configuration of candidate prediction based on previous motion information can support a variable number (0, 1, or larger integers). In this case, s and t can be set to include 0, and can be set based on various block sizes (e.g., maximum encoding/prediction/transform block, minimum encoding/prediction/transform block horizontal or vertical length, etc.).

Depending on the motion information encoding mode, motion information prediction candidate groups can be constructed from the candidates, and other additional candidates can be included. Furthermore, the candidate group configuration settings can be the same or different depending on the motion information encoding mode.

For example, skip mode and merge mode can jointly construct candidate groups, while competition mode can construct a single candidate group.

Here, the candidate group configuration settings can be limited by the following: the category and position of the candidate block (e.g., determined from the position of the sub-block used to obtain motion information in the determined direction among the left/top/top-left/top-right/bottom-left directions), the number of candidates (e.g., total number, maximum number of each category), and the candidate construction method (e.g., priority of each category, priority within each category, etc.).

The number (total) of motion information prediction candidate groups can be k, and k can be an integer from 1 to 6 or greater. In this case, when the number of candidates is one, it may mean that no candidate group selection information is generated, and the motion information of the predefined candidate block is set as the predicted motion information. When there are two or more candidate groups, candidate group selection information can be generated.

The candidate block can be one or more of inter_blk_A, inter_blk_B, and inter_blk_C. In this case, inter_blk_A can be the default included category, and the other category can be an additional supported category, but is not limited to this.

The above description can be applied to the candidate construction for motion information prediction in translational motion models, and the same or similar candidate blocks can be used/referenced in the candidate group construction of non-translational motion models (affine models). On the other hand, the candidate group construction of affine models can differ from that of translational motion models because the motion features and the number of motion vectors are different.

For example, when the motion model of a candidate block is an affine model, the configuration of the motion vector set of the candidate block can be included in the candidate group as is. For example, when the top-left and top-right coordinates are used as control points, the motion vector set of the top-left and top-right coordinates of the candidate block can be included in the candidate group as a candidate.

Alternatively, when the motion model of the candidate block is a translational motion model, the combination of motion vectors of the candidate blocks set based on the positions of the control points can be constructed as a set and included in the candidate group. For example, when the upper-left, upper-right, and lower-left coordinates are used as control points, the motion vector of the upper-left control point can be predicted based on the left, upper, and upper-left motion vectors of the target block (e.g., in the case of a translational motion model), and the motion vector of the upper-right control point can be predicted based on the motion vectors of the upper and upper-right blocks of the target block (e.g., in the case of a translational motion model), and the motion vector of the lower-left control point can be predicted based on the left and right motion models of the target block. In the above examples, the case where the motion model of the candidate block set based on the positions of the control points is a translational motion model has been described, but even in the case of an affine model, the motion vectors of the control point positions can be obtained or derived only by acquiring or deriving them. That is, for the motion vectors of the upper-left, upper-right, and lower-left control points of the target block, the motion vectors can be obtained or derived from the upper-left, upper-right, lower-left, and lower-right control points of the candidate block.

In summary, when the motion model of the candidate block is an affine model, the motion vector set of the corresponding block can be included in the candidate group (A), and when the motion model of the candidate block is a translational motion model, the motion vector set obtained by combining the control points of the motion vectors of the predetermined control points of the target block can be included in the candidate group (B).

In this case, candidate groups can be constructed using either method A or method B, or both methods A and B can be used. Furthermore, method A can be used first for construction, and method B can be used subsequently, but this is not a limitation.

Figure 11 shows an example diagram of statistical candidates according to an embodiment of the present invention.

Referring to Figure 11, the blocks corresponding to A to L in Figure 11 refer to the blocks that have been encoded and are separated from the target block by a predetermined interval.

When candidate blocks for motion information of a target block are limited to blocks adjacent to the target block, they may not reflect various characteristics of the image. Therefore, blocks encoded before the current image can also be considered candidate blocks. This category is called inter_blk_B, and this is referred to as statistical candidate.

Since the number of candidate blocks included in the motion information prediction candidate group is limited, effective statistical candidate management may be required.

(1) A block at a predetermined position that is at a predetermined distance from the target block can be regarded as a candidate block.

As an example, by restricting the x-component of a predetermined coordinate relative to the target block (e.g., the top-left coordinate, etc.), blocks with predetermined intervals can be considered as candidate blocks, and examples such as (-4, 0), (-8, 0), (-16, 0), (-32, 0), etc. are feasible.

As an example, by restricting the y-component of a predetermined coordinate relative to the target block, blocks with a predetermined interval can be considered as candidate blocks, and examples such as (0, -4), (0, -8), (0, -16), (0, -32), etc. are feasible.

As an example, a block with predetermined intervals of x and y components (excluding 0) relative to predetermined coordinates of the target block can be considered a candidate block. Examples such as (-4, -4), (-4, -8), (-8, -4), (-16, 16), (16, -16), etc., are feasible. In this example, the x component may depend on the y component having a positive sign.

Candidate blocks that are considered as statistical candidates can be determined in a limited way based on the encoding settings. For an example described later, refer to Figure 11.

Candidate blocks can be categorized into statistical candidates based on whether they belong to a predetermined unit. Here, the predetermined unit can be determined based on the largest coded block, block, tile, slice, sub-image, image, etc.

For example, when selecting candidate blocks by restricting them to the largest coded block that is the same as the target block, A through C can be the target.

Alternatively, when selecting candidate blocks by restricting the blocks belonging to the largest coded block to which the target block belongs and the left largest coded block, A through C, H, and I can be targets.

Alternatively, when selecting candidate blocks by restricting the blocks belonging to the largest coded block to which the target block belongs and the largest coded block above, A through C, E, and F can be targets.

Alternatively, when selecting candidate blocks by restricting them to blocks belonging to the same slice as the target block, A through I can be the target.

Candidate blocks can be classified as statistical candidates based on whether they are located in a predetermined direction relative to the target block. Here, the predetermined direction can be determined from left, top, upper left, upper right, etc.

For example, when selecting candidate blocks by restricting blocks located in the left and top directions relative to the target block, B, C, F, I, and L can be targets.

Alternatively, when selecting candidate blocks by restricting them to blocks located in the left, top, upper left, or upper right directions relative to the target block, A through L can be the target.

As in the example above, even if candidate blocks considered as statistical candidates are selected, there can be a priority order for including them in the motion information prediction candidate group. That is, as many statistical candidates as there are supported can be selected according to priority.

Priority can be determined by a predetermined order or by various coding elements. Coding elements can be qualified based on the distance between the candidate block and the target block (e.g., whether it is a short distance, the distance between blocks can be checked based on the x and y components) and the relative direction of the candidate block to the target block (e.g., left, top, top-left, top-right direction, left->top->top-right->left, etc.).

(2) A block that has been encoded according to a predetermined encoding order relative to the target block can be regarded as a candidate block.

The following examples assume that raster scan (e.g., maximum coding block) and Z scan (e.g., coding block, prediction block, etc.) are followed, but it should be understood that the following descriptions may vary depending on the scan order.

In the case of example (1) above, priority can be supported for constructing candidate groups for motion information prediction, and similarly, predetermined priority can be supported in this embodiment. Priority can be determined by various coding elements, but for ease of description, it is assumed that priority is determined according to the coding order. In the examples described later, candidate blocks and priorities will be considered together.

Candidate blocks that are considered as statistical candidates can be determined in a limited way based on the encoding settings.

Candidate blocks can be categorized into statistical candidates based on whether they belong to a predetermined unit. Here, the predetermined unit can be determined based on the largest coded block, block, tile, slice, sub-image, image, etc.

For example, when a block belonging to the same image as the target block is selected as a candidate block, the candidate block and priority, such as J-D-E-F-G-K-L-H-I-A-B-C, can be the target.

Alternatively, when a block belonging to the same slice as the target block is selected as a candidate block, the candidate block and its priority, such as D-E-F-G-H-I-A-B-C, can be the target.

Candidate blocks can be classified as statistical candidates based on whether they are located in a predetermined direction relative to the target block. Here, the predetermined direction can be determined based on the left or top direction.

For example, when selecting candidate blocks by restricting blocks located to the left of the target block, the candidate blocks and priorities such as K-L-H-I-A-B-C can be the target.

Alternatively, when selecting candidate blocks by restricting blocks located in the direction above the target block, the candidate block and priority, such as E-F-A-B-C, can be the target.

The above description is based on the following assumptions: candidate blocks and priorities are combined, but not limited to this, and various candidate blocks and priorities can be set.

As mentioned above, the priorities included in the candidate group for motion information prediction can be supported by the descriptions in (1) and (2) above. One or more priorities can be supported, and a single priority can be supported for the entire candidate block. Alternatively, candidate blocks can be classified into two or more categories, and priorities based on the categories can be supported. In the former case, an example could be selecting k candidate blocks based on a single priority, while in the latter case, an example could be selecting p and q (p+q=k) candidate blocks based on each priority (e.g., two categories). In this case, categories can be classified based on predetermined encoding elements, and these encoding elements can include whether the block belongs to a predetermined unit, whether the candidate block is located in a predetermined direction relative to the target block, etc.

An example of selecting candidate blocks based on whether a candidate block belongs to a predetermined unit (e.g., a partitioning unit) described by (1) and (2) has been described. In addition to partitioning units, candidate blocks can be selected by restricting blocks within a predetermined range relative to the target block. For example, it can be identified by boundary points that define the ranges of (x1, y1), (x2, y2), (x3, y3), etc., and by the minimum and maximum values of the x or y components. Values such as x1 to y3 can be integers such as 0, 4, and 8 (based on absolute values).

Statistical candidates can be supported based on either (1) or (2) above, or they can be supported in a combination of (1) and (2). The above description has described how to select candidate blocks for statistical candidates, and the management and updating of statistical candidates and their inclusion in the motion information prediction candidate group will be described later.

The construction of motion information prediction candidate groups can usually be performed on a block-by-block basis, because the motion information of blocks adjacent to the target block may be the same or similar. That is, spatial or temporal candidates can be constructed based on the target block.

Furthermore, statistical candidates can be constructed based on a predetermined unit, since the candidate block is not adjacent to the target block. The predetermined unit can be a higher block that includes the target block.

For example, when the target block is a prediction block, statistical candidates can be constructed on a prediction block basis or on a coding block basis.

Alternatively, when the target block is a coded block, statistical candidates can be constructed on a coded block basis, or statistical candidates can be constructed on a predecessor block basis (e.g., depth information is at least one difference from the target block). In this case, the predecessor block can include the largest coded block, or it can be a unit obtained based on the largest coded block (e.g., an integer multiple of the largest coded block).

As in the examples above, statistical candidates can be constructed based on the target block or higher units, and in the examples described later, it is assumed that statistical candidates are constructed based on the target block.

Furthermore, memory initialization for statistical candidates can be performed based on predetermined units. The predetermined unit can be determined based on an image, sub-image, slice, tile, block, or chunk, and in the case of a chunk, it can be set based on the largest coded block. For example, memory initialization for statistical candidates can be performed based on an integer (1, 2, or greater) of the largest coded block, the row or column units of the largest coded block, etc.

The management and updating of statistical candidates are described below. A memory can be prepared for managing statistical candidates, and it can store motion information for up to k blocks. In this case, k can be an integer from 1 to 6 or larger. The motion information stored in the memory can be determined based on the motion vectors, reference images, and reference directions. Here, the number of motion vectors (e.g., 1 to 3, etc.) can be determined based on motion model selection information. For ease of explanation, the case where a block that has completed its encoding according to a predetermined encoding order based on the target block is considered a candidate block will be described later.

(Case 1) Motion information of blocks pre-coded according to the coding order can be included as candidates in the previous ranking. Additionally, when the maximum number is filled and updated by other candidates, candidates in the previous ranking can be removed, and the order can be shifted forward one by one. In the following example, x represents a blank space that has not yet been constructed.

(Example)

1 order - [a,x,x,x,x,x,x,x,x,x]

2 order - [a,b,x,x,x,x,x,x,x,x]

...

9 order - [a, b, c, d, e, f, g, h, i, x]

10. Sequence: [a, b, c, d, e, f, g, h, i, j] -> Add j. (Full)

11. Sequence - [b, c, d, e, f, g, h, i, j, k] -> Add k. Remove leading a. Shift sequence.

(Case 2) When duplicate motion information exists, remove pre-existing candidates and adjust the order of existing candidates for forwarding.

In this context, duplication implies identical motion information, which can be defined by the motion information encoding mode. As an example, in the merge mode, duplication can be determined based on motion vectors, a reference image, and a reference direction, while in the competition mode, duplication can be determined based on motion vectors and a reference image. In the competition mode, if the reference images for each candidate are identical, duplication can be determined by comparing motion vectors; and if the reference images for each candidate are different, duplication can be determined by comparing motion vectors scaled based on the reference images for each candidate, but this is not the only possibility.

(Example)

1. Sequence - [a, b, c, d, e, f, g, x, x, x]

2. The sequence - [a, b, c, d, e, f, g, d, x, x] -> d overlaps.

-[a, b, c, e, f, g, d, x, x, x] -> Remove the existing d, shift order

(Case 3) When there is duplicate motion information, the corresponding candidate can be marked and managed as a long-term candidate (long-term candidate) according to predetermined conditions.

A separate memory can be supported for long-term candidates, and additional information such as occurrence frequency, in addition to motion information, can be stored and updated. In the example below, the memory for long-term candidate storage can be represented as ().

(Example)

1. Sequence - [(), b, c, d, e, f, g, h, i, j, k]

2. Sequence - [(), b, c, d, e, f, g, h, i, j, k] -> e is the next sequence. Overlapping - [(e), b, c, d, f, g, h, i, j, k] -> Move to the front. Long-term marker.

3. Sequence - [(e), c, d, f, g, h, i, j, k, l] -> Add l. Remove b.

...

10. The sequence - [(e), l, m, n, o, p, q, r, s, t] -> l is the next sequence. Overlapping.

-[(e, l), m, n, o, p, q, r, s, t]->long-term marker

11. Order - [(e, l), m, n, o, p, q, r, s, t] -> l is the next order. 3. Overlap - [(l, e), m, n, o, p, q, r, s, t] -> Changes the long-term candidate order.

The above example illustrates the following situation: existing statistical candidates (short-term candidates) are managed in an integrated manner, but can be managed separately from short-term candidates, and the number of long-term candidates can be 0, 1, 2 or a larger integer.

Short-term candidates can be managed in a first-in, first-out (FIFO) manner based on encoding order, but long-term candidates can be managed based on frequency, considering the redundancy among short-term candidates. In the case of long-term candidates, the order of candidates can be changed according to frequency during memory updates, but this is not a limitation.

The above example could be a case where candidate blocks are classified into two or more categories and priority is supported based on that category. When constructing motion information prediction candidate groups to include statistical candidates, short-term and long-term candidates a and b can be constructed separately based on priority within each candidate. Here, a and b can be integers 0, 1, 2, or larger.

Cases 1 to 3 above are examples based on statistical candidates, and the configuration of (case 1 + case 2) or (case 1 + case 3) is feasible, and statistical candidates can be managed in various modified and added forms.

Figure 12 is a conceptual diagram of statistical candidates based on a non-translational motion model according to an embodiment of the present invention.

Statistical candidates can be used not only for translational motion models but also for supporting non-translational motion models. In the case of translational motion models, due to the high frequency of occurrence, there may be many blocks that can be referenced from spatial or temporal candidates, etc. However, in the case of non-translational motion models, because of the low frequency of occurrence, statistical candidates in addition to spatial or temporal candidates may be needed.

Referring to Figure 12, the motion information { _TV0 , _TV1 } of the target block is predicted based on the motion information { _PV0 , _PV1 } of the block <inter_blk_B> that belongs to the same space as the target block and is far from the target block (hmvp).

Compared to the statistical candidates for the translational motion model described above, the number of motion vectors in the motion information stored in memory can be increased. Therefore, the basic concept of statistical candidates can be the same as or similar to that of the translational motion model, and can be explained by different configurations of the number of motion vectors. The following will describe the differences based on the non-translational motion model. In the example described later, it is assumed that the affine model has three control points (e.g., v0, v1, and v2).

In the case of an affine model, the motion information stored in memory can be determined based on a reference image, a reference direction, and motion vectors v0, v1, and v2. Here, the stored motion vectors can be stored in various forms. For example, the motion vectors can be stored as is, or predetermined difference information can be stored.

Figure 13 is an example diagram of the construction of motion information for storing the position of each control point as a statistical candidate according to an embodiment of the present invention.

Referring to Figure 13, (a) the motion vectors v0, v1 and v2 of the candidate block are represented as v0, v1 and v2, while in Figure 13, (b) the motion vector v0 of the candidate block is represented as v0, and the motion vectors v1 and v2 are represented as v*1 and v*2, which is different from the motion vector v0.

In other words, the motion vector of each control point location can be stored as is, or the difference between the motion vector of another control point location and the original motion vector can be stored. This can be an example of a configuration considered in terms of memory management, and various modification examples are possible.

Whether statistical candidates are supported can be explicitly generated in units such as sequences, images, slices, tiles, and blocks, or implicitly determined based on encoding settings. Since encoding settings can be limited by the various encoding elements described above, a detailed description is omitted.

Here, the support of a statistical candidate can be determined based on the motion information encoding pattern. Alternatively, it can be determined based on the motion model selection information. For example, statistical candidates can be supported among merge_inter, merge_ibc, merge_affine, comp_inter, comp_ibc, and comp_affine.

If statistical candidates are supported for both translational and non-translational motion models, then memory for multiple statistical candidates can be supported.

Next, we will describe a method for constructing motion information prediction candidate groups based on motion information encoding patterns.

The candidate set for motion information prediction in the competitive mode (hereinafter referred to as the competitive mode candidate set) may include k candidates, and k may be an integer of 2, 3, 4 or greater. The competitive mode candidate set may include at least one of spatial candidates or temporal candidates.

Spatial candidates can be derived from at least one of the blocks adjacent to the reference block in the left, top, top-left, top-right, and bottom-left directions. Alternatively, at least one candidate can be derived from the left-adjacent block (left block, bottom-left block) and the top-adjacent block (top-left block, top block, top-right block), under the assumption of this setting, which will be described later.

There can be two or more priorities for constructing candidate groups. In the region adjacent to the left, the priority order can be set to bottom left-left, and in the region adjacent to the top, the priority order can be set to top right-top-top left.

In the examples above, spatial candidates can be derived solely from blocks whose reference images are identical to the target block's. Furthermore, spatial candidates can be derived by scaling the target block's reference image (hereinafter marked *). In this case, within regions adjacent to the left, the priority order can be set to left-bottom-left*-bottom-left* or left-bottom-left*-left*, and within regions adjacent to the top, the priority order can be set to top-right-top-top-left-top-right*-top*-top-left* or top-right-top-left-top-left*-top*-top*-top-right*.

Time candidates can be derived from at least one of the adjacent blocks such as center, left, right, top, bottom, top left, top right, bottom left, and bottom right, based on the block corresponding to the reference block. Priorities can be configured for candidate groups, and priorities such as center-bottom left-right-bottom-bottom left-center-top left can be set.

When the sum of the maximum allowed number of spatial candidates and the maximum allowed number of temporal candidates is less than the number of candidates in the competitive mode, temporal candidates can be included in the candidate group, regardless of the construction of the candidate group for spatial candidates.

Candidates can be included in a candidate group based on priority, availability of each candidate block, and the maximum number of time-limited candidates (the number of q, an integer between 1 and the number of contested mode candidates).

Here, when the maximum allowed number of spatial candidates is set equal to the number of merge pattern candidates, temporal candidates may not be included in the candidate group, and when the maximum allowed number of spatial candidates is not filled, temporal candidates may be included in the candidate group. This example assumes the latter case.

Here, the motion vector of the temporal candidate can be obtained based on the motion vector of the candidate block and the distance interval between the current image and the reference image of the target block, and the reference image of the temporal candidate can be obtained based on the distance interval between the current image and the reference image of the target block, or the reference image of the temporal candidate can be obtained as a predefined reference image (e.g., the reference image index is 0).

If the candidate group for the competitive mode is not filled with spatial candidates, temporal candidates, etc., the candidate group can be constructed by including the default candidate with the zero vector.

The competition mode focuses on the description of comp_inter, and in the case of comp_ibc or comp_affine, can construct candidate groups for similar or different candidates.

For example, in the case of comp_ibc, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. In this case, candidate groups can be constructed by prioritizing spatial candidates, and then candidate groups can be constructed in the order of statistical candidates - combined candidates - default candidates, etc., but the order is not limited to this.

Alternatively, in the case of comp_affine, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. Specifically, a set of motion vectors for a candidate (e.g., a block) can be constructed in candidate group <1>, or a candidate group <2> can be constructed where the motion vectors of candidates (e.g., two or more blocks) are combined based on control point locations. It is possible, but not limited to, to first include the candidates for <1> in the candidate group and then include the candidates for <2> in the candidate group.

Since a detailed description of the above construction of various competitive mode candidate groups can be obtained from comp_inter, a detailed description is omitted.

In the process of constructing a candidate group for a competitive mode, if there is duplicate motion information among the candidates initially included, the candidates initially included can be maintained, and candidates with the next priority can be included in the candidate group.

Here, before constructing candidate groups, the motion vectors of the candidate groups can be scaled based on the distance between the reference image of the target block and the current image. For example, when the distance between the reference image of the target block and the current image and the distance between the reference image of the candidate block and the image to which the candidate block belongs are the same, the motion vector can be included in the candidate group; otherwise, the motion vector scaled based on the distance between the reference image of the target block and the current image can be included in the candidate group.

In this context, redundancy can imply identical motion information, which can be defined by the motion information encoding pattern. In a contention-based mode, duplication can be determined based on motion vectors, a reference image, and a reference direction. For example, if at least one component of a motion vector is different, it can be determined that no duplication exists. Redundancy checks are typically performed when a new candidate is included in the candidate group, but this can be modified to omit the redundancy check.

The motion information prediction candidate set for the merging pattern (hereinafter referred to as the merging pattern candidate set) may include k candidates, and k may be an integer of 2, 3, 4, 5, 6 or greater. The merging pattern candidate set may include at least one of spatial candidates or temporal candidates.

Spatial candidates can be derived from at least one of the blocks adjacent to the reference block in the left, top, top-left, top-right, and bottom-left directions. Priorities can exist for constructing candidate groups, and priorities can be set such as left-top-bottom-top-right-top-left, left-top-top-right-bottom-left-top, and top-left-bottom-left-top-right.

Candidates can be included in a candidate group based on priority, availability of each candidate block (e.g., determined based on encoding mode, block position, etc.), and the maximum allowed number of spatial candidates (the number of p, an integer between 1 and the number of candidate groups for the merge mode). Candidates may not be included in the candidate group in the order tl-tr-bl-t3-13, depending on the maximum allowed number and availability. If the maximum allowed number is 4 and the availability of all candidate blocks is true, the motion information of tl may not be included in the candidate group; conversely, if the availability of some candidate blocks is false, the motion information of tl may be included in the candidate group.

Time candidates can be derived from at least one of the blocks adjacent to the center, left, right, top, bottom, top left, top right, bottom left, and bottom right, based on the block corresponding to the reference block. Priorities can exist for constructing candidate groups, and priorities such as center-bottom left-right-bottom, bottom left-center-top left, etc., can be set.

Candidates can be included in a candidate group based on priority, availability of each candidate block, and the maximum number of time candidates allowed (q, an integer between 1 and the number of merge pattern candidates).

Here, the motion vector of the temporal candidate can be obtained based on the motion vector of the candidate block, and the reference image of the temporal candidate can be obtained based on the reference image of the candidate block, or a predefined reference image can be obtained (e.g., reference image index 0).

For priorities included in the merge pattern candidate group, spatial candidate-temporal candidate can be set, or vice versa, and priorities that combine spatial and temporal candidates can be supported. In this example, spatial candidate-temporal candidate is assumed.

In addition, statistical candidates or combined candidates can also be included in the merged pattern candidate group. Statistical candidates and combined candidates can be constructed after spatial and temporal candidates, but are not limited to this, and various priorities can be set.

For statistical candidates, up to n motion information segments can be managed, and among these, z motion information segments can be included in the merged pattern candidate group as statistical candidates. z can vary depending on the candidate construction already included in the merged pattern candidate group, and can be an integer of 0, 1, 2 or greater, and can be less than or equal to n.

A combined candidate can be derived by merging n candidates already included in the combined pattern candidate group, where n can be an integer of 2, 3, 4, or greater. The number of combined candidates (n) can be information explicitly generated in units such as sequences, pictures, sub-pictures, slices, tiles, blocks, or chunks. Alternatively, it can be implicitly determined based on encoding settings. In this case, the encoding settings can be limited based on one or more factors such as the size, shape, position, image type, and color components of the reference block.

Furthermore, the number of combined candidates can be determined based on the number of unfilled candidates in the merged candidate group. In this case, the number of unfilled candidates in the merged candidate group can be the difference between the number of merged candidate groups and the number of already filled candidates. That is, if the merged candidate group construction is complete, combined candidates may not be added. If the merged candidate group construction is incomplete, combined candidates may be added, but no combined candidates are added when the number of filled candidates in the merged candidate group is less than or equal to one.

If the candidate group for the merge pattern is not filled with spatial candidates, temporal candidates, statistical candidates, or combined candidates, the candidate group can be constructed by including the default candidate with a zero vector.

The merge mode focuses on the description of merge_inter, and in the case of merge_ibc or merge_affine, it can construct candidate groups for similar or different candidates.

For example, in the case of merge_ibc, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. In this case, candidate groups can be constructed by prioritizing spatial candidates, and then candidate groups can be constructed in the order of statistical candidates - combined candidates - default candidates, etc., but the order is not limited to this.

Alternatively, in the case of merge_affine, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, temporal candidates, statistical candidates, combined candidates, and default candidates. Specifically, a set of motion vectors for a candidate (e.g., a block) can be constructed in candidate group <1>, or a candidate group <2> can be constructed where the motion vectors of candidates (e.g., two or more blocks) are combined based on control point locations. It is possible, but not limited to, to first include the candidates for <1> in the candidate group and then include the candidates for <2>.

Since a detailed description of the above construction of various merge pattern candidate groups can be derived from merge_inter, a detailed description will be omitted.

In the process of constructing a candidate group for merging patterns, if there is overlapping motion information among the first included candidates, the first included candidates can be maintained, and candidates with the next priority can be included in the candidate group.

In this context, redundancy implies identical motion information, which can be defined by the motion information encoding pattern. In the case of the merging pattern, overlap can be determined based on motion vectors, a reference image, and a reference direction. For example, if at least one motion vector component is different, it can be determined that no overlap exists. Redundancy checks are typically performed when new candidates are included in the candidate group, but this can be modified to omit them.

Figure 14 is a flowchart illustrating motion information encoding according to an embodiment of the present invention.

A candidate list of motion vector predictions for the target block can be generated S1400. The aforementioned competition mode candidate group may mean a candidate list of motion vectors, and its detailed description will be omitted.

The motion vector difference S1410 of the target block can be reconstructed. The difference between the x and y components of the motion vector can be reconstructed separately, and the difference of each component can have a value of 0 or greater.

The prediction candidate index for the target block can be selected from the motion vector prediction candidate list S1420. The predicted motion vector value for the target block can be derived based on the motion vector obtained from the candidate index in the candidate list. If a predetermined motion vector prediction value can be obtained, the process of selecting the prediction candidate index and index information can be omitted.

The accuracy information S1430 for the motion vector difference can be obtained. This can be the accuracy information typically applied to the x and y components of the motion vector, or the accuracy information applied to each component. If the motion vector difference is 0, the accuracy information can be omitted, and this processing can also be omitted.

An adjustment offset S1440 for the motion vector prediction value can be derived. The offset can be a value added to or subtracted from the x or y component of the motion vector prediction value. The offset can support only one of the x and y components, or it can support both x and y components.

Assuming the predicted motion vector value is (pmv_x, pmv_y), and the offsets are adjusted to offset_x and offset_y, then the predicted motion vector value can be adjusted (or obtained) to (pmv_x + offset_x, pmv_y + offset_y).

Here, the absolute values of offset_x and offset_y can be integers of 0, 1, 2, or larger, and can have values (+1, -1, +2, -2, etc.), where sign information is considered together. Furthermore, offset_x and offset_y can be determined based on a predetermined precision. The predetermined precision can be determined in units of 1/16, 1/8, 1/4, 1/2, and 1 pixel, and can be based on interpolation precision, motion vector precision, etc.

For example, if the interpolation precision is in units of 1/4 pixel, then the interpolation precision can be 0, 1/4, -1/4, 2/4, -2/4, 3/4, or -3/4, which are combinations of absolute value and sign information.

Here, offset_x and offset_y can be backed by a and b respectively, and a and b can be integers of 0, 1, 2 or greater. a and b can have fixed values or variable values. Furthermore, a and b can have the same or different values.

Whether adjustment offsets for motion vector prediction values are supported can be explicitly supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., or implicitly determined based on encoding settings. Additionally, the settings for adjustment offsets (e.g., the range and number of values) can be determined based on encoding settings.

The encoding settings can be determined by taking into account at least one of the encoding elements such as image type, color components, target block state information, motion model selection information (e.g., whether it is a translational motion model), reference image (e.g., whether it is the current image), and motion vector precision difference selection information (e.g., whether it is a predetermined unit among 1/4, 1/2, 1, 2, and 4 units).

For example, the size of the block can be used to determine whether offset and offset settings are supported. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H, which have the width (W) and height (H) of the block. In the case of the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. In the case of the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by one of the first threshold size or the second threshold size, or both.

In this case, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this scenario, the first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and the second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

Furthermore, the adjustment offset can be applied to all candidates included in the motion information prediction candidate group, or it can be applied to only some candidates. In the example described later, it is assumed that all candidates included in the candidate group can be applied, but the candidates to which the adjustment offset is applied can be selected from 0, 1, 2 to the maximum number of candidates.

If offset adjustment is not supported, this processing and offset adjustment information can be omitted.

The motion vector of the target block can be reconstructed by adding the predicted motion vector value and the motion vector difference in S1450. In this case, the motion vector accuracy can be prioritized to unify the processing of the predicted motion vector value or the motion vector difference, and the aforementioned motion vector scaling process can be performed first or during this process.

The configuration and order are not limited to this and can be changed in various ways.

Figure 15 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention.

For ease of explanation, two motion vector prediction candidates are supported, and it is assumed that only one component (x or y component) is compared. Furthermore, it is assumed that the interpolation precision or motion vector precision is 1/4 pixel unit. Additionally, it is assumed that 1/4, 1, and 4 pixel units are supported for the motion vector difference precision (e.g., if 1/4, it is assumed to be binaryized as <0>, if 1, it is assumed to be binaryized as <10>, and if 4, it is assumed to be binaryized as <11>). Furthermore, it is assumed that sign information is omitted for the motion vector difference, and the motion vector difference is processed through univariate binarization (e.g., 0: <0>, 1: <10>, 2: <110>, etc.).

Referring to Figure 15, the value of the actual motion vector (X) is 2, the value of candidate 1 (A) is 0, and the value of candidate 2 (B) is 1/4.

(In cases where differential motion vector precision is not supported)

Since the distance (da) between A and X is 8/4 (9 bits) and the distance (db) between B and X is 7/4 (8 bits), B can be selected as the prediction candidate in terms of bit generation.

(When motion vector difference accuracy is supported)

Since da is 8/4, it can generate 1 pixel unit precision (2 bits) and 2/1 difference information (3 bits), thus generating a total of 5 bits. On the other hand, since db is 7/4, it can generate 1/4 pixel unit precision (1 bit) and 7/4 difference information (8 bits), thus generating a total of 9 bits. In terms of bit generation, the prediction candidate can be chosen as A.

As in the example above, if motion vector difference accuracy is not supported, it may be advantageous to select candidates with short distance intervals from the motion vectors of the target block as prediction candidates. If it is supported, then in addition to the distance interval from the motion vectors of the target block, the amount of information generated based on the accuracy information may be important in selecting prediction candidates.

Figure 16 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention. In the following, it is assumed that motion vector difference accuracy is supported.

Since da is 33/4, it can generate 1/4 pixel unit precision (1 bit) and 33/4 difference information (34 bits), thus generating a total of 35 bits. On the other hand, since db is 21/4, it can generate 1/4 pixel unit precision (1 bit) and 21/4 difference information (22 bits), thus generating a total of 23 bits. Regarding bit generation, prediction candidate B can be selected.

Figure 17 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention. In the following, it is assumed that motion vector difference accuracy is supported and that offset information is adjusted.

In this example, it is assumed that the adjustment offset has 0 and +1 candidates, and a flag (1 bit) indicating whether the adjustment offset has been applied and offset selection information (1 bit) are generated.

Referring to Figure 17, A1 and B1 can be motion vector predictions obtained based on the prediction candidate indices, while A2 and B2 can be new motion vector predictions obtained by modifying A1 and B1 with adjusted offsets. Assume that the distances between A1, A2, B1, B2 and X are da1, da2, db1, and db2, respectively.

(1) Since da1 is 33/4, it can generate 1/4 pixel unit precision (1 bit), offset application flag (1 bit), offset selection information (1 bit) and 33/4 difference information (34 bits), so a total of 37 bits can be generated.

(2) Since da2 is 32/4, it can generate 4-pixel unit precision (2 bits), offset application flag (1 bit), offset selection information (1 bit) and 2/1 difference information (3 bits), so a total of 7 bits can be generated.

(3) Since db1 is 21/4, it can generate 1/4 pixel unit precision (1 bit), offset application flag (1 bit), offset selection information (1 bit) and 21/4 difference information (22 bits), so a total of 25 bits can be generated.

(4) Since db2 is 20/4, it can generate 1 pixel unit precision (2 bits), offset application flag (1 bit), offset selection information (1 bit) and 5/1 difference information (6 bits), so a total of 10 bits can be generated.

For in-situ quantity generation, the prediction candidate can be selected as A, and the offset selection information can be selected as index 1 (+1 in this example).

In the above example, when deriving the motion vector difference based on the existing motion vector prediction values, there may be a situation where many bits are generated due to the small difference between the vector values. This problem can be solved by adjusting the motion vector prediction values.

Predefined flags can be supported to apply adjustment offsets to motion vector predictions. Predefined flags can be configured via offset application flags, offset selection information, etc.

(If only offset application flags are supported)

When the offset application flag is 0, the offset is not applied to the motion vector prediction value, and when the offset application flag is 1, the predetermined offset can be added to or subtracted from the motion vector prediction value.

(If only offset selection information is supported)

The offset set based on offset selection information can be added to or subtracted from the motion vector prediction value.

(If offset application flags and offset selection information are supported)

When the offset application flag is 0, the offset is not applied to the motion vector prediction value, and when the offset application flag is 1, the offset set based on the offset selection information can be added to or subtracted from the motion vector prediction value. This setting is assumed in the example described later.

Meanwhile, offset application flags and offset selection information can be used as information that is unnecessarily generated in some cases. That is, even if the offset is not applied, the offset-related information may be quite inefficient if the information is already zero or the amount of information is reduced based on the maximum precision information. Therefore, it is necessary to support settings that implicitly generate information according to predetermined conditions, rather than always explicitly generating offset-related information.

Next, we assume a motion-related coding sequence (motion vector difference reconstruction -> motion vector difference precision acquisition). In this example, we assume that motion vector difference precision is supported when at least one of the x and y components is not zero.

When the motion vector difference is not 0, one of the following pixel units can be selected: {1/4, 1/2, 1, 4} for the motion vector difference precision.

If the selected information belongs to a predetermined category, information about the adjustment offset can be implicitly omitted; otherwise, information about the adjustment offset can be explicitly generated.

Here, the category can include one of the differential motion vector precision candidates, which can be configured with various categories such as {1/4}, {1/4, 1/2}, {1/4, 1/2, 1}. The minimum precision can also be included in the category.

For example, if the motion vector difference precision is 1/4 pixel (e.g., minimum precision), the offset application flag is implicitly set to 0 (i.e., not applied), and if the motion vector difference precision is not 1/4 pixel, the offset application flag is explicitly generated, and if the offset application flag is 1 (i.e., offset), offset selection information can be generated.

Alternatively, when the motion vector difference precision is 4 pixels (e.g., maximum precision), an offset application flag can be explicitly generated, and when the offset application flag is 1, offset selection information can be generated. Alternatively, when the motion vector difference precision is not 4 pixels, the offset information can be implicitly set to 0.

The above examples assume that offset-related information is implicitly omitted when the motion vector difference indicates minimum precision, and offset-related information is explicitly generated when the motion vector difference indicates maximum precision, but are not limited to this.

Figure 18 is an example diagram illustrating the arrangement of multiple motion vector prediction values according to an embodiment of the present invention.

The above example has described the part about redundancy checks when constructing candidate groups for motion information prediction. Here, redundancy means that the motion information is the same, and as described above, if at least one component of the motion vector is different, it can be determined that there is no redundancy.

Through redundancy checks, multiple candidates for motion vector predictions can be made to not overlap. However, when multiple candidate component elements are very similar (i.e., each candidate's x or y component exists within a predetermined range, and the width or height of that predetermined range is an integer of 1, 2, or greater; or the range can be set based on offset information), motion vector predictions and offset-modified motion vector predictions can overlap. Various settings are feasible for this purpose.

As an example (C1), in the step of constructing a candidate group for motion information prediction, a new candidate can be included in the candidate group if it does not overlap with already included candidates. That is, if there is no overlap only by comparing with the same candidate as previously described, it can be a configuration that can be included in the candidate group.

As an example (C2), a new candidate can be included in a candidate group if it does not overlap with a predetermined number of candidates (group_A) obtained by adding an offset to it, and with a predetermined number of overlaps between already included candidates and candidates (group_B) obtained by adding an offset to it. The predetermined number can be an integer of 0, 1, or greater. If the predetermined number is 0, even if one overlaps, the corresponding new candidate may not be included in the candidate group. Alternatively, (C3), a predetermined offset (a concept distinct from adjusting the offset) can be added to the new candidate to be included in the candidate group, and this offset can be a value that allows group_A to be configured not to overlap with group_B.

Referring to Figure 18, it can be the case that multiple motion vectors belonging to categories A, B, C, and D (in this example, AX, BX, CX, and DX. X is 1 and 2. For example, A1 is a candidate included in the candidate group preceding A2) satisfy the non-overlapping condition (if any component of the candidate motion vector is different).

In this example, it is assumed that -1, 0, and 1 are supported for the x and y components, respectively, and the offset-modified motion vector predictions can be represented as * in the figure. Furthermore, dashed lines (rectangles) represent ranges (e.g., group_A, group_B, etc.) that can be obtained by adding an offset to a predetermined motion vector prediction.

In the case of category A, it can correspond to the case where A1 and A2 do not overlap and group_A1 and group_A2 do not overlap.

In the case of categories B, C, and D, it can correspond to the situation where B1/C1/D1 and B2/C2/D2 do not overlap, but group_B1/C1/D1 and group_B2/C2/D2 partially overlap.

Here, in the case of category B, it could be an example of constructing candidate groups without taking any special actions (C1).

Here, in the case of category C, the following configuration can be used: In this configuration, C2 is determined to be redundant in the redundancy check step, and C3, which has the next higher priority, is included in the candidate group (C2).

Here, in the case of category D, it can be the following: D2 is determined to be redundant in the redundancy check step, and D2 is modified so that group_D2 does not overlap with group_D1 (i.e., D3. D3 is not a motion vector present in the candidate group construction priority) (C3).

Among various categories, it can be applied to the setting of constructing candidate groups of prediction patterns, or various methods other than those mentioned above can be applied.

Figure 19 is a flowchart illustrating motion information encoding in merging mode according to an embodiment of the present invention.

A candidate list S1900 for predicting motion information of the target block can be generated. The above-mentioned candidate group for merging modes may refer to the candidate list for predicting motion information, and its detailed description will be omitted.

A prediction candidate index S1910 for the target block can be selected from the motion information prediction candidate list. The motion vector prediction value of the target block can be derived based on the motion information obtained from the candidate indices in the candidate list. If a predetermined motion information prediction value can be obtained, the process of selecting the prediction candidate index and index information can be omitted.

An adjustment offset S1920 for the motion vector prediction value can be derived. The offset can be a value added to or subtracted from the x or y component of the motion vector prediction value. The offset can support only one of the x or y components, or it can support both x and y components.

Since the adjustment offset in this process can be the same as or similar to the adjustment offsets above, a detailed description is omitted, and a portion of the differences will be described later.

Assuming the predicted motion vector is (pmv_x, pmv_y), and the offsets are adjusted to offset_x and offset_y, the predicted motion vector can be adjusted (or obtained) to (pmv_x + offset_x, pmv_y + offset_y).

Here, the absolute values of offset_x and offset_y can be integers such as 0, 1, 2, 4, 8, 16, 32, 64, 128, etc., and can have sign information considered together. Furthermore, offset_x and offset_y can be determined based on a predetermined precision. The predetermined precision can be determined in units of 1/16, 1/8, 1/4, 1/2, or 1 pixel.

For example, if the motion vector precision is 1/4 pixel unit, then the motion vector precision can be 0, 1/4, -1/4, 1/2, -1/2, 1, -1, 2, -2, etc., which combine absolute value and sign information.

Here, offset_x and offset_y can be resources a and b, respectively, and a and b can be integers such as 0, 1, 2, 4, 8, 16, 32, etc. a and b can have fixed values or variable values. Furthermore, a and b can have the same or different values.

Whether adjustment offsets for motion vector prediction values are supported can be explicitly supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., or implicitly determined based on encoding settings. Additionally, the settings for adjustment offsets (e.g., the range and number of values) can be determined based on encoding settings.

The encoding settings can be determined by taking into account at least one of the encoding elements such as image type, color components, target block state information, motion model selection information (e.g., whether it is a translational motion model), and reference image (e.g., whether it is the current image).

For example, the size of the block can be used to determine whether offset and offset settings are supported. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H, which have the width (W) and height (H) of the block. In the case of the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. In the case of the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by one of the first threshold size or the second threshold size, or both.

In this case, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this scenario, the first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and the second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

Furthermore, the adjustment offset can be applied to all candidates included in the motion information prediction candidate group, or it can be applied to only some candidates. In the example described later, it is assumed that all candidates included in the candidate group can be applied, but the candidates to which the adjustment offset is applied can be selected from a number between 0, 1, or 2 and the maximum number of candidates.

Predefined flags can be used to apply adjustment offsets to motion vector prediction values. These flags can be configured via offset application flags, absolute offset value information, offset sign information, and more.

If offset adjustment is not supported, this processing and offset adjustment information can be omitted.

The motion vector S1930 of the target block can be reconstructed using the predicted motion vector values. Motion information other than the motion vector (e.g., reference image, reference direction, etc.) can be obtained based on the predicted candidate index.

The configuration and order are not limited to this and can be changed in various ways. The background description for supporting offset adjustment in merge mode has been described above with various examples of competition mode, and therefore its detailed description will be omitted.

The methods of this disclosure can be implemented as program instructions executable by various computer devices and stored in a computer-readable medium. The computer-readable medium may include program instructions, data files, and data structures, individually or in combination. The program instructions recorded on the computer-readable medium may be specifically designed for this disclosure or are known to and therefore usable by one of skill in the art of computer software.

The computer-readable medium may include hardware devices particularly suitable for storing and executing program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, etc. The program instructions may include machine language code generated by a compiler or high-level language code that can be executed by an interpreter in a computer. The aforementioned hardware devices may be configured to operate as one or more software modules to perform operations according to this disclosure, and vice versa.

Furthermore, the above-described methods or devices can be implemented by combining or separating all or part of their configurations or functions.

Although the present disclosure has been described above with reference to preferred embodiments thereof, those skilled in the art will understand that various modifications and variations may be made to the present disclosure without departing from the scope and spirit of the present disclosure.

Industrial applicability

This invention can be used to encode/decode video signals.

Image encoding/decoding methods and data transmission methods

This application is a divisional application of the invention patent application filed on September 24, 2019, with application number 201980062809.8 and title "Image Encoding/Decoding Method and Apparatus".

Technical Field

This invention relates to image encoding/decoding methods and devices.

Background Technology

With the widespread use of the Internet and portable terminals, and the development of information and communication technologies, multimedia data is being used more and more. Therefore, in order to provide various services or perform various tasks through image prediction in various systems, there is an urgent need to improve the performance and efficiency of image processing systems. However, research and development achievements have not kept pace with this trend.

Therefore, existing methods and devices for encoding/decoding images need performance improvements in image processing, particularly in image encoding or image decoding.

Summary of the Invention

Technical issues

The purpose of this invention to solve the above problems is to provide an image encoding/decoding device that uses an adjustment offset to modify the motion vector prediction value.

Technical solutions

According to an embodiment of the present invention, a method for decoding an image to achieve the above objectives includes: constructing a motion information prediction candidate list for a target block, selecting a prediction candidate index, obtaining a predicted motion vector adjustment offset, and reconstructing the motion information of the target block.

Here, constructing a candidate list for motion information prediction may also include: when the already included candidates, candidates obtained based on offset information, new candidates, and candidates obtained based on offset information do not overlap, the new candidate is included in the candidate group.

Here, deriving the predicted motion vector adjustment offset may also include: deriving the predicted motion vector adjustment offset based on the offset application flag and/or offset selection information.

Beneficial effects

When using inter-frame prediction according to the present invention as described above, coding performance can be improved by efficiently obtaining the predicted motion vectors.

Attached Figure Description

Figure 1 is a block diagram illustrating an image encoding and decoding system according to an embodiment of the present invention.

Figure 2 is a block diagram illustrating an image encoding device according to an exemplary embodiment of the present invention.

Figure 3 illustrates an image decoding device according to an embodiment of the present invention.

Figure 4 illustrates a tree-based block partitioning method as an embodiment of the present invention.

Figure 5 is an example diagram illustrating various cases of obtaining prediction blocks through inter-frame prediction according to the present invention.

Figure 6 is an example diagram of constructing a list of reference pictures according to an embodiment of the present invention.

Figure 7 is a conceptual diagram illustrating a non-translational motion model according to an embodiment of the present invention.

Figure 8 is an example diagram illustrating motion estimation in units of sub-blocks according to an embodiment of the present invention.

Figure 9 is a flowchart illustrating the encoding of motion information according to an embodiment of the present invention.

Figure 10 is a layout diagram of the target block and its adjacent blocks according to an embodiment of the present invention.

Figure 11 shows an example diagram of statistical candidates according to an embodiment of the present invention.

Figure 12 is a conceptual diagram of statistical candidates based on a non-translational motion model according to an embodiment of the present invention.

Figure 13 is an example diagram of the construction of motion information of the position of each control point stored as a statistical candidate according to an embodiment of the present invention.

Figure 14 is a flowchart illustrating motion information encoding according to an embodiment of the present invention.

Figure 15 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 16 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 17 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 18 is an example diagram illustrating the arrangement of multiple motion vector prediction values according to an embodiment of the present invention.

Figure 19 is a flowchart illustrating motion information encoding under a merging mode according to an embodiment of the present invention.

The best mode of the invention

The image encoding/decoding method and apparatus of the present invention can construct a predicted motion candidate list of a target block, derive a predicted motion vector from the motion candidate list based on the predicted candidate index, reconstruct the predicted motion vector adjustment offset information, and reconstruct the motion vector of the target block based on the predicted motion vector and the predicted motion vector adjustment offset information.

In the image encoding/decoding method and apparatus of the present invention, the motion candidate list may include at least one of spatial candidates, temporal candidates, statistical candidates, or combined candidates.

In the image encoding/decoding method and apparatus of the present invention, the predicted motion vector adjustment offset can be determined based on at least one of an offset application flag or offset selection information.

In the image encoding/decoding method and apparatus of the present invention, information regarding whether the predicted motion vector adjustment offset information is supported may be included in at least one of the sequence, image, sub-image, slice, tile, or block.

In the image encoding/decoding method and apparatus of the present invention, when encoding a target block in a merge mode, the motion vector of the target block can be reconstructed by using a zero vector, and when encoding a target block in a contention mode, the motion vector of the target block can be reconstructed by using a motion vector difference.

The mode of the present invention

This disclosure can be modified in various ways and has various implementations. Specific implementations of this disclosure will be described with reference to the accompanying drawings. However, these implementations are not intended to limit the technical scope of this disclosure, and it should be understood that this disclosure covers various modifications, equivalents, and alternatives within the scope and concept of this disclosure.

The terms first, second, A, and B used in this disclosure may be used to describe various components, rather than to limit them. These expressions are used only to distinguish one component from another. For example, without departing from the scope of this disclosure, a first component may be referred to as a second component, and a second component may be referred to as a first component. The term and/or covers a combination of or any one of several related terms.

When a component is said to be "connected to" another component or "coupled to/coupled to" another component, it should be understood that a component is connected to the other component directly or through any other component. On the other hand, when a component is said to be "directly connected to" or "directly coupled to" another component, it should be understood that there are no other components between the components.

The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. Unless the context clearly specifies otherwise, the singular form includes the plural reference. In this disclosure, the terms “comprising” or “having” indicate the presence of a feature, number, step, operation, component, part, or combination thereof, but do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, terms including technical or scientific terms used in this disclosure may have the same meaning as commonly understood by those skilled in the art. Terms that are commonly defined in dictionaries may be interpreted as having the same or similar meaning as in the context of the relevant art. Unless otherwise defined, these terms should not be interpreted as having an ideal or overly formal meaning.

Typically, an image can include one or more color spaces depending on its color format. The image can include one or more images of the same size or one or more images of different sizes. For example, the YCbCr color configuration can support color formats such as 4:4:4, 4:2:2, 4:2:0, and monochrome (consisting only of Y). For example, YCbCr 4:2:0 can consist of one luminance component (Y in this example) and two chrominance components (Cb/Cr in this example). In this case, the configuration ratio of the chrominance and luminance components can have a width-to-height ratio of 1:2. For example, in the case of 4:4:4, it can have the same configuration ratio in terms of width and height. As in the examples above, when an image includes one or more color spaces, the image can be divided into color spaces.

Images can be classified into I, P, and B based on their image type (e.g., picture type, subpicture type, slice type, tile type, block type, etc.). I images can be images encoded without a reference image. P images can be images encoded using a reference image, allowing only forward prediction. B images can be images encoded using a reference image, allowing bidirectional prediction. However, depending on the encoding settings, some types (P and B) can be combined, or different compositions of image types can be supported.

The various encoded/decoded information fragments generated in this disclosure can be processed explicitly or implicitly. Explicit processing can be understood as the following: generating encoded/decoded information in sequences, images, sub-images, slices, tiles, blocks, chunks, or sub-blocks by an encoder, including selection information in the bitstream, and reconstructing the relevant information into decoded information by a decoder parsing the relevant information at the same unit level as in the encoder. Implicit processing can be understood as processing the encoded/decoded information using the same procedures, rules, etc., at both the encoder and decoder.

Figure 1 is a conceptual diagram illustrating an image encoding and decoding system according to an embodiment of the present disclosure.

Referring to FIG1, each of the image encoding device 105 and the image decoding device 100 may be a user terminal, such as a personal computer (PC), laptop computer, personal digital assistant (PDA), portable multimedia player (PMP), portable game console (PSP), wireless communication terminal, smartphone, or television (TV), or a server terminal such as an application server or service server. Each of the image encoding device 105 and the image decoding device 100 may be any of a variety of devices, each device including: a communication device such as a communication modem that communicates with various devices or wired/wireless communication networks; a memory 120 or 125 that stores various programs and data for inter-frame prediction or intra-frame prediction to encode or decode images; or a processor 110 or 115 that performs calculation and control operations by executing programs.

Furthermore, the image encoding device 105 can transmit images encoded as bitstreams to the image decoding device 100 in real time or non-real time via wired/wireless communication networks such as the Internet, short-range wireless communication networks, wireless local area networks (WLANs), wireless broadband (Wi-Bro) networks, or mobile communication networks, or via various communication interfaces such as cables or universal serial buses (USB). The image decoding device 100 can reconstruct the image from the received bitstream by decoding the bitstream and reproduce the image. Additionally, the image encoding device 105 can transmit images encoded as bitstreams to the image decoding device 100 via a computer-readable recording medium.

Although the aforementioned image encoding and image decoding devices can be separate devices, depending on the implementation, they can be incorporated into a single image encoding/decoding device. In this case, some components of the image encoding device can be substantially the same as their corresponding components of the image decoding device. Therefore, these components can be configured to include the same structure or perform at least the same function.

Therefore, in the following detailed description of the technical components and their operating principles, redundant descriptions of the corresponding technical components will be omitted. Furthermore, since the image decoding device is a computing device that applies the image encoding method executed in the image encoding device to decoding, the following description will focus on the image encoding device.

The computing device may include: a memory storing programs or software modules that perform image encoding and/or image decoding methods; and a processor connected to the memory and executing the programs. An image encoding device may be referred to as an encoder, and an image decoding device may be referred to as a decoder.

Figure 2 is a block diagram illustrating an image encoding apparatus according to an embodiment of the present disclosure.

Referring to FIG2, the image encoding device 20 may include a prediction unit 200, a subtraction unit 205, a transformation unit 210, a quantization unit 215, an inverse quantization unit 220, an inverse transformation unit 225, an addition unit 230, a filter unit 235, an encoded image buffer 240, and an entropy encoding unit 245.

The prediction unit 200 can be implemented using a prediction module as a software module, and generates prediction blocks for the blocks to be encoded via intra-frame prediction or inter-frame prediction. The prediction unit 200 can generate prediction blocks by predicting the target block to be encoded in the image. In other words, the prediction unit 200 can generate prediction blocks with predicted pixel values for each pixel by predicting the pixel values of pixels in the target block based on inter-frame prediction or intra-frame prediction. Furthermore, the prediction unit 200 can provide the encoding unit with information required to generate prediction blocks—such as information about prediction modes, like intra-frame prediction or inter-frame prediction modes—so that the encoding unit can encode the information about the prediction modes. The processing unit undergoing prediction, the prediction method, and specific details about the processing unit can be determined based on the encoding settings. For example, the prediction method and prediction mode can be determined based on the prediction unit, and prediction can be performed based on the transform unit. Additionally, when using a specific encoding mode, the original block can be encoded as is and sent to the decoder without generating prediction blocks through the prediction unit.

Intra-prediction units can have: directional prediction modes, such as horizontal modes, vertical modes, etc., used according to the prediction direction; and non-directional prediction modes, such as DC, plane, etc., using methods such as averaging and interpolating reference pixels. Intra-prediction mode candidate groups can be constructed using directional and non-directional modes, and one of various candidates, such as 35 prediction modes (33 directional + 2 non-directional), 67 prediction modes (65 directional + 2 non-directional), or 131 prediction modes (129 directional + 2 non-directional), can be used as candidate groups.

The intra-prediction unit may include a reference pixel construction unit, a reference pixel filter unit, a reference pixel interpolation unit, a prediction mode determination unit, a prediction block generation unit, and a prediction mode coding unit. The reference pixel construction unit can construct pixels belonging to and adjacent to the target block as reference pixels for intra-prediction. Depending on the coding settings, one adjacent reference pixel line can be constructed as a reference pixel, or another adjacent reference pixel line can be constructed as a reference pixel, or multiple reference pixel lines can be constructed as reference pixels. When some reference pixels are unavailable, available reference pixels can be used to generate a reference pixel. When all reference pixels are unavailable, a predetermined value (e.g., the median of the pixel value range represented by the bit depth) can be used to generate a reference pixel.

To reduce degradation remaining after coding processing, the reference pixel filter unit of the intra-prediction unit can perform filtering on the reference pixel. In this case, the filter used can be a low-pass filter, such as a 3-tap filter [1/4, 1/2, 1/4], a 5-tap filter [2/16, 3/16, 6/16, 3/16, 2/16], etc. Whether to apply filtering and the type of filtering can be determined based on coding information (e.g., block size, shape, prediction mode, etc.).

The reference pixel interpolation unit of the intra-prediction unit can generate fractional-unit pixels based on the prediction mode through linear interpolation of the reference pixels, and can determine the interpolation filter to be applied based on the coding information. In this case, the interpolation filter used may include a 4-tap cubic filter, a 4-tap Gaussian filter, a 6-tap Wiener filter, an 8-tap Kalman filter, etc. Typically, interpolation and low-pass filtering are performed separately, but the filtering process can be performed by combining the filters applied to both processes into one.

The prediction mode determination unit of the intra-frame prediction unit can select at least one optimal prediction mode from the prediction mode candidates, taking coding cost into account, and the prediction block generation unit can use the corresponding prediction mode to generate a prediction block. The prediction mode coding unit can encode the optimal prediction mode based on the prediction value. In this case, the prediction information can be adaptively encoded according to whether the prediction value is correct or not.

In an intra-frame prediction unit, the predicted value is called the most probable mode (MPM), and some modes belonging to the prediction mode candidate group can be constructed into an MPM candidate group. The MPM candidate group can include predicted modes of a predetermined prediction mode (e.g., DC mode, planar mode, vertical mode, horizontal mode, diagonal mode, etc.) or predicted modes of spatially adjacent blocks (e.g., left block, top block, top-left block, top-right block, bottom-left block, etc.). Additionally, modes derived from modes previously included in the MPM candidate group (the difference between +1 and -1 in the case of directional mode) can also be constructed into an MPM candidate group.

There may be a priority for the prediction patterns used to construct MPM candidate groups. The order in which patterns are included in the MPM candidate groups can be determined based on the priority, and the construction of MPM candidate groups can be completed by filling the number of MPM candidate groups according to the priority (determined by the number of prediction pattern candidate groups). In this case, the priority can be determined according to the order of prediction patterns of spatially adjacent blocks, predetermined prediction patterns, and patterns derived from prediction patterns previously included in the MPM candidate groups, but other modifications are also possible.

For example, spatially adjacent blocks can be included in the candidate group in the order of left block, top block, bottom left block, top right block, top left block, etc., and predetermined prediction patterns can be included in the candidate group in the order of DC pattern, planar pattern, vertical pattern, and horizontal pattern. A total of six patterns can be constructed into candidate groups by including patterns obtained by adding +1, -1, etc., to the candidate groups from already included patterns. Alternatively, a total of seven patterns can be constructed into candidate groups by including a priority such as left, top, DC, planar, bottom left, top right, top left, (left+1), (left-1), (top+1).

Subtraction unit 205 can generate residual blocks by subtracting prediction blocks from target blocks. In other words, subtraction unit 205 can calculate the difference between the pixel value of each pixel in the target block to be encoded and the predicted pixel value of the corresponding pixel in the prediction block generated by the prediction unit, to generate a residual signal in the form of a block, i.e., a residual block. Furthermore, subtraction unit 205 can generate residual blocks using units other than those obtained by the block partitioning unit described later.

Transformation unit 210 can transform a spatial signal into a frequency signal. The signal obtained through the transformation process is called the transform coefficient. For example, a residual block having a residual signal received from a subtraction unit can be transformed into a transform block with transform coefficients, and the input signal is determined according to the encoding configuration, which is not limited to the residual signal.

Transformation units can transform residual blocks using transformation schemes such as, but not limited to, Hadamard transform, discrete sine transform (DST)-based transform, or DCT-based transform. These transformation schemes can be changed and modified in various ways.

It can support at least one of the transformation schemes, and can support at least one sub-transformation scheme for each transformation scheme. A sub-transformation scheme can be obtained by modifying a portion of the fundamental vectors in the transformation scheme.

For example, in the case of DCT, one or more sub-transformation schemes DCT-1 to DCT-8 can be supported, and in the case of DST, one or more sub-transformation schemes DST-1 to DST-8 can be supported. A transformation scheme candidate group can be configured with a portion of the sub-transformation schemes. For example, DCT-2, DCT-8, and DST-7 can be grouped into a candidate group for transform.

Transformations can be performed in both the horizontal and vertical directions. For example, a one-dimensional transformation can be performed in the horizontal direction using DCT-2, and a one-dimensional transformation can be performed in the vertical direction using DST-7. Two-dimensional transformations can be used to transform pixel values from the spatial domain to the frequency domain.

A fixed transform scheme can be used, or the transform scheme can be adaptively selected based on the coding configuration. In the latter case, the transform scheme can be selected explicitly or implicitly. When the transform scheme is selected explicitly, information about the transform scheme or set of transform schemes applied in each of the horizontal and vertical directions can be generated, for example, at the block level. When the transform scheme is selected implicitly, the coding configuration can be limited based on image type (I/P/B), color components, block size, block shape, block position, intra-frame prediction mode, etc., and a predetermined transform scheme can be selected based on the coding settings.

In addition, some transformations can be skipped depending on the encoding settings. That is, one or more horizontal and vertical units can be explicitly or implicitly omitted.

Furthermore, the transform unit can send the information needed to generate the transform block to the encoding unit, which encodes the information, includes the encoded information in the bitstream, and sends the bitstream to the decoder. Therefore, the decoder's decoding unit can parse the information from the bitstream for inverse transform.

The quantization unit 215 can quantize the input signal. The signal obtained from the quantization is called the quantization coefficient. For example, the quantization unit 215 can obtain a quantized block with quantization coefficients by quantizing a residual block having residual transform coefficients received from the transform unit, and can determine the input signal according to the encoding settings, which are not limited to the residual transform coefficients.

The quantization unit can quantize the residual block of the transformation using quantization schemes such as, but not limited to, dead-zone uniform boundary value quantization and quantization weighted matrix. These quantization schemes can be changed and modified in various ways.

Quantization can be skipped based on encoding settings. For example, quantization (and inverse quantization) can be skipped based on encoding settings (e.g., a quantization parameter of 0, i.e., a lossless compression environment). In another example, quantization can be omitted when quantization-based compression performance is not performed due to image characteristics. Quantization can be skipped in the entirety or a portion of a quantization block (M×N) (M/2×N/2, M×N/2, or M/2×N), and quantization skip selection information can be set explicitly or implicitly.

The quantization unit can send the information needed to generate the quantization block to the encoding unit, which then encodes the information, includes the encoded information in the bitstream, and sends the bitstream to the decoder. Therefore, the decoder's decoding unit can parse the information from the bitstream for inverse quantization.

Although the above example was described assuming that the residual block is transformed and quantized via a transform unit and a quantization unit, a residual block with transform coefficients can be generated by transforming the residual signal without quantization. The residual block can undergo quantization only without transformation. Alternatively, the residual block can undergo both transformation and quantization. These operations can be determined based on the encoding settings.

The inverse quantization unit 220 performs inverse quantization on the residual block quantized by the quantization unit 215. That is, the inverse quantization unit 220 generates a residual block with frequency coefficients by performing inverse quantization on the quantized frequency coefficient sequence.

The inverse transform unit 225 performs an inverse transform on the residual block dequantized by the inverse quantization unit 220. That is, the inverse transform unit 225 performs an inverse transform on the frequency coefficients of the dequantized residual block to generate a residual block with pixel values, i.e., a reconstructed residual block. The inverse transform unit 225 can perform the inverse transform by reversing the transform scheme used by the transform unit 210.

Addition unit 230 reconstructs the target block by adding the predicted block predicted by prediction unit 200 to the residual block recovered by inverse transform unit 225. The reconstructed target block is stored as a reference picture (or reference block) in encoded picture buffer 240 for use as a reference picture when encoding the next block, another block, or another picture of the target block later.

Filter unit 235 may include one or more post-processing filters, such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF). The deblocking filter removes block distortion occurring at boundaries between blocks in the reconstructed image. The ALF performs filtering based on values obtained by comparing the reconstructed image after filtering the blocks with the deblocking filter with the original image. The SAO reconstructs the pixel-level offset difference between the original image and the residual blocks to which the deblocking filter was applied. These post-processing filters can be applied to the reconstructed image or blocks.

The encoded image buffer 240 can store blocks or images reconstructed by the filter unit 235. The reconstructed blocks or images stored in the encoded image buffer 240 can be provided to the prediction unit 200 that performs intra-frame prediction or inter-frame prediction.

Entropy coding unit 245 can generate a quantization coefficient sequence, a transform coefficient sequence, or a signal sequence by scanning the quantization coefficients, transform coefficients, or residual signals of the generated residual block according to at least one scanning order (e.g., zigzag scanning, vertical scanning, horizontal scanning, etc.). Entropy coding unit 245 can encode the quantization coefficient sequence, transform coefficient sequence, or signal sequence using at least one entropy coding technique. In this case, information about the scanning order can be determined according to coding settings (e.g., image type, coding mode, prediction mode, transform type, etc.), and the relevant information can be implicitly determined or explicitly generated.

Additionally, encoded data, including the encoded information sent from each component, can be generated and output as a bitstream, which can be implemented by a multiplexer (MUX). In this case, encoding can be performed using methods such as exponential Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC) as encoding techniques, but is not limited to these, and various encoding techniques derived from and modified from them can be used.

When entropy encoding (assuming CABAC in this example) is performed on syntax elements such as residual block data and information generated during encoding/decoding processes, an entropy encoding device may include a binarization unit (binarizer), a context modeler, and a binary arithmetic encoding unit (binary arithmetic encoder). In this case, the binary arithmetic encoding unit may include a regular encoding engine and a bypass encoding engine.

Since the syntax elements input to the entropy coding device may not be binary values, if the syntax element is not a binary value, the binarization unit can binarize the syntax element and output a binary string consisting of 0s or 1s. In this case, the binary (bin) can represent bits consisting of 0s or 1s and can be encoded by the binary arithmetic coding unit. In this case, one of the regular coding unit or the bypass coding unit can be selected based on the probability of occurrence of 0s and 1s, and this can be determined according to the encoding/decoding settings. If the syntax element is data with the same frequency of 0s and 1s, the bypass coding unit can be used; otherwise, the regular coding unit can be used.

Various methods can be used when binarizing grammatical elements. For example, fixed-length binarization, unary binarization, truncated Ricean binarization, and K-order exponential Golomb binarization can be used. Additionally, signed or unsigned binarization can be performed depending on the range of values for the grammatical element. The binarization processing of grammatical elements described in this invention can be performed not only on the binarization methods mentioned in the examples above, but also on other additional binarization methods.

Figure 3 is a block diagram illustrating an image decoding apparatus according to an embodiment of the present disclosure.

Referring to Figure 3, the image decoding device 30 can be configured to include an entropy decoder 305, a prediction unit 310, an inverse quantization unit 315, an inverse transform unit 320, an addition/subtraction unit 325, a filter 330, and a decoded image buffer 335.

Furthermore, the prediction unit 310 can be configured to include an intra-frame prediction module and an inter-frame prediction module.

When an image bitstream is received from the image encoding device 20, it can be sent to the entropy decoder 305.

The entropy decoder 305 can decode the bitstream into decoded data including quantization coefficients and decoded information to be sent to each component.

Prediction unit 310 can generate prediction blocks based on data received from entropy decoder 305. A list of reference images can be generated using a default configuration scheme based on reference images stored in decoded image buffer 335.

The intra-frame prediction unit may include a reference sample construction unit, a reference sample filter unit, a reference sample interpolation unit, a prediction block generation unit, and a prediction mode decoding unit. Some of these units may perform the same processing as in the encoder, while others may perform the encoder's processing in reverse.

The inverse quantization unit 315 can inverse quantize the quantized transform coefficients provided in the bitstream and decoded by the entropy decoder 305.

The inverse transform unit 320 can generate residual blocks by applying inverse DCT, inverse integer transform, or similar inverse transform techniques to the transform coefficients.

The inverse quantization unit 315 and the inverse transform unit 320 can perform the processing of the transform unit 210 and the quantization unit 215 of the image encoding device 20 in reverse, and can be implemented in various ways. For example, the inverse quantization unit 315 and the inverse transform unit 320 can use the same processing and inverse transform shared with the transform unit 210 and the quantization unit 215, and can perform the transform and quantization in reverse using information received from the image encoding device 20 about the transform and quantization processing (e.g., transform size, transform shape, quantization type, etc.).

The residual block, which has already undergone inverse quantization and inverse transform, can be added to the prediction block obtained by prediction unit 310 to generate a reconstructed image block. This addition can be performed by addition/subtraction unit 325.

Regarding filter 330, a deblocking filter can be applied when needed to remove blocking artifacts from the reconstructed image blocks. Additional loop filters can be used to improve video quality before and after decoding.

The reconstructed and filtered image blocks can be stored in the decoded image buffer 335.

Although not shown in the accompanying drawings, the image encoding/decoding device may also include image partitioning units and block partitioning units.

Image partitioning units can divide or partition an image into at least one region based on predetermined partitioning units. Here, partitioning units can include sub-images, slices, tiles, blocks, blocks (e.g., maximum coding units), etc.

An image can be divided into one or more rows or columns of tiles. In this case, a tile can be a block-based unit that includes a predetermined non-square area of the image. Alternatively, a tile can be divided into one or more blocks, and a block can consist of blocks arranged in rows or columns.

Slices can be set using one or more configurations, one of which can be composed of bundles (e.g., blocks, regions, tiles, etc.) depending on the scan order, and one of them can be composed of shapes that include non-square regions, and other additional definitions are also possible.

Regarding the definition of slice configuration, relevant information can be generated explicitly or implicitly. Multiple definitions can be set for the configuration of each partitioning unit and slice, and selection information can be generated accordingly.

Slices can be configured in non-square shapes such as blocks, regions, and tiles, and slice position and size information can be represented based on position information for the dividing unit (e.g., top left, bottom right, etc.).

In this invention, the description assumes that an image can be composed of one or more sub-images, a sub-image can be composed of one or more slices, tiles, or blocks, a slice can be composed of one or more tiles or blocks, and a tile can be composed of one or more blocks. However, the invention is not limited thereto.

A partition unit can consist of an integer number of blocks, but is not limited to this, and can consist of decimal numbers instead of integers. That is, when it is not composed of an integer number of blocks, at least one partition unit can consist of sub-blocks.

In addition to non-square slices, there can be partitioning units such as sub-images or tiles, and the position and size information of the units can be represented based on various methods.

For example, the position and size information of non-square cells can be represented based on information about the number of non-square cells, information about the number of columns or rows of non-square cells, information about whether they are evenly divided into columns or rows in non-square cells, information about the width or height of column or row cells in non-square cells, and index information of non-square cells.

In the case of sub-images and tiles, the position and size information of each unit can be represented based on all or part of the information, and based on this, it can be divided or partitioned into one or more units.

Simultaneously, blocks can be divided into various units and sizes using block partitioning units. A basic coding unit (or maximum coding unit, coding tree unit, CTU) can refer to the basic (or initial) unit used for prediction, transformation, quantization, etc., in image coding processing. In this case, a basic coding unit can consist of one luminance basic coding block (maximum coding block or CTB) and two basic chrominance coding blocks, depending on the color format (YCbCr in this example), and the size of each block can be determined according to the color format. Coding blocks (CBs) can be obtained according to the partitioning process. A CB can be understood as a unit that is no longer further subdivided due to certain limitations and can be set as the starting unit for partitioning into subunits. In this disclosure, blocks conceptually include various shapes such as triangles, circles, etc., and are not limited to squares.

While the following description is given in the context of a single color component, it is also applicable, proportionally to the ratios according to the color format, to other color components with some modifications (e.g., in the case of YCbCr 4:2:0, the width-to-height length ratio of the luminance component to the chrominance component is 2:1). Furthermore, although block partitioning depending on other color components (e.g., depending on the block partitioning result of Y in Cb/Cr) is feasible, it should be understood that independent block partitioning for each color component is also feasible. Moreover, while a common block partitioning configuration can be used (considering proportionality to the length ratio), it is also necessary to consider and understand the use of separate block partitioning configurations based on the color components.

In a block partitioning unit, a block can be represented as M×N, and the maximum and minimum values of each block can be obtained within this range. For example, if the maximum and minimum values of a block are 256×256 and 4×4 respectively, then blocks of size ^2m × ²ⁿ (where m and n are integers from 2 to 8 in this example), blocks of size 2m×2m (where m and n are integers from 2 to 128 in this example), or blocks of size m×m (where m and n are integers from 4 to 256 in this example) can be obtained. In this paper, m and n can be equal or different, and one or more ranges, such as maximum and minimum values, can be generated to support the blocks.

For example, information about the maximum and minimum size of a block can be generated, and information about the maximum and minimum size of a block can be generated in some partitioning configurations. In the former case, the information can be about the range of maximum and minimum sizes that can be generated in the image, while in the latter case, the information can be about the maximum and minimum sizes that can be generated based on some partitioning configuration. The partitioning configuration can be defined by image type (I/P/B), color components (YCbCr, etc.), block type (encoding/prediction/transformation/quantization), partition type (index or type), and partitioning scheme (quadtree QT, binary tree BT, and ternary tree TT as tree methods, and SI2, SI3, and SI4 as type methods).

Furthermore, constraints can exist on the aspect ratio (block shape) that can be used for blocks, and at this point, boundary values can be set. Only blocks less than or equal to/less than the boundary value (k) can be supported, where k can be limited according to the aspect ratio A/B (A is the longer or equal value between the width and height, and B is another value). k can be a real number equal to or greater than 1, such as 1.5, 2, 3, 4, etc. As in the example above, constraints on the shape of a block in an image can be supported, or one or more constraints can be supported depending on the partitioning configuration.

In summary, whether block partitioning is supported can be determined based on the above scope and constraints, as well as the partitioning configuration described later. For example, partitioning can be supported when a candidate (child block) split from a block (parent block) meets the block support conditions; otherwise, partitioning may not be supported.

The block partitioning unit can be configured with respect to each component of the image encoding and image decoding devices, and the size and shape of the block can be determined in the process. Different blocks can be configured according to the component. The blocks may include prediction blocks for prediction units, transform blocks for transform units, and quantization blocks for quantization units. However, this disclosure is not limited to this, and block units may be defined separately for other components. Although the shape of each of the inputs and outputs in each component is described as non-square in this disclosure, the inputs and outputs of some components may have any other shape (e.g., non-square triangles).

The size and shape of the initial (or starting) block in the block partitioning unit can be determined from the higher unit. The initial block can be divided into smaller blocks. Once the optimal size and shape are determined based on the block partitioning, that block can be designated as the initial block for the lower unit. The higher unit can be a coding block, and the lower unit can be a prediction block or a transform block, and this disclosure is not limited thereto. Instead, various modifications are possible. Once the initial block of the lower unit is determined as in the example above, a partitioning process can be performed to detect blocks of optimal size and shape, similar to the higher unit.

In summary, a block partitioning unit can divide a basic coding block (or the largest coding block) into at least one coding block, and a coding block can be divided into at least one prediction block/transform block/quantization block. Furthermore, a prediction block can be divided into at least one transform block/quantization block, and a transform block can be divided into at least one quantization block. Some blocks may have dependencies on other blocks (i.e., defined by higher and lower units) or may be independent of other blocks. For example, a prediction block may be a higher unit above a transform block, or it may be a unit independent of the transform block. Various relationships can be established based on the type of block.

Depending on the encoding settings, it can be determined whether higher and lower units are combined. Combining units means that blocks of higher units undergo encoding processing by lower units (e.g., in prediction units, transform units, inverse transform units, etc.) without being split into lower units. In other words, this may mean sharing the partitioning process among multiple units and generating partitioning information in one unit (e.g., a higher unit).

For example, (when a coded block is combined with a prediction block or a transform block), the coded block can undergo prediction, transform, and inverse transform.

For example, (when a coded block is combined with a prediction block), the coded block can undergo prediction, and a transform block that is equal to or smaller in size than the coded block can undergo both transformation and inverse transformation.

For example, (when a coded block is combined with a transform block), a prediction block that is equal to or smaller in size than the coded block can be predicted, and the coded block can be transformed and inverse transformed.

For example, (when a prediction block is combined with a transform block), a prediction block that is equal to or smaller in size than a coding block can undergo prediction, transform, and inverse transform.

For example, (when there is no block combination), a prediction block that is equal to or smaller than the coded block in size can be predicted, and a transform block that is equal to or smaller than the coded block in size can be transformed and inverse transformed.

Although various cases of coded blocks, prediction blocks, and transform blocks have been described in the examples above, this disclosure is not limited thereto.

For the combination of units, a fixed configuration can be supported in the image, or an adaptive configuration can be supported considering various coding factors. Coding factors include image type, color components, coding mode (intra-frame/inter-frame), partitioning configuration, block size/shape/position, aspect ratio, prediction-related information (e.g., intra-frame prediction mode, inter-frame prediction mode, etc.), transform-related information (e.g., transform scheme selection information, etc.), quantization-related information (e.g., quantization region selection information and quantization transform coefficient coding information), etc.

Once a block of optimal size and shape has been detected as described above, pattern information (e.g., partitioning information) for that block can be generated. This pattern information, along with information generated from the components to which the block belongs (e.g., prediction-related information and transformation-related information), can be included in the bitstream and sent to the decoder, where it can be parsed at the same unit level for use in the video decoding process.

The partitioning scheme will now be described. Although for ease of description, it is assumed that the initial blocks are shaped as squares, this disclosure is not limited thereto, and the description can be applied in the same or similar manner to cases where the initial blocks are not squares.

The block partitioning unit can support various types of partitions. For example, it can support tree-based partitioning or index-based partitioning, and other methods as well. In tree-based partitioning, the partition type can be determined based on various types of information (e.g., information indicating whether to perform partitioning, tree type, partition direction, etc.), while in index-based partitioning, specific index information can be used to determine the partition type.

Figure 4 is an example diagram showing the various partition types that can be obtained in the block partitioning unit of this disclosure.

In this example, it is assumed that the partition type shown in Figure 4 was obtained through a single partition operation (or procedure), which should not be construed as limiting the scope of this disclosure. The partition type may also be obtained through multiple partition operations. Furthermore, other partition types not shown in Figure 4 may also be available.

(Tree-based partitioning)

In the tree-based partitioning of this disclosure, QT, BT, and TT can be supported. If one tree method is supported, this can be called single-tree partitioning, while if two or more tree methods are supported, this can be called multi-tree partitioning.

In QT, a block is divided into two partitions (i.e., 4 partitions) in each of the horizontal and vertical directions (n), while in BT, a block is divided into two partitions (b to g) in either the horizontal or vertical direction. In TT, a block is divided into three partitions (h to m) in either the horizontal or vertical direction.

In Qt, a block can be divided into four partitions (o and p) by restricting the partitioning direction to either the horizontal or vertical direction. Furthermore, in BT, it supports partitioning the block into only equal-sized partitions (b and c), partitioning the block into only partitions of different sizes (d to g), or both of these partitioning types. Additionally, in TT, it supports partitioning the block into partitions focused only on a specific direction (1:1:2 or 2:1:1 in the left-to-right or top-to-bottom direction) (h, j, k, and m), partitioning the block into partitions focused on the center (1:2:1) (i and l), or both of these partitioning types. Furthermore, it supports partitioning the block into four partitions in each of the horizontal and vertical directions (i.e., a total of 16 partitions) (q).

The tree method supports partitioning a block into z partitions (b, d, e, h, i, j, o) only horizontally, or into z partitions (c, f, g, k, l, m, p) only vertically, or both partition types. In this paper, z can be an integer equal to or greater than 2, for example, 2, 3, or 4.

In this disclosure, it is assumed that partition type n is supported as QT, partition types b and c are supported as BT, and partition types i and l are supported as TT.

Depending on the encoding settings, one or more tree partitioning schemes can be supported. For example, QT, QT/BT, or QT/BT/TT can be supported.

In the examples above, the basic tree partitioning scheme is QT, and BT and TT are included as additional partitioning schemes depending on whether other trees are supported. However, various modifications can be made. Information indicating whether other trees are supported (bt_enabled_flag, tt_enabled_flag, and bt_tt_enabled_flag, where 0 indicates no support and 1 indicates support) can be implicitly determined based on encoding settings or explicitly determined in units such as sequences, images, sub-images, slices, tiles, blocks, etc.

Partition information may include indications of whether partitioning is performed (tree_part_flag or qt_part_flag, bt_part_flag, tt_part_flag, and bt_tt_part_flag, which can have values of 0 or 1, where 0 indicates no partitioning and 1 indicates partitioning). Additionally, depending on the partitioning scheme (BT and TT), information about the partitioning direction can be added (dir_part_flag, or bt_dir_part_flag, tt_dir_part_flag, and bt_tt_dir_part_flag, which have values of 0 or 1, where 0 indicates <width/horizontal> and 1 indicates <height/vertical>). This information can be generated during the partitioning process.

When multi-tree partitioning is supported, various partition information fragments can be configured. For ease of description, the following description provides an example of how to configure partition information at a depth level (that is, recursive partitioning can be performed by setting one or more supported partition depths).

In example (1), check the information indicating whether to perform partitioning. If partitioning is not performed, the partitioning process ends.

If partitioning is performed, the selection information regarding the partition type is checked (e.g., tree_idx, 0 for QT, 1 for BT, and 2 for TT). The partition direction information is additionally checked based on the selected partition type, and the process proceeds to the next step (if additional partitioning is feasible due to reasons such as the partition depth not yet reaching its maximum value, the process restarts from the beginning; if additional partitioning is not possible, the partitioning process ends).

In Example (2), the system checks whether the partitioning is performed using a certain tree scheme (QT), and the process proceeds to the next step. If the partitioning is not performed using a tree scheme (QT), the system checks whether the partitioning is performed using another tree scheme (BT). In this case, if the partitioning is not performed using that tree scheme, the system checks whether the partitioning is performed using a third tree scheme (TT). If the partitioning is not performed using a third tree scheme (TT), the partitioning process ends.

If partitioning is performed using a tree-based scheme (QT), the process proceeds to the next step. Alternatively, if partitioning is performed using a second-tree scheme (BT), the partition orientation information is checked, and the process proceeds to the next step. If partitioning is performed using a third-tree scheme (TT), the partition orientation information is checked, and the process proceeds to the next step.

In Example (3), the system checks whether to perform partitioning using a tree scheme (QT). If partitioning is not performed using a tree scheme (QT), the system checks whether to perform partitioning using another tree scheme (BT and TT). If partitioning is not performed, the partitioning process ends.

If partitioning is performed using a tree scheme (QT), the process proceeds to the next step. Alternatively, if partitioning is performed using other tree schemes (BT and TT), the partition orientation information is checked, and the process proceeds to the next step.

While the examples above either prioritize the tree partitioning scheme (Examples 2 and 3) or do not prioritize it (Example 1), various modifications are also possible. Furthermore, in the examples above, the partitioning in the current step is independent of the partitioning result of the previous step. However, it is also possible to configure the partitioning in the current step to depend on the partitioning result of the previous step.

In Examples 1 through 3, if some tree partitioning scheme (QT) was executed in a previous step, and the process has thus reached the current step, the same tree partitioning scheme (QT) can also be supported in the current step.

On the other hand, if a certain tree partitioning scheme (QT) was not executed in a previous step and therefore another tree partitioning scheme (BT or TT) was executed, and then the process proceeds to the current step, it can be configured to support other tree partitioning schemes (BT and TT) besides a certain tree partitioning scheme (QT) in the current step and subsequent steps.

In the above cases, the tree configuration supporting block partitioning can be adaptive, and therefore the partitioning information described above can also be configured differently. (Assuming the example described below is Example 3). That is, if partitioning was not performed using a certain tree scheme (QT) in the previous steps, the partitioning process can be performed without considering the tree scheme (QT) in the current step. Furthermore, partitioning information associated with a particular tree scheme (e.g., information indicating whether to perform partitioning, information about the partitioning direction, etc. In this example <QT>, information indicating whether to perform partitioning) can be removed.

The examples above illustrate adaptive partitioning information configuration for cases where block partitioning is allowed (e.g., block size is within the range of maximum and minimum values, partition depth of each tree scheme has not reached the maximum depth (allowed depth), etc.). Even when block partitioning is restricted (e.g., block size does not exist within the range of maximum and minimum values, partition depth of each tree scheme has reached the maximum depth, etc.), partitioning information can still be configured adaptively.

As already mentioned, tree-based partitioning can be performed recursively in this disclosure. For example, if the partition flag of a coded block with partition depth k is set to 0, then block encoding is performed in the coded block with partition depth k. If the partition flag of a coded block with partition depth k is set to 1, then block encoding is performed in N sub-coded blocks with partition depth k+1 according to the partitioning scheme (where N is an integer equal to or greater than 2, such as 2, 3, and 4).

In the above process, a sub-coded block can be set as a coded block (k+1) and partitioned into sub-coded blocks (k+2). This hierarchical partitioning scheme can be determined based on partitioning configurations such as partition range and allowed partition depth.

In this case, the bitstream structure representing the partition information can be selected from one or more scanning methods. For example, the bitstream of partition information can be configured based on the order of partition depth or based on whether partitioning was performed.

For example, in the case of partition depth order, partition information is obtained at the current depth level based on the initial block, and then at the next depth level. In the case of whether partitioning is performed, additional partition information is first obtained from the blocks split from the initial block, and other additional scanning methods can be considered.

Regardless of the tree type (or all trees), the maximum block size and minimum block size can have common settings, separate settings for each tree, or common settings for two or more trees. In this case, the maximum block size can be set to be equal to or less than the maximum encoded block size. If the maximum block size of the first tree is not the same as the maximum block size, partitioning is implicitly performed using a predetermined second-tree method until the maximum block size of the first tree is reached.

Furthermore, regardless of the tree type, a common split depth can be supported; a separate split depth can be supported for each tree; or a common split depth can be supported for two or more trees. Alternatively, split depth can be supported for some trees, while not for others.

It supports explicit syntax elements for setting information, and can also implicitly determine some setting information.

(Index-based partitioning)

In the index-based partitioning of this disclosure, both Constant Partition Index (CSI) and Variable Partition Index (VSI) schemes can be supported.

In the CSI scheme, k sub-blocks can be obtained by partitioning in a predetermined direction, where k can be an integer equal to or greater than 2, such as 2, 3, or 4. Specifically, the size and shape of the sub-blocks can be determined based on k, regardless of the size and shape of the blocks. The predetermined direction can be one or a combination of two or more of the following: horizontal, vertical, and diagonal (top left to bottom right or bottom left to top right).

In the index-based CSI partitioning scheme of this disclosure, z candidates can be obtained by partitioning in a horizontal or vertical direction. In this case, z can be an integer equal to or greater than 2, such as 2, 3, or 4, and the sub-blocks can be equal in width and height, and equal or different in the other of width and height. The width or height ratio of the sub-blocks is _A1 : _A2 : ... : _AZ , and each of _A1 to _AZ can be an integer equal to or greater than 1, such as 1, 2, or 3.

Furthermore, candidates can be obtained by partitioning the blocks horizontally and vertically into x and y partitions, respectively. Each of x and y can be an integer equal to or greater than 1, such as 1, 2, 3, or 4. However, candidates where both x and y are 1 may be restricted (because a already exists). Although Figure 4 shows a case where the sub-blocks have the same width or height ratio, candidates with different width or height ratios can also be included.

Furthermore, candidates can be divided into w partitions along one of the diagonal directions—top left to bottom right and bottom left to top right. In this paper, w can be an integer equal to or greater than 2, such as 2 or 3.

Referring to Figure 4, partition types can be classified into symmetric partition types (b) and asymmetric partition types (d and e) based on the length ratio of each sub-block. Furthermore, partition types can be classified into partition types concentrated in a specific direction (k and m) and central partition types (k). Partition types can be defined by various coding factors, including sub-block shape and sub-block length ratio, and the supported partition types can be implicitly or explicitly determined based on coding settings. Therefore, candidate groups can be determined based on the partition types supported in an index-based partitioning scheme.

In the VSI scheme, with a fixed width (w) or height (h) for each sub-block, one or more sub-blocks can be obtained through partitioning in a predetermined direction. In this document, each of w and h can be an integer equal to or greater than 1, such as 1, 2, 4, or 8. Specifically, the number of sub-blocks can be determined based on the size and shape of the block and the w or h value.

In the index-based VSI partitioning scheme of this disclosure, candidates can be partitioned into sub-blocks, each of which has a fixed width and length. Alternatively, candidates can be partitioned into sub-blocks, each of which has a fixed width and length. Since the width or height of the sub-blocks is fixed, equal partitions in either the horizontal or vertical direction are allowed. However, this disclosure is not limited thereto.

If the size of the block before partitioning is M×N, and the width (w) of each sub-block is fixed, the height (h) of each sub-block is fixed, or both the width (w) and height (h) of each sub-block are fixed, then the number of sub-blocks obtained can be (M*N)/w, (M*N)/h, or (M*N)/w/h.

Depending on the coding settings, only one or both of the CSI and VSI schemes may be supported, and information about the supported schemes may be determined implicitly or explicitly.

This disclosure will be described in the context of the supported CSI schemes.

Candidate groups can be configured to include two or more candidates from an index-based partitioning scheme, depending on the encoding settings.

For example, candidate groups such as {a, b, c}, {a, b, c, n}, or {a to g and n} can be formed. Candidate groups can include examples of block types predicted to occur multiple times based on general statistical characteristics, such as blocks divided into two partitions in the horizontal or vertical direction, or in each of the horizontal and vertical directions.

Alternatively, candidate groups such as {a, b}, {a, o}, or {a, b, o} can be configured, or candidate groups such as {a, c}, {a, p}, or {a, c, p}. A candidate group can be an example that includes candidates in which each candidate is partitioned into two partitions horizontally and four partitions vertically. This could be an example of configuring a candidate group with block types predicted to be primarily partitioned in a specific direction.

Alternatively, candidate groups such as {a, o, p} or {a, n, q} can be configured. This could be an example of configuring candidate groups to include block types that are predicted to be partitioned into many smaller partitions than the blocks before partitioning.

Alternatively, candidate groups such as {a, r, s} can be configured, and can be, for example, determining the best partitioning result of a non-square shape that can be obtained from the block before splitting by other methods (tree methods), and configuring the non-square shape as a candidate group.

As noted in the examples above, various candidate group configurations can be available, and one or more candidate group configurations can be supported, taking into account various encoding/decoding factors.

Once the candidate groups are fully configured, various partition information configurations can be made available.

For example, index selection information can be generated for a candidate group that includes unpartitioned candidates (a) and partitioned candidates (b to s).

Alternatively, information indicating whether to perform partitioning can be generated (information indicating whether the partition type is 'a'). If partitioning is performed (if the partition type is not 'a'), index selection information about the candidate group (b to s) including the candidates to be partitioned can be generated.

Partition information can be configured in many other ways than those described above. In addition to information indicating whether to perform a partition, binary bits can be assigned to the index of each candidate in the candidate group in various ways, such as fixed-length binarization, variable-length binarization, etc. If the number of candidates is 2, one bit can be assigned to the index selection information, and if the number of candidates is 3, one or more bits can be assigned to the index selection information.

Compared to tree-based partitioning schemes, partition types that are predicted to occur multiple times can be included in the candidate group in index-based partitioning schemes.

Since the number of bits used to represent index information can increase based on the number of supported candidate groups, this scheme is suitable for single-level partitioning (e.g., partition depth is limited to 0) rather than tree-based hierarchical partitioning (recursive partitioning). That is, single partitioning operations can be supported without further subdividing the sub-blocks obtained through index-based partitioning.

This may mean that further partitioning into smaller blocks of the same type is not possible (e.g., a coded block obtained through index-based partitioning cannot be further subdivided into coded blocks), and it also means that further partitioning into blocks of different types may also be impossible (e.g., it is not possible to partition a coded block into prediction blocks and coded blocks). Clearly, this disclosure is not limited to the above examples, and other modifications are also possible.

Now, a description will be given of determining the block partitioning configuration primarily based on the block type among the coding factors.

First, encoded blocks can be obtained during the partitioning process. A tree-based partitioning scheme can be used for the partitioning process, and partition types such as a (no partitioning), n (QT), b, c (BT), i, or l (TT) as shown in Figure 4 can be generated based on the tree type. Depending on the encoding configuration, various combinations of tree types, such as QT/QT+BT/QT+BT+TT, may be available.

The following example illustrates the process of dividing the coded blocks obtained in the above steps into prediction blocks and transform blocks. It assumes that prediction, transform, and inverse transform are performed based on the size of each partition.

In example (1), prediction can be performed by setting the size of the prediction block to be equal to the size of the coding block, and transformation and inverse transformation can be performed by setting the size of the transform block to be equal to the size of the coding block (or prediction block).

In example (2), prediction can be performed by setting the size of the prediction block to be equal to the size of the coding block. The transform block can be obtained by partitioning the coding block (or prediction block), and the transform and inverse transform can be performed based on the size of the obtained transform block.

Here, a tree-based partitioning scheme can be used for the partitioning process, and partition types such as a (no partitioning), n (QT), b, c (BT), i, or l (TT) as shown in Figure 4 can be obtained based on the tree type. Depending on the encoding configuration, various combinations of tree types, such as QT/QT+BT/QT+BT+TT, may be available.

Here, the partitioning process can be an index-based partitioning scheme. The partition type can be obtained based on the index type, for example, a (no partition), b, c, or d in Figure 4. Depending on the encoding configuration, various candidate groups such as {a, b, c} and {a, b, c, d} can be configured.

In example (3), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. For the transform block, its size is set to the size of the coded block, and a transform and inverse transform can be performed on the transform block. In this example, the prediction block and the transform block can be independent of each other.

Index-based partitioning schemes can be used in the partitioning process, and the partition type can be obtained based on the index type, such as a (no partition), b to g, n, r, or s in Figure 4. Various candidate groups can be configured according to the encoding configuration, such as {a, b, c, n}, {a to g, n}, and {a, r, s}.

In example (4), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. For the transform block, its size is set to the size of the prediction block, and a transform and inverse transform can be performed on the transform block. In this example, the transform block can have a size equal to the size of the obtained prediction block, or the obtained prediction block can have a size equal to the size of the transform block (setting the size of the transform block to the size of the prediction block).

Tree-based partitioning schemes can be used in the partitioning process, and partition types can be generated based on the tree type, such as a (no partition), b, c (BT), or n (QT) in Figure 4. Depending on the encoding configuration, various combinations of tree types, such as QT/BT/QT+BT, may be available.

Here, an index-based partitioning scheme can be used in the partitioning process, and the partition type can be obtained based on the index type, such as a (no partition), b, c, n, o, or p in Figure 4. Various candidate groups can be configured according to the encoding configuration, such as {a, b}, {a, c}, {a, n}, {a, o}, {a, p}, {a, b, c}, {a, o, p}, {a, b, c, n}, and {a, b, c, n, p}. Furthermore, candidate groups can be configured in a standalone VSI scheme or in a combination of CSI and VSI schemes as an index-based partitioning scheme.

In Example (5), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. Alternatively, a transform block can be obtained by partitioning the coded block, and the transform block undergoes transform and inverse transform based on the size of the obtained transform block. In this example, each of the prediction block and the transform block can be obtained by partitioning the coded block.

Here, tree-based partitioning schemes and index-based partitioning schemes can be used in the partitioning process, and candidate groups can be configured in the same way or in a similar manner as in Example 4.

In this context, the above examples represent possible scenarios depending on whether the process for partitioning each block type is shared, and should not be construed as limiting the scope of this disclosure. Various modifications are also possible. Furthermore, various encoding factors and block types can be considered to determine the block partitioning configuration.

Coding factors may include image type (I/P/B), color components (YCbCr), block size/shape/position, block aspect ratio, block type (coded block, prediction block, transform block, or quantization block), partition status, coding mode (intra-frame/inter-frame), prediction-related information (intra-frame prediction mode or inter-frame prediction mode), transform-related information (transform scheme selection information), and quantization-related information (quantization region selection information and quantization transform coefficient coding information).

In the image coding method according to an embodiment of the present invention, inter-frame prediction can be configured as follows. The inter-frame prediction of the prediction unit may include a reference image construction step, a motion estimation step, a motion compensation step, a motion information determination step, and a motion information coding step. Additionally, the video coding apparatus may include a reference image construction unit, a motion estimation unit, a motion compensation unit, a motion information determination unit, and a motion information coding unit that implement the reference image construction step, the motion estimation step, the motion compensation step, the motion information determination step, and the motion information coding step. Some of the above processes may be omitted, or other processes may be added, and the order may be changed from the above sequence.

In the video decoding method according to an embodiment of the present invention, inter-frame prediction can be configured as follows. The inter-frame prediction of the prediction unit may include a motion information decoding step, a reference image construction step, and a motion compensation step. Furthermore, the image decoding device may include a motion information decoding unit, a reference image construction unit, and a motion compensation unit that implement the motion information decoding step, the reference image construction step, and the motion compensation step. Some of the above processes may be omitted, or other processes may be added, and the order may be changed from the above order.

Since the reference image construction unit and motion compensation unit of the image decoding device perform the same functions as their counterparts in the image encoding device, detailed descriptions are omitted. Furthermore, the motion information decoding unit can be reverse-engineered using the methods employed in the motion information encoding unit. Here, the prediction block generated by the motion compensation unit can be sent to the addition unit.

Figure 5 is an example diagram illustrating various cases of obtaining prediction blocks through inter-frame prediction according to the present invention.

Referring to Figure 5, in unidirectional prediction, prediction block A can be obtained from previously encoded reference images (T-1, T-2) (forward prediction), or prediction block B can be obtained from later encoded reference images (T+1, T+2) (backward prediction). In bidirectional prediction, prediction blocks (C and D) can be generated from multiple previously encoded reference images (T-2 to T+2). Typically, P-image type supports unidirectional prediction, and B-image type supports bidirectional prediction.

As in the example above, the images referenced for the encoding of the current image can be obtained from memory, and a list of reference images can be constructed based on the current image (T) in chronological or display order, including reference images before and after the current image.

Inter-frame prediction (E) can be performed on the current image and on previous or subsequent images based on the current image. Performing inter-frame prediction on the current image can be referred to as non-directional prediction. This can be supported by either I-image type or P/B-image type, and the supported image type can be determined based on the coding settings. Performing inter-frame prediction in the current image uses spatial correlation to generate prediction blocks. Performing inter-frame prediction in other images for the purpose of using temporal correlation is different, but the prediction method (e.g., reference image, motion vector, etc.) may be the same.

Here, it is assumed that P-image and B-image are image types capable of performing inter-frame prediction, but this can also be applied to various image types that are added or replaced. For example, a particular image type may not support intra-frame prediction, but may only support inter-frame prediction, or may only support inter-frame prediction in a predetermined direction (backward direction), or may only support inter-frame prediction in a predetermined direction.

The reference image construction unit can construct and manage reference images for encoding the current image using a reference image list. At least one reference image list can be constructed based on encoding settings (e.g., image type, prediction orientation, etc.), and prediction blocks can be generated from the reference images included in the reference image list.

In the case of unidirectional prediction, inter-frame prediction can be performed on at least one reference image included in reference image list 0 (L0) or reference image list 1 (L1). Additionally, in the case of bidirectional prediction, inter-frame prediction can be performed on at least one reference image included in a combined list (LC) generated by combining L0 and L1.

For example, one-way prediction can be classified as forward prediction (Pred_L0) using a forward reference image list (L0) and backward prediction (Pred_L1) using a backward reference image list (L1). Two-way prediction (Pred_BI) can use both a forward reference image list (L0) and a backward reference image list (L1).

Alternatively, two or more forward predictions performed by copying the forward reference image list (L0) to the backward reference image list (L1) can be included in the bidirectional prediction, and two or more backward predictions performed by copying the backward reference image list (L0) to the forward reference image list (L1) can also be included in the bidirectional prediction.

The prediction direction can be indicated by flag information indicating the corresponding direction (e.g., inter_pred_idc, assuming this value can be adjusted by predFlagL0, predFlagL1, and predFlagBI). predFlagL0 indicates whether forward prediction is performed, and predFlagL1 indicates whether backward prediction is performed. Bidirectional prediction can be indicated by whether the prediction is performed via predFlagBI or by activating both predFlagL0 and predFlagL1 simultaneously (e.g., when each flag is 1).

This invention primarily describes the case of omnidirectional prediction using a list of omnidirectional reference images, but the same or modified application can be applied to other situations.

Typically, the following approach can be used: determine the optimal reference image for the image to be encoded by the encoder, and explicitly send information about the reference image to the decoder. To this end, the reference image construction unit can manage the list of images referenced for inter-frame prediction of the current image, and can set rules for reference image management considering limited memory size.

The transmitted information can be defined as a Reference Image Set (RPS), where images selected in the RPS are classified as reference images and stored in memory (or a Data Point Block (DPB), while images not selected in the RPS are classified as non-reference images. Images may be deleted from memory after one hour. A predetermined number of images (e.g., 14, 15, 16, or more) can be stored in memory, and the size of the memory can be set according to the image level and resolution.

Figure 6 is an example diagram of constructing a list of reference pictures according to an embodiment of the present invention.

Referring to Figure 6, typically, reference images (T-1, T-2) existing before the current image can be assigned to L0 and managed, and reference images (T+1, T+2) existing after the current image can be assigned to L1 and managed. When constructing L0, if the number of reference images for L0 is insufficient, reference images from L1 can be assigned. Similarly, when constructing L1, if the number of reference images for L1 is insufficient, reference images from L0 can be assigned.

Furthermore, the current image can be included in at least one list of reference images. For example, L0 or L1 can include the current image, and L0 can be constructed by adding a reference image (or the current image) with a chronological order of T to a list of reference images preceding the current image, while L1 can be constructed by adding a reference image with a chronological order of T to a list of reference images following the current image.

The construction of the reference image list can be determined based on the encoding settings.

The current image may not be included in the reference image list, but may be managed through a separate memory separate from the reference image list, or the current image may be included in at least one reference image list and managed thereafter.

For example, this can be determined using a signal (curr_pic_ref_enabled_flag) that indicates whether the current image is included in the list of reference images. Here, the signal can be implicitly determined or explicitly generated.

Specifically, when the signal is deactivated (e.g., curr_pic_ref_enabled_flag = 0), the current image may not be included as a reference image in any list of reference images. When the signal is activated (e.g., curr_pic_ref_enabled_flag = 1), whether to include the current image in a predetermined list of reference images can be implicitly determined (e.g., added only to L0, added only to L1, or added to both L0 and L1) or the relevant signal can be explicitly generated and determined (e.g., curr_pic_ref_from_l0_flag, curr_pic_ref_from_l1_flag). This signal can be supported in units such as sequences, images, sub-images, slices, tiles, and blocks.

Here, the current image can be positioned either first or last in the list of reference images shown in Figure 6, and the arrangement order in the list can be determined according to encoding settings (e.g., image type information). For example, type I can be positioned first, and type P/B can be positioned last, but the invention is not limited thereto, and other modified examples are also possible.

Alternatively, a separate reference image store can be supported based on a signal (ibc_enabled_flag) indicating whether block matching (or template matching) is supported in the current image. Here, the signal can be implicitly determined information or explicitly generated information.

Specifically, when the signal is deactivated (e.g., ibc_enabled_flag = 0), this may mean that block matching is not supported in the current image, and when the signal is activated (e.g., ibc_enabled_flag = 1), block matching can be supported in the current image, and reference image memory for this purpose can be supported. In this example, it is assumed that additional memory is provided, but it is also possible to configure block matching to be supported directly from existing memory supported by the current image without providing additional memory.

The reference image construction unit may include a reference image interpolation unit, and whether to perform interpolation processing on fractional-unit pixels can be determined based on the interpolation precision of the inter-frame prediction. For example, when the interpolation precision is in integer units, reference image interpolation processing can be omitted, and when the interpolation precision is in fractional units, reference image interpolation processing can be performed.

The interpolation filters used in the reference image interpolation process can be implicitly determined based on the encoding settings, or explicitly determined from among multiple interpolation filters. Depending on the encoding settings, the configuration of multiple interpolation filters can support fixed candidates or adaptive candidates, and the number of candidates can be an integer of 2, 3, 4 or more. The explicitly determined unit can be determined from a sequence, image, sub-image, slice, tile, block, or chunk.

Here, the encoding settings can be determined based on image type, color components, target block state information (e.g., block size, shape, horizontal/vertical length ratio, etc.), and inter-frame prediction settings (e.g., motion information encoding mode, motion model selection information, motion vector precision selection information, reference image, reference orientation, etc.). Motion vector precision can mean the precision of the motion vectors (i.e., pmv + mvd), but it can be replaced by the precision of the motion vector prediction values (pmv) or the motion vector difference (mvd).

Here, the interpolation filter can have a filter length with k taps, and k can be an integer of 2, 3, 4, 5, 6, 7, 8, or more. The filter coefficients can be derived from equations with various coefficient characteristics, such as Wiener and Kalman filters. The filter information used for interpolation (e.g., filter coefficients, tap information, etc.) can be implicitly determined or derived, or the relevant information can be explicitly generated. In this case, the filter coefficients can be configured to include 0.

Interpolation filters can be applied to a predetermined pixel unit, and the predetermined pixel unit can be restricted to an integer or fractional unit, or can be applied to both integer and fractional units.

For example, an interpolation filter can be applied to k integer unit pixels that are adjacent to the interpolation target pixel (i.e., fractional unit pixels) in the horizontal or vertical direction.

Alternatively, the interpolation filter can be applied to p integer units and q fractional units (p+q=k) adjacent to the interpolation target pixel in the horizontal or vertical direction. In this case, the precision referred to as the fractional unit used for interpolation (e.g., in 1/4 units) can be expressed as the same as or lower than the interpolation target pixel (e.g., 2/4 -> 1/2).

When only one of the x-component and y-component of the interpolated target pixel lies within a fractional unit, interpolation can be performed based on k pixels adjacent in the direction of the fractional component (e.g., the x-axis is horizontal and the y-axis is vertical). When both the x-component and y-component of the interpolated target pixel lie within a fractional unit, a first interpolation can be performed based on x pixels adjacent in either the horizontal or vertical direction, and interpolation can be performed based on y pixels adjacent in the other direction. In this case, the case where x and y are the same as k will be described, but the case is not limited to this, and it is also possible for x and y to be different.

The interpolated target pixel can be obtained through interpolation precision (1/m), where m can be an integer of 1, 2, 4, 8, 16, 32, or higher. The interpolation precision can be implicitly determined based on encoding settings, or it can be explicitly generated based on relevant information. Encoding settings can be limited based on image type, color components, reference image, motion information encoding mode, motion model selection information, etc. The explicitly determined unit can be determined from a sequence, sub-image, slice, tile, or block.

Through the above processing, the interpolation filter settings for the target block can be obtained, and reference image interpolation can be performed based on them. Furthermore, detailed interpolation filter settings based on these settings can be obtained, and reference image interpolation can be performed based on them. That is, while it's possible to obtain the interpolation filter settings for the target block using a fixed candidate, it's also possible to use multiple candidates. In the example described later, it's assumed that the interpolation precision is 1/16 (e.g., 15 pixels to be interpolated).

As an example of setting up detailed interpolation filters, one of a plurality of candidates can be adaptively used for the first pixel unit at a predetermined position, and a predetermined interpolation filter can be used for the second pixel unit at the predetermined position.

The first pixel unit can be determined among all fractional units of pixels supported by interpolation precision (1/16 to 15/16 in this example), and the number of pixels included in the first pixel unit can be a, and a can be determined between 0, 1, 2, ..., (m-1).

The second pixel unit may include pixels excluded from the first pixel unit from the total number of pixels in the fractional units, and the number of pixels included in the second pixel unit can be obtained by subtracting the number of pixels in the first pixel unit from the total number of pixels to be interpolated. In this example, an example of dividing the pixel unit into two will be described, but the invention is not limited thereto and may divide it into three or more.

For example, the first pixel unit can be a unit represented by multiple units such as 1/2, 1/4, 1/8. For example, in the case of a 1/2 unit, it can be a pixel located at 8/16, and in the case of a 1/4 unit, it can be a pixel located at {4/16, 8/16, 12/16}, and in the case of a 1/8 unit, it can be a pixel located at {2/16, 4/16, 6/16, 8/16, 10/16, 12/16, 14/16}.

Detailed interpolation filter settings can be determined based on the coding settings, which can be limited by image type, color components, target block state information, inter-frame prediction settings, etc. Below are examples of detailed interpolation filter settings based on various coding elements. For ease of explanation, assume a is 0 and c is or greater (c is an integer greater than or equal to 1).

For example, in the case of color components (luminance, chrominance), a can be (1,0), (0,1), (1,1). Alternatively, in the case of motion information encoding mode (merging mode, competing mode), a can be (1,0), (0,1), (1,1), (1,3). Or, in the case of motion model selection information (translational motion, non-translational motion A, non-translational motion B), a can be (0,1,1), (1,0,0), (1,1,1), (1,3,7). Alternatively, in the case of motion vector precision (1/2, 1/4, 1/8), a can be (0,0,1), (0,1,0), (1,0,0), (0,1,1), (1,0,1), (1,1,0), (1,1,1). Alternatively, in the case of reference images (current image, other images), a can be (0,1), (1,0), (1,1).

As described above, interpolation can be performed by selecting one of several interpolation precisions, and when interpolation based on adaptive interpolation precision is supported (e.g., adaptive_ref_resolution_enabled_flag. If it is 0, a predetermined interpolation precision is used, and if it is 1, one of several interpolation precisions is used), precision selection information (e.g., ref_resolution_idx) can be generated.

Motion estimation and compensation can be performed based on interpolation accuracy, and the representation and storage units for motion vectors can also be determined based on interpolation accuracy.

For example, when the interpolation precision is 1/2 unit, motion estimation and compensation processing can be performed at 1/2 unit, and motion vectors can be represented at 1/2 unit and used in the encoding process. Furthermore, motion vectors can be stored at 1/2 unit and can be referenced in the encoding process of another block of motion information.

Alternatively, when the interpolation precision is 1/8, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/8 units and used in the encoding process and stored in 1/8 units.

Additionally, motion estimation and compensation processing, as well as motion vectors, can be performed, represented, and stored in units different from interpolation precision, such as integers, 1/2 units, and 1/4 units. These units can be adaptively determined based on inter-frame prediction methods/settings (e.g., motion estimation/compensation methods, motion model selection information, motion information coding modes, etc.).

As an example, assuming an interpolation precision of 1/8, in the case of a translational motion model, motion estimation and compensation can be performed in 1/4 units, and motion vectors can be represented in 1/4 units (assuming units in the encoding process in this example), and stored in 1/8 units. In the case of a non-translational motion model, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/4 units, and stored in 1/8 units.

For example, assuming an interpolation precision of 1/8 unit, in the case of block matching, motion estimation and compensation can be performed at 1/4 unit, and motion vectors can be represented at 1/4 unit and stored at 1/8 unit. In the case of template matching, motion estimation and compensation processing can be performed at 1/8 unit, and motion vectors can be represented at 1/8 unit and stored at 1/8 unit.

For example, assuming an interpolation precision of 1/16, in race mode, motion estimation and compensation processing can be performed in 1/4 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units. In merge mode, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units. In skip mode, motion estimation and compensation can be performed in 1/16 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units.

In summary, the units for motion estimation, motion compensation, motion vector representation, and storage can be adaptively determined based on the inter-frame prediction method or settings and the interpolation accuracy. Specifically, the units for motion estimation, motion compensation, and motion vector representation can be adaptively determined based on the inter-frame prediction method or settings, and the storage unit for motion vectors is typically determined based on the interpolation accuracy, but is not limited to this, and various modified examples are feasible. Furthermore, the examples above have illustrated an example based on one category (e.g., motion model selection information, motion estimation/compensation method, etc.), but settings can also be determined when two or more categories are mixed.

Furthermore, as mentioned above, the interpolation accuracy information can have a predetermined value or can be selected as one of multiple accuracies, and the reference image interpolation accuracy is determined based on the motion estimation and compensation settings supported by the inter-frame prediction method or settings. For example, when the translational motion model supports up to 1/8 unit and the non-translational motion model supports up to 1/16 unit, interpolation can be performed based on the accuracy unit of the non-translational motion model with the highest accuracy.

In other words, reference image interpolation can be performed based on settings for the accuracy information used to support it—such as translational motion model, non-translational motion model, competition mode, merging mode, and skip mode. In this case, the accuracy information can be determined implicitly or explicitly, and when the relevant information is explicitly generated, it can be included in units such as sequences, images, sub-images, slices, tiles, and blocks.

A motion estimation unit refers to the process of estimating (or searching) whether a target block has a high correlation with a predetermined block of a predetermined reference image. The size and shape (M×N) of the target block—on which prediction is performed—can be obtained from the block partitioning unit. As an example, the target block can be determined in the range of 4×4 to 128×128. Typically, inter-frame prediction can be performed on a block-by-block basis, but it can be performed on a block-by-block basis, such as a coded block or a transform block, depending on the block partitioning unit settings. Estimation can be performed within an estimable range of the reference region, and at least one motion estimation method can be used. In the motion estimation method, the estimation order and conditions can be defined for each pixel.

Motion estimation can be performed based on motion estimation methods. For example, in the case of block matching, the region to be compared for motion estimation processing can be the target block, and in the case of template matching, it can be a predetermined region (template) set around the target block. In the former case, the block with the highest relevance can be found within the estimable range of the target block and the reference region, and in the latter case, the region with the highest relevance can be found within the estimable range of the template defined according to the encoding settings and the reference region.

Motion estimation can be performed based on motion models. Additional motion models, besides translational motion models that only consider parallel motion, can be used to perform motion estimation and compensation. For example, motion models that consider motions such as rotation, perspective, zoom in/out, and parallel motion can be used to perform motion estimation and compensation. This can be supported to improve coding performance by generating prediction blocks that reflect the various types of motions occurring based on the regional features of the image.

Figure 7 is a conceptual diagram illustrating a non-translational motion model according to an embodiment of the present invention.

Referring to Figure 7, as an example of an affine model, an example is shown where motion information is represented based on motion vectors _V0 and _V1 at predetermined positions. Since motion can be represented based on multiple motion vectors, accurate motion estimation and compensation are possible.

As in the example above, inter-frame prediction is performed based on a predefined motion model, but inter-frame prediction based on an additional motion model can also be supported. Here, it is assumed that the predefined motion model is a translational motion model and the additional motion model is an affine model, but the invention is not limited to this, and various modifications are possible.

In the case of a translational motion model, motion information (assuming unidirectional prediction) can be represented based on a motion vector, and the control point (reference point) used to indicate the motion information is assumed to be the upper left coordinate, but is not limited to this.

In the case of a non-translational motion model, it can be represented as motion information for various configurations. In this example, it is assumed that the configuration is represented as additional information in a motion vector (relative to the top-left corner coordinates). Some motion estimation and compensation mentioned in the examples described later can be performed not on a block-by-block basis, but on a predetermined sub-block basis. In this case, the size and position of the predetermined sub-blocks can be determined based on each motion model.

Figure 8 is an example diagram illustrating motion estimation on a sub-block basis according to an embodiment of the present invention. Specifically, it illustrates motion estimation on a sub-block basis based on an affine model (two motion vectors).

In the case of a translational motion model, the motion vectors of the pixel units included in the target block can be the same. That is, the motion vectors can be applied to the pixel units together, and a single motion vector ( _V0 ) can be used to perform motion estimation and compensation.

In the case of a non-translational motion model (affine model), the motion vectors of the pixel units included in the target block may not be the same, and a separate motion vector may be required for each pixel. In this case, the motion vectors of the pixel units or sub-block units can be derived based on the motion vectors ( _V0 , _V1 ) at the positions of predetermined control points of the target block, and the derived motion vectors can be used to perform motion estimation and compensation.

For example, the motion vector of a sub-block or pixel unit within the target block (e.g., ( _Vx , _Vy )) can be derived from the equations _Vx = ( _V1x - _V0x )×x/M - ( _V1y - _V0y )×y/N + _V0x , _Vy = ( _V1y - _V0y )×x/M + ( _V1x - _V0x )×y/N + _V0y . In these equations, _V0 (in this example, ( _V0x , _V0y )) refers to the motion vector at the top left of the target block, and _V1 (in this example, ( _V1x , _V1y )) refers to the motion vector at the top right of the target block. Considering the complexity, motion estimation and compensation for a non-translational motion model can be performed on a sub-block basis.

Here, the size of the sub-block (M×N) can be determined according to the encoding settings and can have a fixed size or can be set to an adaptive size. M and N can be integers of 2, 4, 8, 16, or more, and M and N can be the same or different. The size of the sub-block can be explicitly generated in units such as sequences, pictures, sub-pictures, slices, tiles, and blocks. Alternatively, it can be implicitly determined by a mutual commitment between the encoder and decoder, or it can be determined by the encoding settings.

Here, the encoding settings can be defined by one or more elements, including state information, image type, color components, and inter-frame prediction settings for the target block (motion information encoding mode, reference image information, interpolation accuracy, motion model selection information, etc.).

In the examples above, the process of deriving the size of the sub-blocks based on a predetermined non-translational motion model and performing motion estimation and compensation accordingly has been described. As in the examples above, motion estimation and compensation can be performed on a sub-block or pixel-by-pixel basis according to the motion model, and a detailed description of this will be omitted.

The following are various examples of motion information constructed based on motion models.

For example, in the case of a motion model representing rotational motion, the translational motion of a block can be represented by a motion vector, and the rotational motion can be represented by rotation angle information. The rotation angle information can be measured relative to a predetermined position (e.g., the top-left coordinate) and can be represented as k candidates (k is an integer of 1, 2, 3 or more) with predetermined intervals (e.g., angle differences of 0 degrees, 11.25 degrees, 22.25 degrees, etc.) within a predetermined angle range (e.g., between -90 degrees and 90 degrees).

Here, the rotation angle information can be encoded as is during motion information encoding processing, or it can be encoded based on the motion information of adjacent blocks (e.g., motion vectors, rotation angle information) (e.g., prediction + difference information).

Alternatively, the translational motion of a block can be represented by a single motion vector, and the rotational motion of a block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vector can be encoded as is during motion information encoding processing, or encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational motion models or non-translational motion models) (e.g., prediction + difference information), or encoded based on another motion vector in a block representing rotational motion (e.g., prediction + difference information).

For example, in the case of a motion model representing sizing or scaling movements such as zooming in/out, the translational motion of a block can be represented by a single motion vector, and the scaling motion can be represented by scaling information. Scaling information can be represented as scaling information indicating expansion or reduction in the horizontal or vertical direction based on a predetermined position (e.g., top-left coordinate).

Here, scaling can be applied in at least one of the horizontal or vertical directions. Alternatively, it can support individual scaling information applied in the horizontal or vertical directions, or scaling information applied in a common direction. The width and height of the scaling block can be added to a predetermined position (top-left coordinate) to determine the location used for motion estimation and compensation.

Here, scaling information can be encoded as is during motion information encoding processing, or it can be encoded based on motion information of neighboring blocks (e.g., motion vectors, scaling information) (e.g., prediction + difference information).

Alternatively, the translation of a block can be represented by a single motion vector, and the size adjustment of the block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vectors can be encoded as is during motion information encoding processing, or they can be encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational motion models or non-translational motion models) (e.g., prediction + difference information), or they can be encoded based on predetermined coordinates within the block (e.g., lower right coordinates) (e.g., prediction + difference).

In the examples above, the representation of some motions has been described, and it can be represented as motion information for multiple motions.

For example, in the case of motion models representing various or complex motions, the translational motion of a block can be represented by a motion vector, the rotational motion can be represented by rotation angle information, and the sizing adjustment can be represented by scaling information. Since the description of each motion can be derived from the above examples, detailed descriptions will be omitted.

Alternatively, the translational motion of a block can be represented by a single motion vector, and other motions of the block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vectors can be encoded as is during motion information encoding processing, or they can be encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational or non-translational motion models) (e.g., prediction + difference information), or they can be encoded based on other motion vectors within blocks representing various motions (e.g., prediction + difference information).

The above description may pertain to affine models and will be based on the presence of one or two additional motion vectors. In summary, it is assumed that the number of motion vectors used according to the motion model can be 1, 2, or 3, and that a single motion model can be considered based on the number of motion vectors used to represent motion information. Furthermore, when only one motion vector exists, it is assumed to be a predefined motion model.

Multiple motion models can be supported for inter-frame prediction, and this can be determined by a signal indicating support for additional motion models (e.g., `adaptive_motion_mode_enabled_flag`). Here, a signal of 0 indicates support for predefined motion models, and a signal of 1 indicates support for multiple motion models. This signal can be generated in units such as sequences, images, sub-images, slices, tiles, blocks, chunks, etc., but if it is not possible to check them individually, the signal value can be assigned according to predefined settings. Alternatively, whether support is implicit can be determined based on encoding settings. Alternatively, the implicit or explicit case can be determined based on encoding settings. Here, encoding settings can be limited by one or more elements of image type, image class (e.g., a normal image if 0, a 360-degree image if 1), or color components.

Whether multiple motion models are supported can be determined through the process described above. The following assumes that two or more additional motion models are supported, and that multiple motion models are supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., but some configurations may be excluded. In the example described later, it is assumed that motion models A, B, and C are supported, where A is the fundamentally supported motion model, and B and C are additionally supported motion models.

Configuration information about supported motion models can be generated within the unit. That is, configurations for supported motion models such as {A, B}, {A, C}, and {A, B, C} are feasible.

For example, indices (0 to 2) can be assigned to candidates for the above configurations and selected. If index 2 is selected, a motion model configuration supporting {A, C} can be determined, and if index 3 is selected, a motion model configuration supporting {A, B, C} can be determined.

Alternatively, information indicating whether a predetermined motion model is supported can be supported individually. That is, a flag indicating whether B is supported or a flag indicating whether C is supported can be generated. If both flags are 0, then only A is supported. This example could be an instance where information indicating whether multiple motion models are supported is not generated and processed.

When candidate groups of supported motion models are configured as in the example above, the motion model of one of the candidate groups can be explicitly identified and used on a block-by-block basis, or it can be used implicitly.

Typically, a motion estimation unit can be a component present in the encoding device, but it can also be a component that can be included in the decoding device depending on the prediction method (e.g., template matching). For example, in the case of template matching, the decoder can obtain the motion information of the target block by performing motion estimation via a template adjacent to the target block. In this case, motion estimation related information (e.g., motion estimation range, motion estimation method <scan order>, etc.) can be implicitly determined or explicitly generated and included in units such as sequences, pictures, sub-pictures, slices, tiles, blocks, etc.

The motion compensation unit refers to the process of obtaining data of some blocks of a predetermined reference image determined by motion estimation processing as predicted blocks of a target block. Specifically, the predicted blocks of the target block can be generated from at least one region (or block) of at least one reference image based on motion information (e.g., reference image information, motion vector information, etc.) obtained through motion estimation processing.

Motion compensation can be performed based on the following motion compensation methods.

In the case of block matching, the predicted block of the target block can be compensated with data of the regions corresponding to the right M and down N relative to (P _{x + V x} , P _y + V _y ), which are explicitly obtained from the reference image and are obtained through the motion vector (V _x , V _y ) of the target block (M _{× N} ) and the top-left coordinates (P _x , P _y ) of the target block.

In the case of template matching, the predicted block of the target block can be compensated with data of the regions corresponding to the right M and down N relative to (P _{x + V x} , P _y + V _y ), which are implicitly obtained from the reference image and are obtained through the motion vector (V _x , V _y ) of the target block (M _{× N} ) and the top-left coordinates (P _x , P _y ) of the target block.

Alternatively, motion compensation can be performed based on the following motion model.

In the case of the translational motion model, the predicted block of the target block can be compensated with data of the regions corresponding to the _{right M} and down N relative to ( _Px + _Vx , _Py + _Vy ), which are explicitly obtained from the reference image and are obtained by a motion vector ( _Vx , _Vy ) of the target block (M×N ₎ and the upper left coordinates ( _Px , _{Py )} of the target block.

In the case of a non-translational motion model, the predicted sub-blocks can be compensated using data from regions M to the right and N downwards relative to ( _Pmx + _Vnx , _Pmy + _Vny ). This ( _Pmx + _Vnx , _Pmy + _Vny ) is obtained by implicitly acquiring the motion vectors (Vmx, Vny) of m×n sub-blocks via multiple motion vectors ( _V0x , _V0y ) and ( _V1x , _V1y ) of the target block (M×N) explicitly obtained from the reference image _{, along with the top-left coordinates (Pmx , Pny} ) of each sub-block. In other words, the predicted sub-blocks can be collected and compensated for by _the predicted sub-blocks of the target block.

The motion information determination unit can perform processing for selecting the optimal motion information for the target block. Typically, the optimal motion information can be determined based on coding cost and rate distortion techniques using block distortion (e.g., distortion of the target block and the reconstructed block, such as SAD (sum of absolute differences), SSD (sum of squared differences), etc.), where the rate distortion technique considers the bit amount generated based on the corresponding motion information. The predicted block generated based on the motion information determined through the above processing can be sent to the subtraction and addition units. Alternatively, it can be configured according to some prediction methods (e.g., template matching, etc.) that can be included in the decoding device, and in this case, it can be determined based on the block distortion.

In the motion information determination unit, settings related to inter-frame prediction can be considered, such as motion compensation methods and motion models. For example, when multiple motion compensation methods are supported, the motion compensation method selection information, the corresponding motion vectors, and reference image information can be the optimal motion information. Alternatively, when multiple motion models are supported, the motion model selection information, its corresponding motion vectors, and reference image information can be the optimal motion information.

The motion information encoding unit can encode the motion information of the target block obtained through motion information determination processing. In this case, the motion information can consist of information about the region and image referenced for the prediction of the target block. Specifically, it can consist of information about the reference image (e.g., reference image information) and information about the reference region (e.g., motion vector information).

Additionally, settings related to inter-frame prediction (or selection information, such as motion estimation/compensation methods, motion model selection information, etc.) can be included in the motion information of the target block. Information about the reference image and reference region (e.g., the number of motion vectors, etc.) can be configured based on settings related to inter-frame prediction.

Motion information can be encoded by configuring a reference image and information about the reference region into a combination, and the combination of the reference image and information about the reference region can be configured as a motion information encoding mode.

Here, information about the reference image and reference region can be obtained based on adjacent blocks or predefined information (e.g., images encoded before or after the current image, zero motion vectors, etc.).

Neighboring blocks can be classified into blocks <inter_blk_A> that belong to the same space as the target block and are closest to the target block, blocks <inter_blk_B> that belong to the same space and are far from the target block, and blocks <inter_blk_C> that belong to a different space from the target block. Information about the reference image and reference region can be obtained based on neighboring blocks (candidate blocks) that belong to at least one of the categories.

For example, the motion information of the target block can be encoded based on the motion information of the candidate block or the reference image information, and the motion information of the target block can also be encoded based on the information derived from the motion information of the candidate block or the reference image information (or information median, transformation processing, etc.). In other words, the motion information of the target block can be predicted from the candidate block, and information about the motion information can be encoded.

In this invention, the motion information of the target block can be encoded based on one or more motion information encoding modes. Here, the motion information encoding modes can be defined in various ways and can include one or more of skip modes, merge modes, and competition modes.

Based on the template matching (tmp) described above, it can be combined with motion information coding modes, or it can be supported as a standalone motion information coding mode, or it can be included in the detailed configuration of all or some motion information coding modes. This presupposes that template matching is supported at a higher unit (e.g., picture, sub-picture, slice, etc.), but the flags regarding support can be considered as part of the inter-frame prediction settings.

Based on the method described above for performing block matching (ibc) within the current image, it can be combined with motion information coding modes, or it can be supported as a standalone motion information coding mode, or it can be included in the detailed configuration of all or some motion information coding modes. This presupposes that block matching is supported in a higher unit within the current image, but the flag regarding support can be considered part of the inter-frame prediction settings.

Based on the aforementioned motion model (affine), motion information coding modes can be combined with other motion information coding modes, or they can be supported as standalone motion information coding modes, or they can be included in the detailed configuration of all or some motion information coding modes. This presupposes that higher units support non-translational motion models, but the flags regarding support can be considered part of the inter-frame prediction settings.

For example, single motion information encoding patterns such as temp_inter, temp_tmp, temp_ibc, and temp_affine can be supported. Alternatively, combined motion information encoding patterns such as temp_inter_tmp, temp_inter_ibc, temp_inter_affine, and temp_inter_tmp_ibc can be supported. Alternatively, the motion information prediction candidate group for constructing temp can include template-based candidates, candidates based on methods for performing block matching in the current image, and affine-based candidates.

Here, temp can refer to skip mode, merge mode, and comp mode. For example, it is the same as the motion information encoding modes such as skip_inter, skip_tmp, skip_ibc, and skip_affine in skip mode; merge_inter, merge_tmp, merge_ibc, and merge_affine in merge mode; and comp_inter, comp_tmp, comp_ibc, and comp_affine in comp mode.

When skip mode, merge mode, and competition mode are supported, and candidates considering the above factors are included in the motion information prediction candidate group for each mode, a mode can be selected by distinguishing between the flags for skip mode, merge mode, and competition mode. For example, a flag indicates whether skip mode is supported; if the value is 1, skip mode is selected. A flag indicates whether merge mode is supported; if the value is 1, merge mode is selected, and if the value is 0, competition mode can be selected. Additionally, candidates based on inter, tmp, ibc, and affine can be included in the motion information prediction candidate group for each mode.

Alternatively, when multiple motion information encoding modes are supported under a common mode, in addition to the flags used to select one of the skip mode, merge mode, and competition mode, additional flags can be supported to distinguish the detailed mode of the selected mode. For example, when the merge mode is selected, this means additionally supporting flags to select a detailed mode among merge_inter, merge_tmp, merge_ibc, merge_affine, etc.—which are detailed modes associated with the merge mode. Alternatively, flags indicating whether it is merge_inter can be supported, and if it is not merge_inter, additional flags for selection among merge_tmp, merge_ibc, merge_affine, etc. can be supported.

The encoding settings can support all or part of the motion information encoding mode candidates. Here, the encoding settings can be limited by more than one of the following elements: target block state information, image type, image category, color components, and inter-frame prediction support settings (e.g., whether template matching is supported in the current image, non-translation motion model support elements, etc., and whether block matching is supported).

For example, the supported motion information encoding patterns can be determined based on the block size. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H of the block's width (W) and height (H). With the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. With the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by either the first or second threshold size, or both.

In this scenario, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this case, a first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and a second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

For example, supported motion information encoding modes can be determined based on the image type. In this case, the image type can include at least one of skip mode, merge mode, and competition mode. In this case, a method for performing block matching (or template matching) in the current image, a single motion information encoding mode for an affine model (hereinafter referred to as an element) can be supported, or motion information encoding modes can be supported by combining two or more elements. Alternatively, elements other than the predetermined motion information encoding modes can be configured as motion information prediction candidate groups.

P/B image types may include at least one of skip mode, merge mode, and competition mode. In this case, methods for performing general inter-frame prediction, template matching, block matching on the current image can be supported, as well as a single motion information coding mode for an affine model (hereinafter referred to as an element), or motion information coding modes can be supported by combining two or more elements. Alternatively, elements other than the predetermined motion information coding mode can be configured as motion information prediction candidate groups.

Figure 9 is a flowchart illustrating the encoding of motion information according to an embodiment of the present invention.

Referring to Figure 9, the motion information encoding mode S900 of the target block can be checked.

Motion information coding modes can be defined by combining and setting predetermined information (e.g., motion information) used for inter-frame prediction. Predetermined information may include one or more of the following: predicted motion vector, motion vector difference, motion vector difference accuracy, reference image information, reference orientation, motion model information, information about the presence or absence of residual components, etc. Motion information coding modes may include at least one of skip mode, merge mode, and contention mode, and may include other additional modes.

The configuration and settings of explicitly generated information and implicitly determined information can be determined based on the motion information encoding pattern. In this case, in the explicit case, each piece of information can be generated individually or in combination (e.g., in the form of an index).

For example, in skip or merge modes, the predicted motion vector, reference image information, and/or reference orientation can be defined based on (a) predetermined index information. In this case, the index may include one or more candidates, and each candidate can be set based on the motion information of a predetermined block (e.g., the predicted motion vector, reference image information, and reference orientation of the corresponding block are composed of a combination). Additionally, motion vector differences (e.g., zero vectors) can be implicitly handled.

For example, in competition mode, the predicted motion vector, reference image information, and/or reference orientation can be defined based on (one or more) predetermined index information. In this case, the index can support each information segment individually, or a combination of two or more information segments can be supported. In this case, the index can include one or more candidates, and each candidate can be set based on the motion information of a predetermined block (e.g., predicted motion vector, etc.), or can be configured with predetermined values (e.g., the interval between current images is processed as 1, 2, 3, etc., and the reference orientation information is processed as L0, L1, etc.) (e.g., reference image information, reference orientation information, etc.). In addition, motion vector differences can be explicitly generated, and motion vector difference accuracy information can be additionally generated based on the motion vector difference (e.g., if it is not a zero vector).

As an example, in skip mode, information about the presence or absence of residual components can be implicitly processed (e.g., processed as <cbf_flag=0>), and the motion model can be implicitly processed as a predetermined value (e.g., supporting parallel motion models).

For example, in merge or competition modes, information about the presence or absence of residual components can be explicitly generated, and the motion model can be explicitly selected. In this case, information for classifying the motion model can be generated under a single motion information encoding mode, or separate motion information encoding modes can be supported.

In the above description, when supporting one index, index selection information may not be generated, while when supporting two or more indexes, index selection information may be generated.

In this invention, a candidate group comprising two or more indices is referred to as a motion information prediction candidate group. Furthermore, existing motion information encoding patterns can be modified or new motion information encoding patterns can be supported through various predetermined information and settings.

Referring to Figure 9, the predicted motion vector S910 of the target block can be obtained.

The predicted motion vector can be determined as a predetermined value, or a candidate set of predicted motion vectors can be constructed and selected from the candidate set. When the predicted motion vector can be determined as a predetermined value, it implies an implicit determination; and when a candidate set of predicted motion vectors can be constructed and selected from the candidate set, index information for selecting the predicted motion vector can be explicitly generated.

Here, the predetermined value can be set based on the motion vector of a block at a predetermined position (e.g., left, top, top left, top right, bottom left, etc.), or it can be set based on the motion vectors of two or more blocks, or it can be set to a default value (e.g., zero vector).

Depending on the motion information encoding mode, the predicted motion vectors can have the same or different candidate group construction settings. For example, skip mode/merge mode/competition mode can support a, b, and c prediction candidates respectively, and the number of each candidate can be the same or different. In this case, a, b, and c can include integers greater than or equal to 1, such as 2, 3, 5, 6, 7, etc. The order, settings, etc., of constructing candidate groups will be described through other implementations. In the examples described later, the construction of the skip mode candidate group will be described based on the same assumptions as the merge mode, but it is not limited to this, and some can have different constructions.

The predicted motion vectors obtained based on the index information can be used as is to reconstruct the motion vectors of the target block, or they can be adjusted based on the distance between reference images and the reference orientation.

For example, in competitive mode, reference image information can be generated separately. When the distance between the current image and the reference image of the target block differs from the distance between the image containing the predicted motion vector candidate block and the reference image of the corresponding block, adjustments can be made based on the distance between the current image and the reference image of the target block.

Furthermore, the number of motion vectors derived from the motion model can be different. That is, in addition to the motion vector at the upper left control point, motion vectors for the upper right and lower left control points can also be derived from the motion model.

Referring to Figure 9, the motion vector difference S920 of the target block can be reconstructed.

In skip mode and merge mode, the motion vector difference can be obtained as a zero vector value, and in competition mode, the difference information of each component can be reconstructed.

Motion vectors can be represented according to a predetermined precision (e.g., based on interpolation precision settings or motion model selection information settings). Alternatively, they can be represented based on one of several precisions, and predetermined information for this purpose can be generated. In this case, the predetermined information can relate to the selection of motion vector precision.

For example, when one component of the motion vector is 32/16, motion component data for 32 can be obtained based on an implicit precision setting in units of 1/16 pixels. Alternatively, the motion component can be converted to 2/1 based on an explicit precision setting in units of 2 pixels, and motion component data for 2 can be obtained.

The above configuration can be a method for accurately processing motion vectors with the same precision, or it can be obtained by adding the motion vector difference to a predicted motion vector obtained through the same precision transformation process.

Alternatively, the motion vector difference can be represented with a predetermined precision, or it can be represented with one of multiple precisions. In the case where the motion vector difference can be represented with one of multiple precisions, precision selection information can be generated.

Precision can include at least one of 1/16, 1/8, 1/4, 1/2, 1, 2, 4 pixel units, and the number of candidates can be an integer greater than or equal to 1, 2, 3, 4, 5. Here, the supported precision candidate group configuration (e.g., division by number, candidate, etc.) can be explicitly determined in units such as sequences, pictures, sub-pictures, slices, tiles, and blocks. Alternatively, it can be implicitly determined based on encoding settings, which can be defined by at least one or more elements of image type, reference image, color components, and motion model selection information.

The following describes a case where precision candidates are formed based on various coding elements and precision is described by assuming a difference in motion vectors, which can be similar to or applicable to motion vectors.

For example, in the case of a translational motion model, candidate groups such as {1/4, 1}, {1/4, 1/2}, {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} can be configured, and in the case of a non-translational motion model, precision candidate groups such as {1/16, 1/4, 1}, {1/16, 1/8, 1}, {1/16, 1, 4}, {1/16, 1/4, 1, 2}, and {1/16, 1/4, 1, 4} can be configured. This assumes that the minimum precision supported in the case of the translational motion model is 1/4 pixel unit, while the minimum precision supported in the case of the non-translational motion model is 1/16 pixel unit, and additional precision beyond the minimum precision can be included in the candidate groups.

Alternatively, when the reference image is a different image, candidate groups such as {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} can be configured. Additionally, when the reference image is the current image, candidate groups such as {1, 2}, {1, 4}, and {1, 2, 4} can be configured. In this case, the configuration for the reference image being the current image can be as follows: for block matching, interpolation is not performed in fractional units, and if interpolation is performed in fractional units, it can be configured to include fractional units in the precision candidates.

Alternatively, when the color component is a luminance component, candidate groups such as {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} are feasible. And when the color component is a chrominance component, candidate groups such as {1/8, 1/4}, {1/8, 1/2}, {1/8, 1}, {1/8, 1/4, 1/2}, {1/8, 1/2, 2}, and {1/8, 1/4, 1/2, 2} are feasible. In this case, when the color component is a chrominance component, candidate groups proportional to the luminance component can be formed based on the color component composition ratio (e.g., 4:2:0, 4:2:2, 4:4:4, etc.), or a single candidate group can be configured.

Although the above examples describe the case where there are multiple precision candidates for each coded element, it is possible for there to be a single candidate configuration (i.e., represented with a predetermined minimum precision).

In summary, the motion vector difference with minimum accuracy can be reconstructed based on the accuracy of the motion vector difference in the competition mode, and the motion vector difference with zero vector value can be obtained in the skip mode and the merge mode.

Here, the number of motion vector differences reconstructed from the motion model can be different. That is, in addition to the motion vector difference of the position of the upper left control point, the motion vector differences of the positions of the upper right and lower left control points can also be derived from the motion model.

Alternatively, a single motion vector difference precision can be applied to multiple motion vector differences, or a single motion vector difference precision can be applied to each motion vector difference, depending on the encoding settings.

Additionally, information about the accuracy of the motion vector difference can be generated when at least one motion vector difference is not zero, and otherwise it can be omitted, but is not limited to this.

Referring to Figure 9, the motion vector S930 of the target block can be reconstructed.

The motion vector of the target block can be reconstructed by obtaining the predicted motion vector and the difference between the motion vectors—which are obtained through processing prior to this step—and summing them. In this case, when reconstructing by selecting one of multiple accuracies, accuracy unification processing can be performed.

For example, a motion vector obtained by adding the predicted motion vector (related to the accuracy of the motion vector) and the motion vector difference can be reconstructed based on accuracy selection information. Alternatively, the motion vector difference (related to the accuracy of the differential motion vector) can be reconstructed based on accuracy selection information, and the motion vector can be obtained by adding the reconstructed motion vector difference to the predicted motion vector.

Referring to Figure 9, motion compensation S940 can be performed based on the motion information of the target block.

In addition to the predicted motion vector and motion vector difference mentioned above, motion information can also include reference image information, reference orientation, motion model information, etc. In skip mode and merge mode, some information (e.g., reference image information, reference orientation, etc.) can be implicitly determined, while in competition mode, relevant information can be explicitly processed.

The predicted block can be obtained by performing motion compensation based on the motion information obtained through the above processing.

Referring to Figure 9, decoding S950 can be performed on the residual components of the target block.

The target block can be reconstructed by adding the residual components obtained through the above processing to the predicted block. In this case, explicit or implicit processing can be performed on the information regarding the presence or absence of the residual components, depending on the motion information encoding pattern.

Figure 10 is a layout diagram of the target block and its adjacent blocks according to an embodiment of the present invention.

Referring to Figure 10, blocks <inter_blk_A> adjacent to the target block in the left, top, top left, top right, and bottom left directions, and blocks <inter_blk_B> adjacent to the block corresponding to the target block in the spatial Col_Pic that are different in time in the center, left, right, top, bottom, top left, top right, bottom left, and bottom right directions, can be configured as candidate blocks for predicting the motion information (e.g., motion vectors) of the target block.

In the case of inter_blk_A, the orientation of adjacent blocks can be determined based on the encoding order such as raster scan or z scan, and existing orientations can be removed, or adjacent blocks in the right, bottom, and lower right directions can additionally include candidate blocks.

Referring to Figure 10, Col_Pic can be an adjacent image before or after the current image (e.g., when the interval between images is 1), and the corresponding block in the image can have the same position as the target block.

Alternatively, Col_Pic can be an image in which the interval between images is predefined relative to the current image (e.g., the interval between images is z, where z is an integer of 1, 2, or 3), and the corresponding block can be set to a position moved from the predetermined coordinates (e.g., top left) of the target block by a predetermined disparity vector, and the disparity vector can be set to a predefined value.

Alternatively, Col_Pic can be set based on motion information of adjacent blocks (e.g., a reference image), and disparity vectors can be set based on motion information of adjacent blocks (e.g., motion vectors) to determine the position of the corresponding block.

In this case, k neighboring blocks can be referenced, and k can be an integer of 1, 2, or greater. If k is 2 or greater, Col_Pic and the disparity vector can be obtained based on calculations such as maximum, minimum, median, or weighted average of motion information of neighboring blocks (e.g., reference images or motion vectors). For example, the disparity vector can be set as the motion vector of the left or top block, and can be set as the median or average of the motion vectors of the left and bottom blocks.

The settings for time candidates can be determined based on motion information configuration settings. For example, the position of Col_Pic and the position of the corresponding block can be determined based on whether motion information is configured on a block-by-block or sub-block-by-sub-block basis. For this motion information, motion information configured on a block-by-block or sub-block-by-sub-block basis should be included in the motion information prediction candidate group. For example, when obtaining motion information on a sub-block basis, the block at the position where the predetermined disparity vector has been moved can be set as the position of the corresponding block.

The above examples illustrate the following: information about the location of the block corresponding to Col_Pic is implicitly determined, and related information can also be explicitly generated in units such as sequences, pictures, slices, tile groups, tiles, and blocks.

On the other hand, when partitioning units such as sub-images, slices, and tiles are not limited to a single image (e.g., the current image) and are shared between images, and if the position of the block corresponding to Col_Pic differs from the partitioning unit to which the target block belongs (i.e., if it extends or deviates from the boundary of the partitioning unit to which the target block belongs), it can be determined based on motion information of subsequent candidates according to a predetermined priority. Alternatively, it can be set to the same position in the image as the target block.

The motion information of spatially adjacent blocks and temporally adjacent blocks can be included in the motion information prediction candidate group of the target block, which is called the spatial candidate and the temporal candidate. A_a spatial candidates and b_b temporal candidates can be supported, and a and b can be integers from 1 to 6 or greater. In this case, a can be greater than or equal to b.

Based on the configuration of the predicted candidates according to the previous motion information, a fixed number of spatial and temporal candidates can be supported, or a variable number (e.g., including 0) can be supported.

Additionally, blocks <inter_blk_C> that are not adjacent to the target block can be considered during the candidate group construction.

For example, motion information of blocks located relative to a target block (e.g., top-left coordinates) at predetermined intervals can be the target to be included in the candidate group. The intervals can be p and q (depending on each component of the motion vector), where p and q can be integers of 2 or greater, such as 0, 2, 4, 8, 16, 32, and p and q can be the same or different. That is, assuming the top-left coordinates of the target block are (m, n), motion information of blocks at positions (m ± p, n ± q) can be included in the candidate group.

For example, motion information of blocks that have been encoded before the target block can be used as targets to be included in the candidate group. In this case, based on a predetermined encoding order relative to the target block (e.g., raster scan, Z-scan, etc.), a predetermined number of recently encoded blocks can be considered as candidates, and since encoding is performed using a first-in-first-out method such as FIFO, blocks with older encoding orders can be removed from the candidates.

Because the above example involves a case where motion information of a non-nearest neighbor block that has already been encoded relative to the target block is included in the candidate group, this is called a statistical candidate.

In this case, the reference block used to obtain statistical candidates can be set as the target block, but a higher block of the target block can be set (e.g., if the target block is in coded block <1>/predicted block <2>, the higher block is the maximum coded block <1>/coded block <2>, etc.). k statistical candidates can be supported, and k can be an integer from 1 to 6 or greater.

Motion information obtained by combining spatial, temporal, and statistical candidates can be included in a candidate group. For example, candidates can be obtained by applying a weighted average to each component of multiple (2, 3, or more) motion information previously included in the candidate group, and candidates can be obtained by processing each component of multiple motion information using median, maximum, minimum, etc. This is called a combined candidate, and r combined candidates can be supported, where r can be an integer of 1, 2, 3, or greater.

When the candidates used for combination do not have the same information about the reference image, they can be set as the reference image information for the combined candidates based on the reference image information about one of the candidates, or they can be set to a predefined value.

Based on the configuration of the predicted candidates according to the previous motion information, it can support a fixed number of statistical candidates or combined candidates, or it can support a variable number.

Additionally, default candidates with predetermined values such as (s, t) can be included in the candidate group, and the configuration of candidate prediction based on previous motion information can support a variable number (0, 1, or larger integers). In this case, s and t can be set to include 0, and can be set based on various block sizes (e.g., maximum encoding/prediction/transform block, minimum encoding/prediction/transform block horizontal or vertical length, etc.).

Depending on the motion information encoding mode, motion information prediction candidate groups can be constructed from the candidates, and other additional candidates can be included. Furthermore, the candidate group configuration settings can be the same or different depending on the motion information encoding mode.

For example, skip mode and merge mode can jointly construct candidate groups, while competition mode can construct a single candidate group.

Here, the candidate group configuration settings can be limited by the following: the category and position of the candidate block (e.g., determined from the position of the sub-block used to obtain motion information in the determined direction among the left/top/top-left/top-right/bottom-left directions), the number of candidates (e.g., total number, maximum number of each category), and the candidate construction method (e.g., priority of each category, priority within each category, etc.).

The number (total) of motion information prediction candidate groups can be k, and k can be an integer from 1 to 6 or greater. In this case, when the number of candidates is one, it may mean that no candidate group selection information is generated, and the motion information of the predefined candidate block is set as the predicted motion information. When there are two or more candidate groups, candidate group selection information can be generated.

The candidate block can be one or more of inter_blk_A, inter_blk_B, and inter_blk_C. In this case, inter_blk_A can be the default included category, and the other category can be an additional supported category, but is not limited to this.

The above description can be applied to the candidate construction for motion information prediction in translational motion models, and the same or similar candidate blocks can be used/referenced in the candidate group construction of non-translational motion models (affine models). On the other hand, the candidate group construction of affine models can differ from that of translational motion models because the motion features and the number of motion vectors are different.

For example, when the motion model of a candidate block is an affine model, the configuration of the motion vector set of the candidate block can be included in the candidate group as is. For example, when the top-left and top-right coordinates are used as control points, the motion vector set of the top-left and top-right coordinates of the candidate block can be included in the candidate group as a candidate.

Alternatively, when the motion model of the candidate block is a translational motion model, the combination of motion vectors of the candidate blocks set based on the positions of the control points can be constructed as a set and included in the candidate group. For example, when the upper-left, upper-right, and lower-left coordinates are used as control points, the motion vector of the upper-left control point can be predicted based on the left, upper, and upper-left motion vectors of the target block (e.g., in the case of a translational motion model), and the motion vector of the upper-right control point can be predicted based on the motion vectors of the upper and upper-right blocks of the target block (e.g., in the case of a translational motion model), and the motion vector of the lower-left control point can be predicted based on the left and right motion models of the target block. In the above examples, the case where the motion model of the candidate block set based on the positions of the control points is a translational motion model has been described, but even in the case of an affine model, the motion vectors of the control point positions can be obtained or derived only by acquiring or deriving them. That is, for the motion vectors of the upper-left, upper-right, and lower-left control points of the target block, the motion vectors can be obtained or derived from the upper-left, upper-right, lower-left, and lower-right control points of the candidate block.

In summary, when the motion model of the candidate block is an affine model, the motion vector set of the corresponding block can be included in the candidate group (A), and when the motion model of the candidate block is a translational motion model, the motion vector set obtained by combining the control points of the motion vectors of the predetermined control points of the target block can be included in the candidate group (B).

In this case, candidate groups can be constructed using either method A or method B, or both methods A and B can be used. Furthermore, method A can be used first for construction, and method B can be used subsequently, but this is not a limitation.

Figure 11 shows an example diagram of statistical candidates according to an embodiment of the present invention.

Referring to Figure 11, the blocks corresponding to A to L in Figure 11 refer to the blocks that have been encoded and are separated from the target block by a predetermined interval.

When candidate blocks for motion information of a target block are limited to blocks adjacent to the target block, they may not reflect various characteristics of the image. Therefore, blocks encoded before the current image can also be considered candidate blocks. This category is called inter_blk_B, and this is referred to as statistical candidate.

Since the number of candidate blocks included in the motion information prediction candidate group is limited, effective statistical candidate management may be required.

(1) A block at a predetermined position that is at a predetermined distance from the target block can be regarded as a candidate block.

As an example, by restricting the x-component of a predetermined coordinate relative to the target block (e.g., the top-left coordinate, etc.), blocks with predetermined intervals can be considered as candidate blocks, and examples such as (-4, 0), (-8, 0), (-16, 0), (-32, 0), etc. are feasible.

As an example, by restricting the y-component of a predetermined coordinate relative to the target block, blocks with a predetermined interval can be considered as candidate blocks, and examples such as (0, -4), (0, -8), (0, -16), (0, -32), etc. are feasible.

As an example, a block with predetermined intervals of x and y components (excluding 0) relative to predetermined coordinates of the target block can be considered a candidate block. Examples such as (-4, -4), (-4, -8), (-8, -4), (-16, 16), (16, -16), etc., are feasible. In this example, the x component may depend on the y component having a positive sign.

Candidate blocks that are considered as statistical candidates can be determined in a limited way based on the encoding settings. For an example described later, refer to Figure 11.

Candidate blocks can be categorized into statistical candidates based on whether they belong to a predetermined unit. Here, the predetermined unit can be determined based on the largest coded block, block, tile, slice, sub-image, image, etc.

For example, when selecting candidate blocks by restricting them to the largest coded block that is the same as the target block, A through C can be the target.

Alternatively, when selecting candidate blocks by restricting the blocks belonging to the largest coded block to which the target block belongs and the left largest coded block, A through C, H, and I can be targets.

Alternatively, when selecting candidate blocks by restricting the blocks belonging to the largest coded block to which the target block belongs and the largest coded block above, A through C, E, and F can be targets.

Alternatively, when selecting candidate blocks by restricting them to blocks belonging to the same slice as the target block, A through I can be the target.

Candidate blocks can be classified as statistical candidates based on whether they are located in a predetermined direction relative to the target block. Here, the predetermined direction can be determined from left, top, upper left, upper right, etc.

For example, when selecting candidate blocks by restricting blocks located in the left and top directions relative to the target block, B, C, F, I, and L can be targets.

Alternatively, when selecting candidate blocks by restricting them to blocks located in the left, top, upper left, or upper right directions relative to the target block, A through L can be the target.

As in the example above, even if candidate blocks considered as statistical candidates are selected, there can be a priority order for including them in the motion information prediction candidate group. That is, as many statistical candidates as there are supported can be selected according to priority.

Priority can be determined by a predetermined order or by various coding elements. Coding elements can be qualified based on the distance between the candidate block and the target block (e.g., whether it is a short distance, the distance between blocks can be checked based on the x and y components) and the relative direction of the candidate block to the target block (e.g., left, top, top-left, top-right direction, left->top->top-right->left, etc.).

(2) A block that has been encoded according to a predetermined encoding order relative to the target block can be regarded as a candidate block.

The following examples assume that raster scan (e.g., maximum coding block) and Z scan (e.g., coding block, prediction block, etc.) are followed, but it should be understood that the following descriptions may vary depending on the scan order.

In the case of example (1) above, priority can be supported for constructing candidate groups for motion information prediction, and similarly, predetermined priority can be supported in this embodiment. Priority can be determined by various coding elements, but for ease of description, it is assumed that priority is determined according to the coding order. In the examples described later, candidate blocks and priorities will be considered together.

Candidate blocks that are considered as statistical candidates can be determined in a limited way based on the encoding settings.

Candidate blocks can be categorized into statistical candidates based on whether they belong to a predetermined unit. Here, the predetermined unit can be determined based on the largest coded block, block, tile, slice, sub-image, image, etc.

For example, when a block belonging to the same image as the target block is selected as a candidate block, the candidate block and priority, such as J-D-E-F-G-K-L-H-I-A-B-C, can be the target.

Alternatively, when a block belonging to the same slice as the target block is selected as a candidate block, the candidate block and its priority, such as D-E-F-G-H-I-A-B-C, can be the target.

Candidate blocks can be classified as statistical candidates based on whether they are located in a predetermined direction relative to the target block. Here, the predetermined direction can be determined based on the left or top direction.

For example, when selecting candidate blocks by restricting blocks located to the left of the target block, the candidate blocks and priorities such as K-L-H-I-A-B-C can be the target.

Alternatively, when selecting candidate blocks by restricting blocks located in the direction above the target block, the candidate block and priority, such as E-F-A-B-C, can be the target.

The above description is based on the following assumptions: candidate blocks and priorities are combined, but not limited to this, and various candidate blocks and priorities can be set.

As mentioned above, the priorities included in the candidate group for motion information prediction can be supported by the descriptions in (1) and (2) above. One or more priorities can be supported, and a single priority can be supported for the entire candidate block. Alternatively, candidate blocks can be classified into two or more categories, and priorities based on the categories can be supported. In the former case, an example could be selecting k candidate blocks based on a single priority, while in the latter case, an example could be selecting p and q (p+q=k) candidate blocks based on each priority (e.g., two categories). In this case, categories can be classified based on predetermined encoding elements, and these encoding elements can include whether the block belongs to a predetermined unit, whether the candidate block is located in a predetermined direction relative to the target block, etc.

An example of selecting candidate blocks based on whether a candidate block belongs to a predetermined unit (e.g., a partitioning unit) described by (1) and (2) has been described. In addition to partitioning units, candidate blocks can be selected by restricting blocks within a predetermined range relative to the target block. For example, it can be identified by boundary points that define the ranges of (x1, y1), (x2, y2), (x3, y3), etc., and by the minimum and maximum values of the x or y components. Values such as x1 to y3 can be integers such as 0, 4, and 8 (based on absolute values).

Statistical candidates can be supported based on either (1) or (2) above, or they can be supported in a combination of (1) and (2). The above description has described how to select candidate blocks for statistical candidates, and the management and updating of statistical candidates and their inclusion in the motion information prediction candidate group will be described later.

The construction of motion information prediction candidate groups can usually be performed on a block-by-block basis, because the motion information of blocks adjacent to the target block may be the same or similar. That is, spatial or temporal candidates can be constructed based on the target block.

Furthermore, statistical candidates can be constructed based on a predetermined unit, since the candidate block is not adjacent to the target block. The predetermined unit can be a higher block that includes the target block.

For example, when the target block is a prediction block, statistical candidates can be constructed on a prediction block basis or on a coding block basis.

Alternatively, when the target block is a coded block, statistical candidates can be constructed on a coded block basis, or statistical candidates can be constructed on a predecessor block basis (e.g., depth information is at least one difference from the target block). In this case, the predecessor block can include the largest coded block, or it can be a unit obtained based on the largest coded block (e.g., an integer multiple of the largest coded block).

As in the examples above, statistical candidates can be constructed based on the target block or higher units, and in the examples described later, it is assumed that statistical candidates are constructed based on the target block.

Furthermore, memory initialization for statistical candidates can be performed based on predetermined units. The predetermined unit can be determined based on an image, sub-image, slice, tile, block, or chunk, and in the case of a chunk, it can be set based on the largest coded block. For example, memory initialization for statistical candidates can be performed based on an integer (1, 2, or greater) of the largest coded block, the row or column units of the largest coded block, etc.

The management and updating of statistical candidates are described below. A memory can be prepared for managing statistical candidates, and it can store motion information for up to k blocks. In this case, k can be an integer from 1 to 6 or larger. The motion information stored in the memory can be determined based on the motion vectors, reference images, and reference directions. Here, the number of motion vectors (e.g., 1 to 3, etc.) can be determined based on motion model selection information. For ease of explanation, the case where a block that has completed its encoding according to a predetermined encoding order based on the target block is considered a candidate block will be described later.

(Case 1) Motion information of blocks pre-coded according to the coding order can be included as candidates in the previous ranking. Additionally, when the maximum number is filled and updated by other candidates, candidates in the previous ranking can be removed, and the order can be shifted forward one by one. In the following example, x represents a blank space that has not yet been constructed.

(Example)

1 order - [a,x,x,x,x,x,x,x,x,x]

2 order - [a,b,x,x,x,x,x,x,x,x]

...

9 order - [a, b, c, d, e, f, g, h, i, x]

10. Sequence: [a, b, c, d, e, f, g, h, i, j] -> Add j. (Full)

11. Sequence - [b, c, d, e, f, g, h, i, j, k] -> Add k. Remove leading a. Shift sequence.

(Case 2) When duplicate motion information exists, remove pre-existing candidates and adjust the order of existing candidates for forwarding.

In this context, duplication implies identical motion information, which can be defined by the motion information encoding mode. As an example, in the merge mode, duplication can be determined based on motion vectors, a reference image, and a reference direction, while in the competition mode, duplication can be determined based on motion vectors and a reference image. In the competition mode, if the reference images for each candidate are identical, duplication can be determined by comparing motion vectors; and if the reference images for each candidate are different, duplication can be determined by comparing motion vectors scaled based on the reference images for each candidate, but this is not the only possibility.

(Example)

1. Sequence - [a, b, c, d, e, f, g, x, x, x]

2. The sequence - [a, b, c, d, e, f, g, d, x, x] -> d overlaps.

-[a, b, c, e, f, g, d, x, x, x] -> Remove the existing d, shift order

(Case 3) When there is duplicate motion information, the corresponding candidate can be marked and managed as a long-term candidate (long-term candidate) according to predetermined conditions.

A separate memory can be supported for long-term candidates, and additional information such as occurrence frequency, in addition to motion information, can be stored and updated. In the example below, the memory for long-term candidate storage can be represented as ().

(Example)

1. Sequence - [(), b, c, d, e, f, g, h, i, j, k]

2. Sequence - [(), b, c, d, e, f, g, h, i, j, k] -> e is the next sequence. Overlapping - [(e), b, c, d, f, g, h, i, j, k] -> Move to the front. Long-term marker.

3. Sequence - [(e), c, d, f, g, h, i, j, k, l] -> Add l. Remove b.

...

10. The sequence - [(e), l, m, n, o, p, q, r, s, t] -> l is the next sequence. Overlapping.

-[(e, l), m, n, o, p, q, r, s, t]->long-term marker

11. Order - [(e, l), m, n, o, p, q, r, s, t] -> l is the next order. 3. Overlap - [(l, e), m, n, o, p, q, r, s, t] -> Changes the long-term candidate order.

The above example illustrates the following situation: existing statistical candidates (short-term candidates) are managed in an integrated manner, but can be managed separately from short-term candidates, and the number of long-term candidates can be 0, 1, 2 or a larger integer.

Short-term candidates can be managed in a first-in, first-out (FIFO) manner based on encoding order, but long-term candidates can be managed based on frequency, considering the redundancy among short-term candidates. In the case of long-term candidates, the order of candidates can be changed according to frequency during memory updates, but this is not a limitation.

The above example could be a case where candidate blocks are classified into two or more categories and priority is supported based on that category. When constructing motion information prediction candidate groups to include statistical candidates, short-term and long-term candidates a and b can be constructed separately based on priority within each candidate. Here, a and b can be integers 0, 1, 2, or larger.

Cases 1 to 3 above are examples based on statistical candidates, and the configuration of (case 1 + case 2) or (case 1 + case 3) is feasible, and statistical candidates can be managed in various modified and added forms.

Figure 12 is a conceptual diagram of statistical candidates based on a non-translational motion model according to an embodiment of the present invention.

Statistical candidates can be used not only for translational motion models but also for supporting non-translational motion models. In the case of translational motion models, due to the high frequency of occurrence, there may be many blocks that can be referenced from spatial or temporal candidates, etc. However, in the case of non-translational motion models, because of the low frequency of occurrence, statistical candidates in addition to spatial or temporal candidates may be needed.

Referring to Figure 12, the motion information { _TV0 , _TV1 } of the target block is predicted based on the motion information { _PV0 , _PV1 } of the block <inter_blk_B> that belongs to the same space as the target block and is far from the target block (hmvp).

Compared to the statistical candidates for the translational motion model described above, the number of motion vectors in the motion information stored in memory can be increased. Therefore, the basic concept of statistical candidates can be the same as or similar to that of the translational motion model, and can be explained by different configurations of the number of motion vectors. The following will describe the differences based on the non-translational motion model. In the example described later, it is assumed that the affine model has three control points (e.g., v0, v1, and v2).

In the case of an affine model, the motion information stored in memory can be determined based on a reference image, a reference direction, and motion vectors v0, v1, and v2. Here, the stored motion vectors can be stored in various forms. For example, the motion vectors can be stored as is, or predetermined difference information can be stored.

Figure 13 is an example diagram of the construction of motion information for storing the position of each control point as a statistical candidate according to an embodiment of the present invention.

Referring to Figure 13, (a) the motion vectors v0, v1 and v2 of the candidate block are represented as v0, v1 and v2, while in Figure 13, (b) the motion vector v0 of the candidate block is represented as v0, and the motion vectors v1 and v2 are represented as v*1 and v*2, which is different from the motion vector v0.

In other words, the motion vector of each control point location can be stored as is, or the difference between the motion vector of another control point location and the original motion vector can be stored. This can be an example of a configuration considered in terms of memory management, and various modification examples are possible.

Whether statistical candidates are supported can be explicitly generated in units such as sequences, images, slices, tiles, and blocks, or implicitly determined based on encoding settings. Since encoding settings can be limited by the various encoding elements described above, a detailed description is omitted.

Here, the support of a statistical candidate can be determined based on the motion information encoding pattern. Alternatively, it can be determined based on the motion model selection information. For example, statistical candidates can be supported among merge_inter, merge_ibc, merge_affine, comp_inter, comp_ibc, and comp_affine.

If statistical candidates are supported for both translational and non-translational motion models, then memory for multiple statistical candidates can be supported.

Next, we will describe a method for constructing motion information prediction candidate groups based on motion information encoding patterns.

The candidate set for motion information prediction in the competitive mode (hereinafter referred to as the competitive mode candidate set) may include k candidates, and k may be an integer of 2, 3, 4 or greater. The competitive mode candidate set may include at least one of spatial candidates or temporal candidates.

Spatial candidates can be derived from at least one of the blocks adjacent to the reference block in the left, top, top-left, top-right, and bottom-left directions. Alternatively, at least one candidate can be derived from the left-adjacent block (left block, bottom-left block) and the top-adjacent block (top-left block, top block, top-right block), under the assumption of this setting, which will be described later.

There can be two or more priorities for constructing candidate groups. In the region adjacent to the left, the priority order can be set to bottom left-left, and in the region adjacent to the top, the priority order can be set to top right-top-top left.

In the examples above, spatial candidates can be derived solely from blocks whose reference images are identical to the target block's. Furthermore, spatial candidates can be derived by scaling the target block's reference image (hereinafter marked *). In this case, within regions adjacent to the left, the priority order can be set to left-bottom-left*-bottom-left* or left-bottom-left*-left*, and within regions adjacent to the top, the priority order can be set to top-right-top-top-left-top-right*-top*-top-left* or top-right-top-left-top-left*-top*-top*-top-right*.

Time candidates can be derived from at least one of the adjacent blocks such as center, left, right, top, bottom, top left, top right, bottom left, and bottom right, based on the block corresponding to the reference block. Priorities can be configured for candidate groups, and priorities such as center-bottom left-right-bottom-bottom left-center-top left can be set.

When the sum of the maximum allowed number of spatial candidates and the maximum allowed number of temporal candidates is less than the number of candidates in the competitive mode, temporal candidates can be included in the candidate group, regardless of the construction of the candidate group for spatial candidates.

Candidates can be included in a candidate group based on priority, availability of each candidate block, and the maximum number of time-limited candidates (the number of q, an integer between 1 and the number of contested mode candidates).

Here, when the maximum allowed number of spatial candidates is set equal to the number of merge pattern candidates, temporal candidates may not be included in the candidate group, and when the maximum allowed number of spatial candidates is not filled, temporal candidates may be included in the candidate group. This example assumes the latter case.

Here, the motion vector of the temporal candidate can be obtained based on the motion vector of the candidate block and the distance interval between the current image and the reference image of the target block, and the reference image of the temporal candidate can be obtained based on the distance interval between the current image and the reference image of the target block, or the reference image of the temporal candidate can be obtained as a predefined reference image (e.g., the reference image index is 0).

If the candidate group for the competitive mode is not filled with spatial candidates, temporal candidates, etc., the candidate group can be constructed by including the default candidate with the zero vector.

The competition mode focuses on the description of comp_inter, and in the case of comp_ibc or comp_affine, can construct candidate groups for similar or different candidates.

For example, in the case of comp_ibc, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. In this case, candidate groups can be constructed by prioritizing spatial candidates, and then candidate groups can be constructed in the order of statistical candidates - combined candidates - default candidates, etc., but the order is not limited to this.

Alternatively, in the case of comp_affine, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. Specifically, a set of motion vectors for a candidate (e.g., a block) can be constructed in candidate group <1>, or a candidate group <2> can be constructed where the motion vectors of candidates (e.g., two or more blocks) are combined based on control point locations. It is possible, but not limited to, to first include the candidates for <1> in the candidate group and then include the candidates for <2> in the candidate group.

Since a detailed description of the above construction of various competitive mode candidate groups can be obtained from comp_inter, a detailed description is omitted.

In the process of constructing a candidate group for a competitive mode, if there is duplicate motion information among the candidates initially included, the candidates initially included can be maintained, and candidates with the next priority can be included in the candidate group.

Here, before constructing candidate groups, the motion vectors of the candidate groups can be scaled based on the distance between the reference image of the target block and the current image. For example, when the distance between the reference image of the target block and the current image and the distance between the reference image of the candidate block and the image to which the candidate block belongs are the same, the motion vector can be included in the candidate group; otherwise, the motion vector scaled based on the distance between the reference image of the target block and the current image can be included in the candidate group.

In this context, redundancy can imply identical motion information, which can be defined by the motion information encoding pattern. In a contention-based mode, duplication can be determined based on motion vectors, a reference image, and a reference direction. For example, if at least one component of a motion vector is different, it can be determined that no duplication exists. Redundancy checks are typically performed when a new candidate is included in the candidate group, but this can be modified to omit the redundancy check.

The motion information prediction candidate set for the merging pattern (hereinafter referred to as the merging pattern candidate set) may include k candidates, and k may be an integer of 2, 3, 4, 5, 6 or greater. The merging pattern candidate set may include at least one of spatial candidates or temporal candidates.

Spatial candidates can be derived from at least one of the blocks adjacent to the reference block in the left, top, top-left, top-right, and bottom-left directions. Priorities can exist for constructing candidate groups, and priorities can be set such as left-top-bottom-top-right-top-left, left-top-top-right-bottom-left-top, and top-left-bottom-left-top-right.

Candidates can be included in a candidate group based on priority, availability of each candidate block (e.g., determined based on encoding mode, block position, etc.), and the maximum allowed number of spatial candidates (the number of p, an integer between 1 and the number of candidate groups for the merge mode). Candidates may not be included in the candidate group in the order tl-tr-bl-t3-13, depending on the maximum allowed number and availability. If the maximum allowed number is 4 and the availability of all candidate blocks is true, the motion information of tl may not be included in the candidate group; conversely, if the availability of some candidate blocks is false, the motion information of tl may be included in the candidate group.

Time candidates can be derived from at least one of the blocks adjacent to the center, left, right, top, bottom, top left, top right, bottom left, and bottom right, based on the block corresponding to the reference block. Priorities can exist for constructing candidate groups, and priorities such as center-bottom left-right-bottom, bottom left-center-top left, etc., can be set.

Candidates can be included in a candidate group based on priority, availability of each candidate block, and the maximum number of time candidates allowed (q, an integer between 1 and the number of merge pattern candidates).

Here, the motion vector of the temporal candidate can be obtained based on the motion vector of the candidate block, and the reference image of the temporal candidate can be obtained based on the reference image of the candidate block, or a predefined reference image can be obtained (e.g., reference image index 0).

For priorities included in the merge pattern candidate group, spatial candidate-temporal candidate can be set, or vice versa, and priorities that combine spatial and temporal candidates can be supported. In this example, spatial candidate-temporal candidate is assumed.

In addition, statistical candidates or combined candidates can also be included in the merged pattern candidate group. Statistical candidates and combined candidates can be constructed after spatial and temporal candidates, but are not limited to this, and various priorities can be set.

For statistical candidates, up to n motion information segments can be managed, and among these, z motion information segments can be included in the merged pattern candidate group as statistical candidates. z can vary depending on the candidate construction already included in the merged pattern candidate group, and can be an integer of 0, 1, 2 or greater, and can be less than or equal to n.

A combined candidate can be derived by merging n candidates already included in the combined pattern candidate group, where n can be an integer of 2, 3, 4, or greater. The number of combined candidates (n) can be information explicitly generated in units such as sequences, pictures, sub-pictures, slices, tiles, blocks, or chunks. Alternatively, it can be implicitly determined based on encoding settings. In this case, the encoding settings can be limited based on one or more factors such as the size, shape, position, image type, and color components of the reference block.

Furthermore, the number of combined candidates can be determined based on the number of unfilled candidates in the merged candidate group. In this case, the number of unfilled candidates in the merged candidate group can be the difference between the number of merged candidate groups and the number of already filled candidates. That is, if the merged candidate group construction is complete, combined candidates may not be added. If the merged candidate group construction is incomplete, combined candidates may be added, but no combined candidates are added when the number of filled candidates in the merged candidate group is less than or equal to one.

If the candidate group for the merge pattern is not filled with spatial candidates, temporal candidates, statistical candidates, or combined candidates, the candidate group can be constructed by including the default candidate with a zero vector.

The merge mode focuses on the description of merge_inter, and in the case of merge_ibc or merge_affine, it can construct candidate groups for similar or different candidates.

For example, in the case of merge_ibc, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. In this case, candidate groups can be constructed by prioritizing spatial candidates, and then candidate groups can be constructed in the order of statistical candidates - combined candidates - default candidates, etc., but the order is not limited to this.

Alternatively, in the case of merge_affine, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, temporal candidates, statistical candidates, combined candidates, and default candidates. Specifically, a set of motion vectors for a candidate (e.g., a block) can be constructed in candidate group <1>, or a candidate group <2> can be constructed where the motion vectors of candidates (e.g., two or more blocks) are combined based on control point locations. It is possible, but not limited to, to first include the candidates for <1> in the candidate group and then include the candidates for <2>.

Since a detailed description of the above construction of various merge pattern candidate groups can be derived from merge_inter, a detailed description will be omitted.

In the process of constructing a candidate group for merging patterns, if there is overlapping motion information among the first included candidates, the first included candidates can be maintained, and candidates with the next priority can be included in the candidate group.

In this context, redundancy implies identical motion information, which can be defined by the motion information encoding pattern. In the case of the merging pattern, overlap can be determined based on motion vectors, a reference image, and a reference direction. For example, if at least one motion vector component is different, it can be determined that no overlap exists. Redundancy checks are typically performed when new candidates are included in the candidate group, but this can be modified to omit them.

Figure 14 is a flowchart illustrating motion information encoding according to an embodiment of the present invention.

A candidate list of motion vector predictions for the target block can be generated S1400. The aforementioned competition mode candidate group may mean a candidate list of motion vectors, and its detailed description will be omitted.

The motion vector difference S1410 of the target block can be reconstructed. The difference between the x and y components of the motion vector can be reconstructed separately, and the difference of each component can have a value of 0 or greater.

The prediction candidate index for the target block can be selected from the motion vector prediction candidate list S1420. The predicted motion vector value for the target block can be derived based on the motion vector obtained from the candidate index in the candidate list. If a predetermined motion vector prediction value can be obtained, the process of selecting the prediction candidate index and index information can be omitted.

The accuracy information S1430 for the motion vector difference can be obtained. This can be the accuracy information typically applied to the x and y components of the motion vector, or the accuracy information applied to each component. If the motion vector difference is 0, the accuracy information can be omitted, and this processing can also be omitted.

An adjustment offset S1440 for the motion vector prediction value can be derived. The offset can be a value added to or subtracted from the x or y component of the motion vector prediction value. The offset can support only one of the x and y components, or it can support both x and y components.

Assuming the predicted motion vector value is (pmv_x, pmv_y), and the offsets are adjusted to offset_x and offset_y, then the predicted motion vector value can be adjusted (or obtained) to (pmv_x + offset_x, pmv_y + offset_y).

Here, the absolute values of offset_x and offset_y can be integers of 0, 1, 2, or larger, and can have values (+1, -1, +2, -2, etc.), where sign information is considered together. Furthermore, offset_x and offset_y can be determined based on a predetermined precision. The predetermined precision can be determined in units of 1/16, 1/8, 1/4, 1/2, and 1 pixel, and can be based on interpolation precision, motion vector precision, etc.

For example, if the interpolation precision is in units of 1/4 pixel, then the interpolation precision can be 0, 1/4, -1/4, 2/4, -2/4, 3/4, or -3/4, which are combinations of absolute value and sign information.

Here, offset_x and offset_y can be backed by a and b respectively, and a and b can be integers of 0, 1, 2 or greater. a and b can have fixed values or variable values. Furthermore, a and b can have the same or different values.

Whether adjustment offsets for motion vector prediction values are supported can be explicitly supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., or implicitly determined based on encoding settings. Additionally, the settings for adjustment offsets (e.g., the range and number of values) can be determined based on encoding settings.

The encoding settings can be determined by taking into account at least one of the encoding elements such as image type, color components, target block state information, motion model selection information (e.g., whether it is a translational motion model), reference image (e.g., whether it is the current image), and motion vector precision difference selection information (e.g., whether it is a predetermined unit among 1/4, 1/2, 1, 2, and 4 units).

For example, the size of the block can be used to determine whether offset and offset settings are supported. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H, which have the width (W) and height (H) of the block. In the case of the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. In the case of the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by one of the first threshold size or the second threshold size, or both.

In this case, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this scenario, the first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and the second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

Furthermore, the adjustment offset can be applied to all candidates included in the motion information prediction candidate group, or it can be applied to only some candidates. In the example described later, it is assumed that all candidates included in the candidate group can be applied, but the candidates to which the adjustment offset is applied can be selected from 0, 1, 2 to the maximum number of candidates.

If offset adjustment is not supported, this processing and offset adjustment information can be omitted.

The motion vector of the target block can be reconstructed by adding the predicted motion vector value and the motion vector difference in S1450. In this case, the motion vector accuracy can be prioritized to unify the processing of the predicted motion vector value or the motion vector difference, and the aforementioned motion vector scaling process can be performed first or during this process.

The configuration and order are not limited to this and can be changed in various ways.

Figure 15 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention.

For ease of explanation, two motion vector prediction candidates are supported, and it is assumed that only one component (x or y component) is compared. Furthermore, it is assumed that the interpolation precision or motion vector precision is 1/4 pixel unit. Additionally, it is assumed that 1/4, 1, and 4 pixel units are supported for the motion vector difference precision (e.g., if 1/4, it is assumed to be binaryized as <0>, if 1, it is assumed to be binaryized as <10>, and if 4, it is assumed to be binaryized as <11>). Furthermore, it is assumed that sign information is omitted for the motion vector difference, and the motion vector difference is processed through univariate binarization (e.g., 0: <0>, 1: <10>, 2: <110>, etc.).

Referring to Figure 15, the value of the actual motion vector (X) is 2, the value of candidate 1 (A) is 0, and the value of candidate 2 (B) is 1/4.

(In cases where differential motion vector precision is not supported)

Since the distance (da) between A and X is 8/4 (9 bits) and the distance (db) between B and X is 7/4 (8 bits), B can be selected as the prediction candidate in terms of bit generation.

(When motion vector difference accuracy is supported)

Since da is 8/4, it can generate 1 pixel unit precision (2 bits) and 2/1 difference information (3 bits), thus generating a total of 5 bits. On the other hand, since db is 7/4, it can generate 1/4 pixel unit precision (1 bit) and 7/4 difference information (8 bits), thus generating a total of 9 bits. In terms of bit generation, the prediction candidate can be chosen as A.

As in the example above, if motion vector difference accuracy is not supported, it may be advantageous to select candidates with short distance intervals from the motion vectors of the target block as prediction candidates. If it is supported, then in addition to the distance interval from the motion vectors of the target block, the amount of information generated based on the accuracy information may be important in selecting prediction candidates.

Figure 16 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention. In the following, it is assumed that motion vector difference accuracy is supported.

Since da is 33/4, it can generate 1/4 pixel unit precision (1 bit) and 33/4 difference information (34 bits), thus generating a total of 35 bits. On the other hand, since db is 21/4, it can generate 1/4 pixel unit precision (1 bit) and 21/4 difference information (22 bits), thus generating a total of 23 bits. Regarding bit generation, prediction candidate B can be selected.

Figure 17 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention. In the following, it is assumed that motion vector difference accuracy is supported and that offset information is adjusted.

In this example, it is assumed that the adjustment offset has 0 and +1 candidates, and a flag (1 bit) indicating whether the adjustment offset has been applied and offset selection information (1 bit) are generated.

Referring to Figure 17, A1 and B1 can be motion vector predictions obtained based on the prediction candidate indices, while A2 and B2 can be new motion vector predictions obtained by modifying A1 and B1 with adjusted offsets. Assume that the distances between A1, A2, B1, B2 and X are da1, da2, db1, and db2, respectively.

(1) Since da1 is 33/4, it can generate 1/4 pixel unit precision (1 bit), offset application flag (1 bit), offset selection information (1 bit) and 33/4 difference information (34 bits), so a total of 37 bits can be generated.

(2) Since da2 is 32/4, it can generate 4-pixel unit precision (2 bits), offset application flag (1 bit), offset selection information (1 bit) and 2/1 difference information (3 bits), so a total of 7 bits can be generated.

(3) Since db1 is 21/4, it can generate 1/4 pixel unit precision (1 bit), offset application flag (1 bit), offset selection information (1 bit) and 21/4 difference information (22 bits), so a total of 25 bits can be generated.

(4) Since db2 is 20/4, it can generate 1 pixel unit precision (2 bits), offset application flag (1 bit), offset selection information (1 bit) and 5/1 difference information (6 bits), so a total of 10 bits can be generated.

For in-situ quantity generation, the prediction candidate can be selected as A, and the offset selection information can be selected as index 1 (+1 in this example).

In the above example, when deriving the motion vector difference based on the existing motion vector prediction values, there may be a situation where many bits are generated due to the small difference between the vector values. This problem can be solved by adjusting the motion vector prediction values.

Predefined flags can be supported to apply adjustment offsets to motion vector predictions. Predefined flags can be configured via offset application flags, offset selection information, etc.

(If only offset application flags are supported)

When the offset application flag is 0, the offset is not applied to the motion vector prediction value, and when the offset application flag is 1, the predetermined offset can be added to or subtracted from the motion vector prediction value.

(If only offset selection information is supported)

The offset set based on offset selection information can be added to or subtracted from the motion vector prediction value.

(If offset application flags and offset selection information are supported)

When the offset application flag is 0, the offset is not applied to the motion vector prediction value, and when the offset application flag is 1, the offset set based on the offset selection information can be added to or subtracted from the motion vector prediction value. This setting is assumed in the example described later.

Meanwhile, offset application flags and offset selection information can be used as information that is unnecessarily generated in some cases. That is, even if the offset is not applied, the offset-related information may be quite inefficient if the information is already zero or the amount of information is reduced based on the maximum precision information. Therefore, it is necessary to support settings that implicitly generate information according to predetermined conditions, rather than always explicitly generating offset-related information.

Next, we assume a motion-related coding sequence (motion vector difference reconstruction -> motion vector difference precision acquisition). In this example, we assume that motion vector difference precision is supported when at least one of the x and y components is not zero.

When the motion vector difference is not 0, one of the following pixel units can be selected: {1/4, 1/2, 1, 4} for the motion vector difference precision.

If the selected information belongs to a predetermined category, information about the adjustment offset can be implicitly omitted; otherwise, information about the adjustment offset can be explicitly generated.

Here, the category can include one of the differential motion vector precision candidates, which can be configured with various categories such as {1/4}, {1/4, 1/2}, {1/4, 1/2, 1}. The minimum precision can also be included in the category.

For example, if the motion vector difference precision is 1/4 pixel (e.g., minimum precision), the offset application flag is implicitly set to 0 (i.e., not applied), and if the motion vector difference precision is not 1/4 pixel, the offset application flag is explicitly generated, and if the offset application flag is 1 (i.e., offset), offset selection information can be generated.

Alternatively, when the motion vector difference precision is 4 pixels (e.g., maximum precision), an offset application flag can be explicitly generated, and when the offset application flag is 1, offset selection information can be generated. Alternatively, when the motion vector difference precision is not 4 pixels, the offset information can be implicitly set to 0.

The above examples assume that offset-related information is implicitly omitted when the motion vector difference indicates minimum precision, and offset-related information is explicitly generated when the motion vector difference indicates maximum precision, but are not limited to this.

Figure 18 is an example diagram illustrating the arrangement of multiple motion vector prediction values according to an embodiment of the present invention.

The above example has described the part about redundancy checks when constructing candidate groups for motion information prediction. Here, redundancy means that the motion information is the same, and as described above, if at least one component of the motion vector is different, it can be determined that there is no redundancy.

Through redundancy checks, multiple candidates for motion vector predictions can be made to not overlap. However, when multiple candidate component elements are very similar (i.e., each candidate's x or y component exists within a predetermined range, and the width or height of that predetermined range is an integer of 1, 2, or greater; or the range can be set based on offset information), motion vector predictions and offset-modified motion vector predictions can overlap. Various settings are feasible for this purpose.

As an example (C1), in the step of constructing a candidate group for motion information prediction, a new candidate can be included in the candidate group if it does not overlap with already included candidates. That is, if there is no overlap only by comparing with the same candidate as previously described, it can be a configuration that can be included in the candidate group.

As an example (C2), a new candidate can be included in a candidate group if it does not overlap with a predetermined number of candidates (group_A) obtained by adding an offset to it, and with a predetermined number of overlaps between already included candidates and candidates (group_B) obtained by adding an offset to it. The predetermined number can be an integer of 0, 1, or greater. If the predetermined number is 0, even if one overlaps, the corresponding new candidate may not be included in the candidate group. Alternatively, (C3), a predetermined offset (a concept distinct from adjusting the offset) can be added to the new candidate to be included in the candidate group, and this offset can be a value that allows group_A to be configured not to overlap with group_B.

Referring to Figure 18, it can be the case that multiple motion vectors belonging to categories A, B, C, and D (in this example, AX, BX, CX, and DX. X is 1 and 2. For example, A1 is a candidate included in the candidate group preceding A2) satisfy the non-overlapping condition (if any component of the candidate motion vector is different).

In this example, it is assumed that -1, 0, and 1 are supported for the x and y components, respectively, and the offset-modified motion vector predictions can be represented as * in the figure. Furthermore, dashed lines (rectangles) represent ranges (e.g., group_A, group_B, etc.) that can be obtained by adding an offset to a predetermined motion vector prediction.

In the case of category A, it can correspond to the case where A1 and A2 do not overlap and group_A1 and group_A2 do not overlap.

In the case of categories B, C, and D, it can correspond to the situation where B1/C1/D1 and B2/C2/D2 do not overlap, but group_B1/C1/D1 and group_B2/C2/D2 partially overlap.

Here, in the case of category B, it could be an example of constructing candidate groups without taking any special actions (C1).

Here, in the case of category C, the following configuration can be used: In this configuration, C2 is determined to be redundant in the redundancy check step, and C3, which has the next higher priority, is included in the candidate group (C2).

Here, in the case of category D, it can be the following: D2 is determined to be redundant in the redundancy check step, and D2 is modified so that group_D2 does not overlap with group_D1 (i.e., D3. D3 is not a motion vector present in the candidate group construction priority) (C3).

Among various categories, it can be applied to the setting of constructing candidate groups of prediction patterns, or various methods other than those mentioned above can be applied.

Figure 19 is a flowchart illustrating motion information encoding in merging mode according to an embodiment of the present invention.

A candidate list S1900 for predicting motion information of the target block can be generated. The above-mentioned candidate group for merging modes may refer to the candidate list for predicting motion information, and its detailed description will be omitted.

A prediction candidate index S1910 for the target block can be selected from the motion information prediction candidate list. The motion vector prediction value of the target block can be derived based on the motion information obtained from the candidate indices in the candidate list. If a predetermined motion information prediction value can be obtained, the process of selecting the prediction candidate index and index information can be omitted.

An adjustment offset S1920 for the motion vector prediction value can be derived. The offset can be a value added to or subtracted from the x or y component of the motion vector prediction value. The offset can support only one of the x or y components, or it can support both x and y components.

Since the adjustment offset in this process can be the same as or similar to the adjustment offsets above, a detailed description is omitted, and a portion of the differences will be described later.

Assuming the predicted motion vector is (pmv_x, pmv_y), and the offsets are adjusted to offset_x and offset_y, the predicted motion vector can be adjusted (or obtained) to (pmv_x + offset_x, pmv_y + offset_y).

Here, the absolute values of offset_x and offset_y can be integers such as 0, 1, 2, 4, 8, 16, 32, 64, 128, etc., and can have sign information considered together. Furthermore, offset_x and offset_y can be determined based on a predetermined precision. The predetermined precision can be determined in units of 1/16, 1/8, 1/4, 1/2, or 1 pixel.

For example, if the motion vector precision is 1/4 pixel unit, then the motion vector precision can be 0, 1/4, -1/4, 1/2, -1/2, 1, -1, 2, -2, etc., which combine absolute value and sign information.

Here, offset_x and offset_y can be resources a and b, respectively, and a and b can be integers such as 0, 1, 2, 4, 8, 16, 32, etc. a and b can have fixed values or variable values. Furthermore, a and b can have the same or different values.

Whether adjustment offsets for motion vector prediction values are supported can be explicitly supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., or implicitly determined based on encoding settings. Additionally, the settings for adjustment offsets (e.g., the range and number of values) can be determined based on encoding settings.

The encoding settings can be determined by taking into account at least one of the encoding elements such as image type, color components, target block state information, motion model selection information (e.g., whether it is a translational motion model), and reference image (e.g., whether it is the current image).

For example, the size of the block can be used to determine whether offset and offset settings are supported. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H, which have the width (W) and height (H) of the block. In the case of the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. In the case of the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by one of the first threshold size or the second threshold size, or both.

In this case, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this scenario, the first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and the second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

Furthermore, the adjustment offset can be applied to all candidates included in the motion information prediction candidate group, or it can be applied to only some candidates. In the example described later, it is assumed that all candidates included in the candidate group can be applied, but the candidates to which the adjustment offset is applied can be selected from a number between 0, 1, or 2 and the maximum number of candidates.

Predefined flags can be used to apply adjustment offsets to motion vector prediction values. These flags can be configured via offset application flags, absolute offset value information, offset sign information, and more.

If offset adjustment is not supported, this processing and offset adjustment information can be omitted.

The motion vector S1930 of the target block can be reconstructed using the predicted motion vector values. Motion information other than the motion vector (e.g., reference image, reference direction, etc.) can be obtained based on the predicted candidate index.

The configuration and order are not limited to this and can be changed in various ways. The background description for supporting offset adjustment in merge mode has been described above with various examples of competition mode, and therefore its detailed description will be omitted.

The methods of this disclosure can be implemented as program instructions executable by various computer devices and stored in a computer-readable medium. The computer-readable medium may include program instructions, data files, and data structures, individually or in combination. The program instructions recorded on the computer-readable medium may be specifically designed for this disclosure or are known to and therefore usable by one of skill in the art of computer software.

The computer-readable medium may include hardware devices particularly suitable for storing and executing program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, etc. The program instructions may include machine language code generated by a compiler or high-level language code that can be executed by an interpreter in a computer. The aforementioned hardware devices may be configured to operate as one or more software modules to perform operations according to this disclosure, and vice versa.

Furthermore, the above-described methods or devices can be implemented by combining or separating all or part of their configurations or functions.

Although the present disclosure has been described above with reference to preferred embodiments thereof, those skilled in the art will understand that various modifications and variations may be made to the present disclosure without departing from the scope and spirit of the present disclosure.

Industrial applicability

This invention can be used to encode/decode video signals.

This application is a divisional application of the invention patent application filed on September 24, 2019, with application number 201980062809.8 and title "Image Encoding/Decoding Method and Apparatus".

Technical Field

This invention relates to image encoding/decoding methods and devices.

Background Technology

With the widespread use of the Internet and portable terminals, and the development of information and communication technologies, multimedia data is being used more and more. Therefore, in order to provide various services or perform various tasks through image prediction in various systems, there is an urgent need to improve the performance and efficiency of image processing systems. However, research and development achievements have not kept pace with this trend.

Therefore, existing methods and devices for encoding/decoding images need performance improvements in image processing, particularly in image encoding or image decoding.

Summary of the Invention

Technical issues

The purpose of this invention to solve the above problems is to provide an image encoding/decoding device that uses an adjustment offset to modify the motion vector prediction value.

Technical solutions

According to an embodiment of the present invention, a method for decoding an image to achieve the above objectives includes: constructing a motion information prediction candidate list for a target block, selecting a prediction candidate index, obtaining a predicted motion vector adjustment offset, and reconstructing the motion information of the target block.

Here, constructing a candidate list for motion information prediction may also include: when the already included candidates, candidates obtained based on offset information, new candidates, and candidates obtained based on offset information do not overlap, the new candidate is included in the candidate group.

Here, deriving the predicted motion vector adjustment offset may also include: deriving the predicted motion vector adjustment offset based on the offset application flag and/or offset selection information.

Beneficial effects

When using inter-frame prediction according to the present invention as described above, coding performance can be improved by efficiently obtaining the predicted motion vectors.

Attached Figure Description

Figure 1 is a block diagram illustrating an image encoding and decoding system according to an embodiment of the present invention.

Figure 2 is a block diagram illustrating an image encoding device according to an exemplary embodiment of the present invention.

Figure 3 illustrates an image decoding device according to an embodiment of the present invention.

Figure 4 illustrates a tree-based block partitioning method as an embodiment of the present invention.

Figure 5 is an example diagram illustrating various cases of obtaining prediction blocks through inter-frame prediction according to the present invention.

Figure 6 is an example diagram of constructing a list of reference pictures according to an embodiment of the present invention.

Figure 7 is a conceptual diagram illustrating a non-translational motion model according to an embodiment of the present invention.

Figure 8 is an example diagram illustrating motion estimation in units of sub-blocks according to an embodiment of the present invention.

Figure 9 is a flowchart illustrating the encoding of motion information according to an embodiment of the present invention.

Figure 10 is a layout diagram of the target block and its adjacent blocks according to an embodiment of the present invention.

Figure 11 shows an example diagram of statistical candidates according to an embodiment of the present invention.

Figure 12 is a conceptual diagram of statistical candidates based on a non-translational motion model according to an embodiment of the present invention.

Figure 13 is an example diagram of the construction of motion information of the position of each control point stored as a statistical candidate according to an embodiment of the present invention.

Figure 14 is a flowchart illustrating motion information encoding according to an embodiment of the present invention.

Figure 15 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 16 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 17 is an example diagram of motion vector prediction candidates and motion vectors of a target block according to an embodiment of the present invention.

Figure 18 is an example diagram illustrating the arrangement of multiple motion vector prediction values according to an embodiment of the present invention.

Figure 19 is a flowchart illustrating motion information encoding under a merging mode according to an embodiment of the present invention.

The best mode of the invention

The image encoding/decoding method and apparatus of the present invention can construct a predicted motion candidate list of a target block, derive a predicted motion vector from the motion candidate list based on the predicted candidate index, reconstruct the predicted motion vector adjustment offset information, and reconstruct the motion vector of the target block based on the predicted motion vector and the predicted motion vector adjustment offset information.

In the image encoding/decoding method and apparatus of the present invention, the motion candidate list may include at least one of spatial candidates, temporal candidates, statistical candidates, or combined candidates.

In the image encoding/decoding method and apparatus of the present invention, the predicted motion vector adjustment offset can be determined based on at least one of an offset application flag or offset selection information.

In the image encoding/decoding method and apparatus of the present invention, information regarding whether the predicted motion vector adjustment offset information is supported may be included in at least one of the sequence, image, sub-image, slice, tile, or block.

In the image encoding/decoding method and apparatus of the present invention, when encoding a target block in a merge mode, the motion vector of the target block can be reconstructed by using a zero vector, and when encoding a target block in a contention mode, the motion vector of the target block can be reconstructed by using a motion vector difference.

The mode of the present invention

This disclosure can be modified in various ways and has various implementations. Specific implementations of this disclosure will be described with reference to the accompanying drawings. However, these implementations are not intended to limit the technical scope of this disclosure, and it should be understood that this disclosure covers various modifications, equivalents, and alternatives within the scope and concept of this disclosure.

The terms first, second, A, and B used in this disclosure may be used to describe various components, rather than to limit them. These expressions are used only to distinguish one component from another. For example, without departing from the scope of this disclosure, a first component may be referred to as a second component, and a second component may be referred to as a first component. The term and/or covers a combination of or any one of several related terms.

When a component is said to be "connected to" another component or "coupled to/coupled to" another component, it should be understood that a component is connected to the other component directly or through any other component. On the other hand, when a component is said to be "directly connected to" or "directly coupled to" another component, it should be understood that there are no other components between the components.

The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. Unless the context clearly specifies otherwise, the singular form includes the plural reference. In this disclosure, the terms “comprising” or “having” indicate the presence of a feature, number, step, operation, component, part, or combination thereof, but do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, terms including technical or scientific terms used in this disclosure may have the same meaning as commonly understood by those skilled in the art. Terms that are commonly defined in dictionaries may be interpreted as having the same or similar meaning as in the context of the relevant art. Unless otherwise defined, these terms should not be interpreted as having an ideal or overly formal meaning.

Typically, an image can include one or more color spaces depending on its color format. The image can include one or more images of the same size or one or more images of different sizes. For example, the YCbCr color configuration can support color formats such as 4:4:4, 4:2:2, 4:2:0, and monochrome (consisting only of Y). For example, YCbCr 4:2:0 can consist of one luminance component (Y in this example) and two chrominance components (Cb/Cr in this example). In this case, the configuration ratio of the chrominance and luminance components can have a width-to-height ratio of 1:2. For example, in the case of 4:4:4, it can have the same configuration ratio in terms of width and height. As in the examples above, when an image includes one or more color spaces, the image can be divided into color spaces.

Images can be classified into I, P, and B based on their image type (e.g., picture type, subpicture type, slice type, tile type, block type, etc.). I images can be images encoded without a reference image. P images can be images encoded using a reference image, allowing only forward prediction. B images can be images encoded using a reference image, allowing bidirectional prediction. However, depending on the encoding settings, some types (P and B) can be combined, or different compositions of image types can be supported.

The various encoded/decoded information fragments generated in this disclosure can be processed explicitly or implicitly. Explicit processing can be understood as the following: generating encoded/decoded information in sequences, images, sub-images, slices, tiles, blocks, chunks, or sub-blocks by an encoder, including selection information in the bitstream, and reconstructing the relevant information into decoded information by a decoder parsing the relevant information at the same unit level as in the encoder. Implicit processing can be understood as processing the encoded/decoded information using the same procedures, rules, etc., at both the encoder and decoder.

Figure 1 is a conceptual diagram illustrating an image encoding and decoding system according to an embodiment of the present disclosure.

Referring to FIG1, each of the image encoding device 105 and the image decoding device 100 may be a user terminal, such as a personal computer (PC), laptop computer, personal digital assistant (PDA), portable multimedia player (PMP), portable game console (PSP), wireless communication terminal, smartphone, or television (TV), or a server terminal such as an application server or service server. Each of the image encoding device 105 and the image decoding device 100 may be any of a variety of devices, each device including: a communication device such as a communication modem that communicates with various devices or wired/wireless communication networks; a memory 120 or 125 that stores various programs and data for inter-frame prediction or intra-frame prediction to encode or decode images; or a processor 110 or 115 that performs calculation and control operations by executing programs.

Furthermore, the image encoding device 105 can transmit images encoded as bitstreams to the image decoding device 100 in real time or non-real time via wired/wireless communication networks such as the Internet, short-range wireless communication networks, wireless local area networks (WLANs), wireless broadband (Wi-Bro) networks, or mobile communication networks, or via various communication interfaces such as cables or universal serial buses (USB). The image decoding device 100 can reconstruct the image from the received bitstream by decoding the bitstream and reproduce the image. Additionally, the image encoding device 105 can transmit images encoded as bitstreams to the image decoding device 100 via a computer-readable recording medium.

Although the aforementioned image encoding and image decoding devices can be separate devices, depending on the implementation, they can be incorporated into a single image encoding/decoding device. In this case, some components of the image encoding device can be substantially the same as their corresponding components of the image decoding device. Therefore, these components can be configured to include the same structure or perform at least the same function.

Therefore, in the following detailed description of the technical components and their operating principles, redundant descriptions of the corresponding technical components will be omitted. Furthermore, since the image decoding device is a computing device that applies the image encoding method executed in the image encoding device to decoding, the following description will focus on the image encoding device.

The computing device may include: a memory storing programs or software modules that perform image encoding and/or image decoding methods; and a processor connected to the memory and executing the programs. An image encoding device may be referred to as an encoder, and an image decoding device may be referred to as a decoder.

Figure 2 is a block diagram illustrating an image encoding apparatus according to an embodiment of the present disclosure.

Referring to FIG2, the image encoding device 20 may include a prediction unit 200, a subtraction unit 205, a transformation unit 210, a quantization unit 215, an inverse quantization unit 220, an inverse transformation unit 225, an addition unit 230, a filter unit 235, an encoded image buffer 240, and an entropy encoding unit 245.

The prediction unit 200 can be implemented using a prediction module as a software module, and generates prediction blocks for the blocks to be encoded via intra-frame prediction or inter-frame prediction. The prediction unit 200 can generate prediction blocks by predicting the target block to be encoded in the image. In other words, the prediction unit 200 can generate prediction blocks with predicted pixel values for each pixel by predicting the pixel values of pixels in the target block based on inter-frame prediction or intra-frame prediction. Furthermore, the prediction unit 200 can provide the encoding unit with information required to generate prediction blocks—such as information about prediction modes, like intra-frame prediction or inter-frame prediction modes—so that the encoding unit can encode the information about the prediction modes. The processing unit undergoing prediction, the prediction method, and specific details about the processing unit can be determined based on the encoding settings. For example, the prediction method and prediction mode can be determined based on the prediction unit, and prediction can be performed based on the transform unit. Additionally, when using a specific encoding mode, the original block can be encoded as is and sent to the decoder without generating prediction blocks through the prediction unit.

Intra-prediction units can have: directional prediction modes, such as horizontal modes, vertical modes, etc., used according to the prediction direction; and non-directional prediction modes, such as DC, plane, etc., using methods such as averaging and interpolating reference pixels. Intra-prediction mode candidate groups can be constructed using directional and non-directional modes, and one of various candidates, such as 35 prediction modes (33 directional + 2 non-directional), 67 prediction modes (65 directional + 2 non-directional), or 131 prediction modes (129 directional + 2 non-directional), can be used as candidate groups.

The intra-prediction unit may include a reference pixel construction unit, a reference pixel filter unit, a reference pixel interpolation unit, a prediction mode determination unit, a prediction block generation unit, and a prediction mode coding unit. The reference pixel construction unit can construct pixels belonging to and adjacent to the target block as reference pixels for intra-prediction. Depending on the coding settings, one adjacent reference pixel line can be constructed as a reference pixel, or another adjacent reference pixel line can be constructed as a reference pixel, or multiple reference pixel lines can be constructed as reference pixels. When some reference pixels are unavailable, available reference pixels can be used to generate a reference pixel. When all reference pixels are unavailable, a predetermined value (e.g., the median of the pixel value range represented by the bit depth) can be used to generate a reference pixel.

To reduce degradation remaining after coding processing, the reference pixel filter unit of the intra-prediction unit can perform filtering on the reference pixel. In this case, the filter used can be a low-pass filter, such as a 3-tap filter [1/4, 1/2, 1/4], a 5-tap filter [2/16, 3/16, 6/16, 3/16, 2/16], etc. Whether to apply filtering and the type of filtering can be determined based on coding information (e.g., block size, shape, prediction mode, etc.).

The reference pixel interpolation unit of the intra-prediction unit can generate fractional-unit pixels based on the prediction mode through linear interpolation of the reference pixels, and can determine the interpolation filter to be applied based on the coding information. In this case, the interpolation filter used may include a 4-tap cubic filter, a 4-tap Gaussian filter, a 6-tap Wiener filter, an 8-tap Kalman filter, etc. Typically, interpolation and low-pass filtering are performed separately, but the filtering process can be performed by combining the filters applied to both processes into one.

The prediction mode determination unit of the intra-frame prediction unit can select at least one optimal prediction mode from the prediction mode candidates, taking coding cost into account, and the prediction block generation unit can use the corresponding prediction mode to generate a prediction block. The prediction mode coding unit can encode the optimal prediction mode based on the prediction value. In this case, the prediction information can be adaptively encoded according to whether the prediction value is correct or not.

In an intra-frame prediction unit, the predicted value is called the most probable mode (MPM), and some modes belonging to the prediction mode candidate group can be constructed into an MPM candidate group. The MPM candidate group can include predicted modes of a predetermined prediction mode (e.g., DC mode, planar mode, vertical mode, horizontal mode, diagonal mode, etc.) or predicted modes of spatially adjacent blocks (e.g., left block, top block, top-left block, top-right block, bottom-left block, etc.). Additionally, modes derived from modes previously included in the MPM candidate group (the difference between +1 and -1 in the case of directional mode) can also be constructed into an MPM candidate group.

There may be a priority for the prediction patterns used to construct MPM candidate groups. The order in which patterns are included in the MPM candidate groups can be determined based on the priority, and the construction of MPM candidate groups can be completed by filling the number of MPM candidate groups according to the priority (determined by the number of prediction pattern candidate groups). In this case, the priority can be determined according to the order of prediction patterns of spatially adjacent blocks, predetermined prediction patterns, and patterns derived from prediction patterns previously included in the MPM candidate groups, but other modifications are also possible.

For example, spatially adjacent blocks can be included in the candidate group in the order of left block, top block, bottom left block, top right block, top left block, etc., and predetermined prediction patterns can be included in the candidate group in the order of DC pattern, planar pattern, vertical pattern, and horizontal pattern. A total of six patterns can be constructed into candidate groups by including patterns obtained by adding +1, -1, etc., to the candidate groups from already included patterns. Alternatively, a total of seven patterns can be constructed into candidate groups by including a priority such as left, top, DC, planar, bottom left, top right, top left, (left+1), (left-1), (top+1).

Subtraction unit 205 can generate residual blocks by subtracting prediction blocks from target blocks. In other words, subtraction unit 205 can calculate the difference between the pixel value of each pixel in the target block to be encoded and the predicted pixel value of the corresponding pixel in the prediction block generated by the prediction unit, to generate a residual signal in the form of a block, i.e., a residual block. Furthermore, subtraction unit 205 can generate residual blocks using units other than those obtained by the block partitioning unit described later.

Transformation unit 210 can transform a spatial signal into a frequency signal. The signal obtained through the transformation process is called the transform coefficient. For example, a residual block having a residual signal received from a subtraction unit can be transformed into a transform block with transform coefficients, and the input signal is determined according to the encoding configuration, which is not limited to the residual signal.

Transformation units can transform residual blocks using transformation schemes such as, but not limited to, Hadamard transform, discrete sine transform (DST)-based transform, or DCT-based transform. These transformation schemes can be changed and modified in various ways.

It can support at least one of the transformation schemes, and can support at least one sub-transformation scheme for each transformation scheme. A sub-transformation scheme can be obtained by modifying a portion of the fundamental vectors in the transformation scheme.

For example, in the case of DCT, one or more sub-transformation schemes DCT-1 to DCT-8 can be supported, and in the case of DST, one or more sub-transformation schemes DST-1 to DST-8 can be supported. A transformation scheme candidate group can be configured with a portion of the sub-transformation schemes. For example, DCT-2, DCT-8, and DST-7 can be grouped into a candidate group for transform.

Transformations can be performed in both the horizontal and vertical directions. For example, a one-dimensional transformation can be performed in the horizontal direction using DCT-2, and a one-dimensional transformation can be performed in the vertical direction using DST-7. Two-dimensional transformations can be used to transform pixel values from the spatial domain to the frequency domain.

A fixed transform scheme can be used, or the transform scheme can be adaptively selected based on the coding configuration. In the latter case, the transform scheme can be selected explicitly or implicitly. When the transform scheme is selected explicitly, information about the transform scheme or set of transform schemes applied in each of the horizontal and vertical directions can be generated, for example, at the block level. When the transform scheme is selected implicitly, the coding configuration can be limited based on image type (I/P/B), color components, block size, block shape, block position, intra-frame prediction mode, etc., and a predetermined transform scheme can be selected based on the coding settings.

In addition, some transformations can be skipped depending on the encoding settings. That is, one or more horizontal and vertical units can be explicitly or implicitly omitted.

Furthermore, the transform unit can send the information needed to generate the transform block to the encoding unit, which encodes the information, includes the encoded information in the bitstream, and sends the bitstream to the decoder. Therefore, the decoder's decoding unit can parse the information from the bitstream for inverse transform.

The quantization unit 215 can quantize the input signal. The signal obtained from the quantization is called the quantization coefficient. For example, the quantization unit 215 can obtain a quantized block with quantization coefficients by quantizing a residual block having residual transform coefficients received from the transform unit, and can determine the input signal according to the encoding settings, which are not limited to the residual transform coefficients.

The quantization unit can quantize the residual block of the transformation using quantization schemes such as, but not limited to, dead-zone uniform boundary value quantization and quantization weighted matrix. These quantization schemes can be changed and modified in various ways.

Quantization can be skipped based on encoding settings. For example, quantization (and inverse quantization) can be skipped based on encoding settings (e.g., a quantization parameter of 0, i.e., a lossless compression environment). In another example, quantization can be omitted when quantization-based compression performance is not performed due to image characteristics. Quantization can be skipped in the entirety or a portion of a quantization block (M×N) (M/2×N/2, M×N/2, or M/2×N), and quantization skip selection information can be set explicitly or implicitly.

The quantization unit can send the information needed to generate the quantization block to the encoding unit, which then encodes the information, includes the encoded information in the bitstream, and sends the bitstream to the decoder. Therefore, the decoder's decoding unit can parse the information from the bitstream for inverse quantization.

Although the above example was described assuming that the residual block is transformed and quantized via a transform unit and a quantization unit, a residual block with transform coefficients can be generated by transforming the residual signal without quantization. The residual block can undergo quantization only without transformation. Alternatively, the residual block can undergo both transformation and quantization. These operations can be determined based on the encoding settings.

The inverse quantization unit 220 performs inverse quantization on the residual block quantized by the quantization unit 215. That is, the inverse quantization unit 220 generates a residual block with frequency coefficients by performing inverse quantization on the quantized frequency coefficient sequence.

The inverse transform unit 225 performs an inverse transform on the residual block dequantized by the inverse quantization unit 220. That is, the inverse transform unit 225 performs an inverse transform on the frequency coefficients of the dequantized residual block to generate a residual block with pixel values, i.e., a reconstructed residual block. The inverse transform unit 225 can perform the inverse transform by reversing the transform scheme used by the transform unit 210.

Addition unit 230 reconstructs the target block by adding the predicted block predicted by prediction unit 200 to the residual block recovered by inverse transform unit 225. The reconstructed target block is stored as a reference picture (or reference block) in encoded picture buffer 240 for use as a reference picture when encoding the next block, another block, or another picture of the target block later.

Filter unit 235 may include one or more post-processing filters, such as a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF). The deblocking filter removes block distortion occurring at boundaries between blocks in the reconstructed image. The ALF performs filtering based on values obtained by comparing the reconstructed image after filtering the blocks with the deblocking filter with the original image. The SAO reconstructs the pixel-level offset difference between the original image and the residual blocks to which the deblocking filter was applied. These post-processing filters can be applied to the reconstructed image or blocks.

The encoded image buffer 240 can store blocks or images reconstructed by the filter unit 235. The reconstructed blocks or images stored in the encoded image buffer 240 can be provided to the prediction unit 200 that performs intra-frame prediction or inter-frame prediction.

Entropy coding unit 245 can generate a quantization coefficient sequence, a transform coefficient sequence, or a signal sequence by scanning the quantization coefficients, transform coefficients, or residual signals of the generated residual block according to at least one scanning order (e.g., zigzag scanning, vertical scanning, horizontal scanning, etc.). Entropy coding unit 245 can encode the quantization coefficient sequence, transform coefficient sequence, or signal sequence using at least one entropy coding technique. In this case, information about the scanning order can be determined according to coding settings (e.g., image type, coding mode, prediction mode, transform type, etc.), and the relevant information can be implicitly determined or explicitly generated.

Additionally, encoded data, including the encoded information sent from each component, can be generated and output as a bitstream, which can be implemented by a multiplexer (MUX). In this case, encoding can be performed using methods such as exponential Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC) as encoding techniques, but is not limited to these, and various encoding techniques derived from and modified from them can be used.

When entropy encoding (assuming CABAC in this example) is performed on syntax elements such as residual block data and information generated during encoding/decoding processes, an entropy encoding device may include a binarization unit (binarizer), a context modeler, and a binary arithmetic encoding unit (binary arithmetic encoder). In this case, the binary arithmetic encoding unit may include a regular encoding engine and a bypass encoding engine.

Since the syntax elements input to the entropy coding device may not be binary values, if the syntax element is not a binary value, the binarization unit can binarize the syntax element and output a binary string consisting of 0s or 1s. In this case, the binary (bin) can represent bits consisting of 0s or 1s and can be encoded by the binary arithmetic coding unit. In this case, one of the regular coding unit or the bypass coding unit can be selected based on the probability of occurrence of 0s and 1s, and this can be determined according to the encoding/decoding settings. If the syntax element is data with the same frequency of 0s and 1s, the bypass coding unit can be used; otherwise, the regular coding unit can be used.

Various methods can be used when binarizing grammatical elements. For example, fixed-length binarization, unary binarization, truncated Ricean binarization, and K-order exponential Golomb binarization can be used. Additionally, signed or unsigned binarization can be performed depending on the range of values for the grammatical element. The binarization processing of grammatical elements described in this invention can be performed not only on the binarization methods mentioned in the examples above, but also on other additional binarization methods.

Figure 3 is a block diagram illustrating an image decoding apparatus according to an embodiment of the present disclosure.

Referring to Figure 3, the image decoding device 30 can be configured to include an entropy decoder 305, a prediction unit 310, an inverse quantization unit 315, an inverse transform unit 320, an addition/subtraction unit 325, a filter 330, and a decoded image buffer 335.

Furthermore, the prediction unit 310 can be configured to include an intra-frame prediction module and an inter-frame prediction module.

When an image bitstream is received from the image encoding device 20, it can be sent to the entropy decoder 305.

The entropy decoder 305 can decode the bitstream into decoded data including quantization coefficients and decoded information to be sent to each component.

Prediction unit 310 can generate prediction blocks based on data received from entropy decoder 305. A list of reference images can be generated using a default configuration scheme based on reference images stored in decoded image buffer 335.

The intra-frame prediction unit may include a reference sample construction unit, a reference sample filter unit, a reference sample interpolation unit, a prediction block generation unit, and a prediction mode decoding unit. Some of these units may perform the same processing as in the encoder, while others may perform the encoder's processing in reverse.

The inverse quantization unit 315 can inverse quantize the quantized transform coefficients provided in the bitstream and decoded by the entropy decoder 305.

The inverse transform unit 320 can generate residual blocks by applying inverse DCT, inverse integer transform, or similar inverse transform techniques to the transform coefficients.

The inverse quantization unit 315 and the inverse transform unit 320 can perform the processing of the transform unit 210 and the quantization unit 215 of the image encoding device 20 in reverse, and can be implemented in various ways. For example, the inverse quantization unit 315 and the inverse transform unit 320 can use the same processing and inverse transform shared with the transform unit 210 and the quantization unit 215, and can perform the transform and quantization in reverse using information received from the image encoding device 20 about the transform and quantization processing (e.g., transform size, transform shape, quantization type, etc.).

The residual block, which has already undergone inverse quantization and inverse transform, can be added to the prediction block obtained by prediction unit 310 to generate a reconstructed image block. This addition can be performed by addition/subtraction unit 325.

Regarding filter 330, a deblocking filter can be applied when needed to remove blocking artifacts from the reconstructed image blocks. Additional loop filters can be used to improve video quality before and after decoding.

The reconstructed and filtered image blocks can be stored in the decoded image buffer 335.

Although not shown in the accompanying drawings, the image encoding/decoding device may also include image partitioning units and block partitioning units.

Image partitioning units can divide or partition an image into at least one region based on predetermined partitioning units. Here, partitioning units can include sub-images, slices, tiles, blocks, blocks (e.g., maximum coding units), etc.

An image can be divided into one or more rows or columns of tiles. In this case, a tile can be a block-based unit that includes a predetermined non-square area of the image. Alternatively, a tile can be divided into one or more blocks, and a block can consist of blocks arranged in rows or columns.

Slices can be set using one or more configurations, one of which can be composed of bundles (e.g., blocks, regions, tiles, etc.) depending on the scan order, and one of them can be composed of shapes that include non-square regions, and other additional definitions are also possible.

Regarding the definition of slice configuration, relevant information can be generated explicitly or implicitly. Multiple definitions can be set for the configuration of each partitioning unit and slice, and selection information can be generated accordingly.

Slices can be configured in non-square shapes such as blocks, regions, and tiles, and slice position and size information can be represented based on position information for the dividing unit (e.g., top left, bottom right, etc.).

In this invention, the description assumes that an image can be composed of one or more sub-images, a sub-image can be composed of one or more slices, tiles, or blocks, a slice can be composed of one or more tiles or blocks, and a tile can be composed of one or more blocks. However, the invention is not limited thereto.

A partition unit can consist of an integer number of blocks, but is not limited to this, and can consist of decimal numbers instead of integers. That is, when it is not composed of an integer number of blocks, at least one partition unit can consist of sub-blocks.

In addition to non-square slices, there can be partitioning units such as sub-images or tiles, and the position and size information of the units can be represented based on various methods.

For example, the position and size information of non-square cells can be represented based on information about the number of non-square cells, information about the number of columns or rows of non-square cells, information about whether they are evenly divided into columns or rows in non-square cells, information about the width or height of column or row cells in non-square cells, and index information of non-square cells.

In the case of sub-images and tiles, the position and size information of each unit can be represented based on all or part of the information, and based on this, it can be divided or partitioned into one or more units.

Simultaneously, blocks can be divided into various units and sizes using block partitioning units. A basic coding unit (or maximum coding unit, coding tree unit, CTU) can refer to the basic (or initial) unit used for prediction, transformation, quantization, etc., in image coding processing. In this case, a basic coding unit can consist of one luminance basic coding block (maximum coding block or CTB) and two basic chrominance coding blocks, depending on the color format (YCbCr in this example), and the size of each block can be determined according to the color format. Coding blocks (CBs) can be obtained according to the partitioning process. A CB can be understood as a unit that is no longer further subdivided due to certain limitations and can be set as the starting unit for partitioning into subunits. In this disclosure, blocks conceptually include various shapes such as triangles, circles, etc., and are not limited to squares.

While the following description is given in the context of a single color component, it is also applicable, proportionally to the ratios according to the color format, to other color components with some modifications (e.g., in the case of YCbCr 4:2:0, the width-to-height length ratio of the luminance component to the chrominance component is 2:1). Furthermore, although block partitioning depending on other color components (e.g., depending on the block partitioning result of Y in Cb/Cr) is feasible, it should be understood that independent block partitioning for each color component is also feasible. Moreover, while a common block partitioning configuration can be used (considering proportionality to the length ratio), it is also necessary to consider and understand the use of separate block partitioning configurations based on the color components.

In a block partitioning unit, a block can be represented as M×N, and the maximum and minimum values of each block can be obtained within this range. For example, if the maximum and minimum values of a block are 256×256 and 4×4 respectively, then blocks of size ^2m × ²ⁿ (where m and n are integers from 2 to 8 in this example), blocks of size 2m×2m (where m and n are integers from 2 to 128 in this example), or blocks of size m×m (where m and n are integers from 4 to 256 in this example) can be obtained. In this paper, m and n can be equal or different, and one or more ranges, such as maximum and minimum values, can be generated to support the blocks.

For example, information about the maximum and minimum size of a block can be generated, and information about the maximum and minimum size of a block can be generated in some partitioning configurations. In the former case, the information can be about the range of maximum and minimum sizes that can be generated in the image, while in the latter case, the information can be about the maximum and minimum sizes that can be generated based on some partitioning configuration. The partitioning configuration can be defined by image type (I/P/B), color components (YCbCr, etc.), block type (encoding/prediction/transformation/quantization), partition type (index or type), and partitioning scheme (quadtree QT, binary tree BT, and ternary tree TT as tree methods, and SI2, SI3, and SI4 as type methods).

Furthermore, constraints can exist on the aspect ratio (block shape) that can be used for blocks, and at this point, boundary values can be set. Only blocks less than or equal to/less than the boundary value (k) can be supported, where k can be limited according to the aspect ratio A/B (A is the longer or equal value between the width and height, and B is another value). k can be a real number equal to or greater than 1, such as 1.5, 2, 3, 4, etc. As in the example above, constraints on the shape of a block in an image can be supported, or one or more constraints can be supported depending on the partitioning configuration.

In summary, whether block partitioning is supported can be determined based on the above scope and constraints, as well as the partitioning configuration described later. For example, partitioning can be supported when a candidate (child block) split from a block (parent block) meets the block support conditions; otherwise, partitioning may not be supported.

The block partitioning unit can be configured with respect to each component of the image encoding and image decoding devices, and the size and shape of the block can be determined in the process. Different blocks can be configured according to the component. The blocks may include prediction blocks for prediction units, transform blocks for transform units, and quantization blocks for quantization units. However, this disclosure is not limited to this, and block units may be defined separately for other components. Although the shape of each of the inputs and outputs in each component is described as non-square in this disclosure, the inputs and outputs of some components may have any other shape (e.g., non-square triangles).

The size and shape of the initial (or starting) block in the block partitioning unit can be determined from the higher unit. The initial block can be divided into smaller blocks. Once the optimal size and shape are determined based on the block partitioning, that block can be designated as the initial block for the lower unit. The higher unit can be a coding block, and the lower unit can be a prediction block or a transform block, and this disclosure is not limited thereto. Instead, various modifications are possible. Once the initial block of the lower unit is determined as in the example above, a partitioning process can be performed to detect blocks of optimal size and shape, similar to the higher unit.

In summary, a block partitioning unit can divide a basic coding block (or the largest coding block) into at least one coding block, and a coding block can be divided into at least one prediction block/transform block/quantization block. Furthermore, a prediction block can be divided into at least one transform block/quantization block, and a transform block can be divided into at least one quantization block. Some blocks may have dependencies on other blocks (i.e., defined by higher and lower units) or may be independent of other blocks. For example, a prediction block may be a higher unit above a transform block, or it may be a unit independent of the transform block. Various relationships can be established based on the type of block.

Depending on the encoding settings, it can be determined whether higher and lower units are combined. Combining units means that blocks of higher units undergo encoding processing by lower units (e.g., in prediction units, transform units, inverse transform units, etc.) without being split into lower units. In other words, this may mean sharing the partitioning process among multiple units and generating partitioning information in one unit (e.g., a higher unit).

For example, (when a coded block is combined with a prediction block or a transform block), the coded block can undergo prediction, transform, and inverse transform.

For example, (when a coded block is combined with a prediction block), the coded block can undergo prediction, and a transform block that is equal to or smaller in size than the coded block can undergo both transformation and inverse transformation.

For example, (when a coded block is combined with a transform block), a prediction block that is equal to or smaller in size than the coded block can be predicted, and the coded block can be transformed and inverse transformed.

For example, (when a prediction block is combined with a transform block), a prediction block that is equal to or smaller in size than a coding block can undergo prediction, transform, and inverse transform.

For example, (when there is no block combination), a prediction block that is equal to or smaller than the coded block in size can be predicted, and a transform block that is equal to or smaller than the coded block in size can be transformed and inverse transformed.

Although various cases of coded blocks, prediction blocks, and transform blocks have been described in the examples above, this disclosure is not limited thereto.

For the combination of units, a fixed configuration can be supported in the image, or an adaptive configuration can be supported considering various coding factors. Coding factors include image type, color components, coding mode (intra-frame/inter-frame), partitioning configuration, block size/shape/position, aspect ratio, prediction-related information (e.g., intra-frame prediction mode, inter-frame prediction mode, etc.), transform-related information (e.g., transform scheme selection information, etc.), quantization-related information (e.g., quantization region selection information and quantization transform coefficient coding information), etc.

Once a block of optimal size and shape has been detected as described above, pattern information (e.g., partitioning information) for that block can be generated. This pattern information, along with information generated from the components to which the block belongs (e.g., prediction-related information and transformation-related information), can be included in the bitstream and sent to the decoder, where it can be parsed at the same unit level for use in the video decoding process.

The partitioning scheme will now be described. Although for ease of description, it is assumed that the initial blocks are shaped as squares, this disclosure is not limited thereto, and the description can be applied in the same or similar manner to cases where the initial blocks are not squares.

The block partitioning unit can support various types of partitions. For example, it can support tree-based partitioning or index-based partitioning, and other methods as well. In tree-based partitioning, the partition type can be determined based on various types of information (e.g., information indicating whether to perform partitioning, tree type, partition direction, etc.), while in index-based partitioning, specific index information can be used to determine the partition type.

Figure 4 is an example diagram showing the various partition types that can be obtained in the block partitioning unit of this disclosure.

In this example, it is assumed that the partition type shown in Figure 4 was obtained through a single partition operation (or procedure), which should not be construed as limiting the scope of this disclosure. The partition type may also be obtained through multiple partition operations. Furthermore, other partition types not shown in Figure 4 may also be available.

(Tree-based partitioning)

In the tree-based partitioning of this disclosure, QT, BT, and TT can be supported. If one tree method is supported, this can be called single-tree partitioning, while if two or more tree methods are supported, this can be called multi-tree partitioning.

In QT, a block is divided into two partitions (i.e., 4 partitions) in each of the horizontal and vertical directions (n), while in BT, a block is divided into two partitions (b to g) in either the horizontal or vertical direction. In TT, a block is divided into three partitions (h to m) in either the horizontal or vertical direction.

In Qt, a block can be divided into four partitions (o and p) by restricting the partitioning direction to either the horizontal or vertical direction. Furthermore, in BT, it supports partitioning the block into only equal-sized partitions (b and c), partitioning the block into only partitions of different sizes (d to g), or both of these partitioning types. Additionally, in TT, it supports partitioning the block into partitions focused only on a specific direction (1:1:2 or 2:1:1 in the left-to-right or top-to-bottom direction) (h, j, k, and m), partitioning the block into partitions focused on the center (1:2:1) (i and l), or both of these partitioning types. Furthermore, it supports partitioning the block into four partitions in each of the horizontal and vertical directions (i.e., a total of 16 partitions) (q).

The tree method supports partitioning a block into z partitions (b, d, e, h, i, j, o) only horizontally, or into z partitions (c, f, g, k, l, m, p) only vertically, or both partition types. In this paper, z can be an integer equal to or greater than 2, for example, 2, 3, or 4.

In this disclosure, it is assumed that partition type n is supported as QT, partition types b and c are supported as BT, and partition types i and l are supported as TT.

Depending on the encoding settings, one or more tree partitioning schemes can be supported. For example, QT, QT/BT, or QT/BT/TT can be supported.

In the examples above, the basic tree partitioning scheme is QT, and BT and TT are included as additional partitioning schemes depending on whether other trees are supported. However, various modifications can be made. Information indicating whether other trees are supported (bt_enabled_flag, tt_enabled_flag, and bt_tt_enabled_flag, where 0 indicates no support and 1 indicates support) can be implicitly determined based on encoding settings or explicitly determined in units such as sequences, images, sub-images, slices, tiles, blocks, etc.

Partition information may include indications of whether partitioning is performed (tree_part_flag or qt_part_flag, bt_part_flag, tt_part_flag, and bt_tt_part_flag, which can have values of 0 or 1, where 0 indicates no partitioning and 1 indicates partitioning). Additionally, depending on the partitioning scheme (BT and TT), information about the partitioning direction can be added (dir_part_flag, or bt_dir_part_flag, tt_dir_part_flag, and bt_tt_dir_part_flag, which have values of 0 or 1, where 0 indicates <width/horizontal> and 1 indicates <height/vertical>). This information can be generated during the partitioning process.

When multi-tree partitioning is supported, various partition information fragments can be configured. For ease of description, the following description provides an example of how to configure partition information at a depth level (that is, recursive partitioning can be performed by setting one or more supported partition depths).

In example (1), check the information indicating whether to perform partitioning. If partitioning is not performed, the partitioning process ends.

If partitioning is performed, the selection information regarding the partition type is checked (e.g., tree_idx, 0 for QT, 1 for BT, and 2 for TT). The partition direction information is additionally checked based on the selected partition type, and the process proceeds to the next step (if additional partitioning is feasible due to reasons such as the partition depth not yet reaching its maximum value, the process restarts from the beginning; if additional partitioning is not possible, the partitioning process ends).

In Example (2), the system checks whether the partitioning is performed using a certain tree scheme (QT), and the process proceeds to the next step. If the partitioning is not performed using a tree scheme (QT), the system checks whether the partitioning is performed using another tree scheme (BT). In this case, if the partitioning is not performed using that tree scheme, the system checks whether the partitioning is performed using a third tree scheme (TT). If the partitioning is not performed using a third tree scheme (TT), the partitioning process ends.

If partitioning is performed using a tree-based scheme (QT), the process proceeds to the next step. Alternatively, if partitioning is performed using a second-tree scheme (BT), the partition orientation information is checked, and the process proceeds to the next step. If partitioning is performed using a third-tree scheme (TT), the partition orientation information is checked, and the process proceeds to the next step.

In Example (3), the system checks whether to perform partitioning using a tree scheme (QT). If partitioning is not performed using a tree scheme (QT), the system checks whether to perform partitioning using another tree scheme (BT and TT). If partitioning is not performed, the partitioning process ends.

If partitioning is performed using a tree scheme (QT), the process proceeds to the next step. Alternatively, if partitioning is performed using other tree schemes (BT and TT), the partition orientation information is checked, and the process proceeds to the next step.

While the examples above either prioritize the tree partitioning scheme (Examples 2 and 3) or do not prioritize it (Example 1), various modifications are also possible. Furthermore, in the examples above, the partitioning in the current step is independent of the partitioning result of the previous step. However, it is also possible to configure the partitioning in the current step to depend on the partitioning result of the previous step.

In Examples 1 through 3, if some tree partitioning scheme (QT) was executed in a previous step, and the process has thus reached the current step, the same tree partitioning scheme (QT) can also be supported in the current step.

On the other hand, if a certain tree partitioning scheme (QT) was not executed in a previous step and therefore another tree partitioning scheme (BT or TT) was executed, and then the process proceeds to the current step, it can be configured to support other tree partitioning schemes (BT and TT) besides a certain tree partitioning scheme (QT) in the current step and subsequent steps.

In the above cases, the tree configuration supporting block partitioning can be adaptive, and therefore the partitioning information described above can also be configured differently. (Assuming the example described below is Example 3). That is, if partitioning was not performed using a certain tree scheme (QT) in the previous steps, the partitioning process can be performed without considering the tree scheme (QT) in the current step. Furthermore, partitioning information associated with a particular tree scheme (e.g., information indicating whether to perform partitioning, information about the partitioning direction, etc. In this example <QT>, information indicating whether to perform partitioning) can be removed.

The examples above illustrate adaptive partitioning information configuration for cases where block partitioning is allowed (e.g., block size is within the range of maximum and minimum values, partition depth of each tree scheme has not reached the maximum depth (allowed depth), etc.). Even when block partitioning is restricted (e.g., block size does not exist within the range of maximum and minimum values, partition depth of each tree scheme has reached the maximum depth, etc.), partitioning information can still be configured adaptively.

As already mentioned, tree-based partitioning can be performed recursively in this disclosure. For example, if the partition flag of a coded block with partition depth k is set to 0, then block encoding is performed in the coded block with partition depth k. If the partition flag of a coded block with partition depth k is set to 1, then block encoding is performed in N sub-coded blocks with partition depth k+1 according to the partitioning scheme (where N is an integer equal to or greater than 2, such as 2, 3, and 4).

In the above process, a sub-coded block can be set as a coded block (k+1) and partitioned into sub-coded blocks (k+2). This hierarchical partitioning scheme can be determined based on partitioning configurations such as partition range and allowed partition depth.

In this case, the bitstream structure representing the partition information can be selected from one or more scanning methods. For example, the bitstream of partition information can be configured based on the order of partition depth or based on whether partitioning was performed.

For example, in the case of partition depth order, partition information is obtained at the current depth level based on the initial block, and then at the next depth level. In the case of whether partitioning is performed, additional partition information is first obtained from the blocks split from the initial block, and other additional scanning methods can be considered.

Regardless of the tree type (or all trees), the maximum block size and minimum block size can have common settings, separate settings for each tree, or common settings for two or more trees. In this case, the maximum block size can be set to be equal to or less than the maximum encoded block size. If the maximum block size of the first tree is not the same as the maximum block size, partitioning is implicitly performed using a predetermined second-tree method until the maximum block size of the first tree is reached.

Furthermore, regardless of the tree type, a common split depth can be supported; a separate split depth can be supported for each tree; or a common split depth can be supported for two or more trees. Alternatively, split depth can be supported for some trees, while not for others.

It supports explicit syntax elements for setting information, and can also implicitly determine some setting information.

(Index-based partitioning)

In the index-based partitioning of this disclosure, both Constant Partition Index (CSI) and Variable Partition Index (VSI) schemes can be supported.

In the CSI scheme, k sub-blocks can be obtained by partitioning in a predetermined direction, where k can be an integer equal to or greater than 2, such as 2, 3, or 4. Specifically, the size and shape of the sub-blocks can be determined based on k, regardless of the size and shape of the blocks. The predetermined direction can be one or a combination of two or more of the following: horizontal, vertical, and diagonal (top left to bottom right or bottom left to top right).

In the index-based CSI partitioning scheme of this disclosure, z candidates can be obtained by partitioning in a horizontal or vertical direction. In this case, z can be an integer equal to or greater than 2, such as 2, 3, or 4, and the sub-blocks can be equal in width and height, and equal or different in the other of width and height. The width or height ratio of the sub-blocks is _A1 : _A2 : ... : _AZ , and each of _A1 to _AZ can be an integer equal to or greater than 1, such as 1, 2, or 3.

Furthermore, candidates can be obtained by partitioning the blocks horizontally and vertically into x and y partitions, respectively. Each of x and y can be an integer equal to or greater than 1, such as 1, 2, 3, or 4. However, candidates where both x and y are 1 may be restricted (because a already exists). Although Figure 4 shows a case where the sub-blocks have the same width or height ratio, candidates with different width or height ratios can also be included.

Furthermore, candidates can be divided into w partitions along one of the diagonal directions—top left to bottom right and bottom left to top right. In this paper, w can be an integer equal to or greater than 2, such as 2 or 3.

Referring to Figure 4, partition types can be classified into symmetric partition types (b) and asymmetric partition types (d and e) based on the length ratio of each sub-block. Furthermore, partition types can be classified into partition types concentrated in a specific direction (k and m) and central partition types (k). Partition types can be defined by various coding factors, including sub-block shape and sub-block length ratio, and the supported partition types can be implicitly or explicitly determined based on coding settings. Therefore, candidate groups can be determined based on the partition types supported in an index-based partitioning scheme.

In the VSI scheme, with a fixed width (w) or height (h) for each sub-block, one or more sub-blocks can be obtained through partitioning in a predetermined direction. In this document, each of w and h can be an integer equal to or greater than 1, such as 1, 2, 4, or 8. Specifically, the number of sub-blocks can be determined based on the size and shape of the block and the w or h value.

In the index-based VSI partitioning scheme of this disclosure, candidates can be partitioned into sub-blocks, each of which has a fixed width and length. Alternatively, candidates can be partitioned into sub-blocks, each of which has a fixed width and length. Since the width or height of the sub-blocks is fixed, equal partitions in either the horizontal or vertical direction are allowed. However, this disclosure is not limited thereto.

If the size of the block before partitioning is M×N, and the width (w) of each sub-block is fixed, the height (h) of each sub-block is fixed, or both the width (w) and height (h) of each sub-block are fixed, then the number of sub-blocks obtained can be (M*N)/w, (M*N)/h, or (M*N)/w/h.

Depending on the coding settings, only one or both of the CSI and VSI schemes may be supported, and information about the supported schemes may be determined implicitly or explicitly.

This disclosure will be described in the context of the supported CSI schemes.

Candidate groups can be configured to include two or more candidates from an index-based partitioning scheme, depending on the encoding settings.

For example, candidate groups such as {a, b, c}, {a, b, c, n}, or {a to g and n} can be formed. Candidate groups can include examples of block types predicted to occur multiple times based on general statistical characteristics, such as blocks divided into two partitions in the horizontal or vertical direction, or in each of the horizontal and vertical directions.

Alternatively, candidate groups such as {a, b}, {a, o}, or {a, b, o} can be configured, or candidate groups such as {a, c}, {a, p}, or {a, c, p}. A candidate group can be an example that includes candidates in which each candidate is partitioned into two partitions horizontally and four partitions vertically. This could be an example of configuring a candidate group with block types predicted to be primarily partitioned in a specific direction.

Alternatively, candidate groups such as {a, o, p} or {a, n, q} can be configured. This could be an example of configuring candidate groups to include block types that are predicted to be partitioned into many smaller partitions than the blocks before partitioning.

Alternatively, candidate groups such as {a, r, s} can be configured, and can be, for example, determining the best partitioning result of a non-square shape that can be obtained from the block before splitting by other methods (tree methods), and configuring the non-square shape as a candidate group.

As noted in the examples above, various candidate group configurations can be available, and one or more candidate group configurations can be supported, taking into account various encoding/decoding factors.

Once the candidate groups are fully configured, various partition information configurations can be made available.

For example, index selection information can be generated for a candidate group that includes unpartitioned candidates (a) and partitioned candidates (b to s).

Alternatively, information indicating whether to perform partitioning can be generated (information indicating whether the partition type is 'a'). If partitioning is performed (if the partition type is not 'a'), index selection information about the candidate group (b to s) including the candidates to be partitioned can be generated.

Partition information can be configured in many other ways than those described above. In addition to information indicating whether to perform a partition, binary bits can be assigned to the index of each candidate in the candidate group in various ways, such as fixed-length binarization, variable-length binarization, etc. If the number of candidates is 2, one bit can be assigned to the index selection information, and if the number of candidates is 3, one or more bits can be assigned to the index selection information.

Compared to tree-based partitioning schemes, partition types that are predicted to occur multiple times can be included in the candidate group in index-based partitioning schemes.

Since the number of bits used to represent index information can increase based on the number of supported candidate groups, this scheme is suitable for single-level partitioning (e.g., partition depth is limited to 0) rather than tree-based hierarchical partitioning (recursive partitioning). That is, single partitioning operations can be supported without further subdividing the sub-blocks obtained through index-based partitioning.

This may mean that further partitioning into smaller blocks of the same type is not possible (e.g., a coded block obtained through index-based partitioning cannot be further subdivided into coded blocks), and it also means that further partitioning into blocks of different types may also be impossible (e.g., it is not possible to partition a coded block into prediction blocks and coded blocks). Clearly, this disclosure is not limited to the above examples, and other modifications are also possible.

Now, a description will be given of determining the block partitioning configuration primarily based on the block type among the coding factors.

First, encoded blocks can be obtained during the partitioning process. A tree-based partitioning scheme can be used for the partitioning process, and partition types such as a (no partitioning), n (QT), b, c (BT), i, or l (TT) as shown in Figure 4 can be generated based on the tree type. Depending on the encoding configuration, various combinations of tree types, such as QT/QT+BT/QT+BT+TT, may be available.

The following example illustrates the process of dividing the coded blocks obtained in the above steps into prediction blocks and transform blocks. It assumes that prediction, transform, and inverse transform are performed based on the size of each partition.

In example (1), prediction can be performed by setting the size of the prediction block to be equal to the size of the coding block, and transformation and inverse transformation can be performed by setting the size of the transform block to be equal to the size of the coding block (or prediction block).

In example (2), prediction can be performed by setting the size of the prediction block to be equal to the size of the coding block. The transform block can be obtained by partitioning the coding block (or prediction block), and the transform and inverse transform can be performed based on the size of the obtained transform block.

Here, a tree-based partitioning scheme can be used for the partitioning process, and partition types such as a (no partitioning), n (QT), b, c (BT), i, or l (TT) as shown in Figure 4 can be obtained based on the tree type. Depending on the encoding configuration, various combinations of tree types, such as QT/QT+BT/QT+BT+TT, may be available.

Here, the partitioning process can be an index-based partitioning scheme. The partition type can be obtained based on the index type, for example, a (no partition), b, c, or d in Figure 4. Depending on the encoding configuration, various candidate groups such as {a, b, c} and {a, b, c, d} can be configured.

In example (3), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. For the transform block, its size is set to the size of the coded block, and a transform and inverse transform can be performed on the transform block. In this example, the prediction block and the transform block can be independent of each other.

Index-based partitioning schemes can be used in the partitioning process, and the partition type can be obtained based on the index type, such as a (no partition), b to g, n, r, or s in Figure 4. Various candidate groups can be configured according to the encoding configuration, such as {a, b, c, n}, {a to g, n}, and {a, r, s}.

In example (4), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. For the transform block, its size is set to the size of the prediction block, and a transform and inverse transform can be performed on the transform block. In this example, the transform block can have a size equal to the size of the obtained prediction block, or the obtained prediction block can have a size equal to the size of the transform block (setting the size of the transform block to the size of the prediction block).

Tree-based partitioning schemes can be used in the partitioning process, and partition types can be generated based on the tree type, such as a (no partition), b, c (BT), or n (QT) in Figure 4. Depending on the encoding configuration, various combinations of tree types, such as QT/BT/QT+BT, may be available.

Here, an index-based partitioning scheme can be used in the partitioning process, and the partition type can be obtained based on the index type, such as a (no partition), b, c, n, o, or p in Figure 4. Various candidate groups can be configured according to the encoding configuration, such as {a, b}, {a, c}, {a, n}, {a, o}, {a, p}, {a, b, c}, {a, o, p}, {a, b, c, n}, and {a, b, c, n, p}. Furthermore, candidate groups can be configured in a standalone VSI scheme or in a combination of CSI and VSI schemes as an index-based partitioning scheme.

In Example (5), a prediction block can be obtained by partitioning the coded block, and the prediction block undergoes prediction based on the size of the obtained prediction block. Alternatively, a transform block can be obtained by partitioning the coded block, and the transform block undergoes transform and inverse transform based on the size of the obtained transform block. In this example, each of the prediction block and the transform block can be obtained by partitioning the coded block.

Here, tree-based partitioning schemes and index-based partitioning schemes can be used in the partitioning process, and candidate groups can be configured in the same way or in a similar manner as in Example 4.

In this context, the above examples represent possible scenarios depending on whether the process for partitioning each block type is shared, and should not be construed as limiting the scope of this disclosure. Various modifications are also possible. Furthermore, various encoding factors and block types can be considered to determine the block partitioning configuration.

Coding factors may include image type (I/P/B), color components (YCbCr), block size/shape/position, block aspect ratio, block type (coded block, prediction block, transform block, or quantization block), partition status, coding mode (intra-frame/inter-frame), prediction-related information (intra-frame prediction mode or inter-frame prediction mode), transform-related information (transform scheme selection information), and quantization-related information (quantization region selection information and quantization transform coefficient coding information).

In the image coding method according to an embodiment of the present invention, inter-frame prediction can be configured as follows. The inter-frame prediction of the prediction unit may include a reference image construction step, a motion estimation step, a motion compensation step, a motion information determination step, and a motion information coding step. Additionally, the video coding apparatus may include a reference image construction unit, a motion estimation unit, a motion compensation unit, a motion information determination unit, and a motion information coding unit that implement the reference image construction step, the motion estimation step, the motion compensation step, the motion information determination step, and the motion information coding step. Some of the above processes may be omitted, or other processes may be added, and the order may be changed from the above sequence.

In the video decoding method according to an embodiment of the present invention, inter-frame prediction can be configured as follows. The inter-frame prediction of the prediction unit may include a motion information decoding step, a reference image construction step, and a motion compensation step. Furthermore, the image decoding device may include a motion information decoding unit, a reference image construction unit, and a motion compensation unit that implement the motion information decoding step, the reference image construction step, and the motion compensation step. Some of the above processes may be omitted, or other processes may be added, and the order may be changed from the above order.

Since the reference image construction unit and motion compensation unit of the image decoding device perform the same functions as their counterparts in the image encoding device, detailed descriptions are omitted. Furthermore, the motion information decoding unit can be reverse-engineered using the methods employed in the motion information encoding unit. Here, the prediction block generated by the motion compensation unit can be sent to the addition unit.

Figure 5 is an example diagram illustrating various cases of obtaining prediction blocks through inter-frame prediction according to the present invention.

Referring to Figure 5, in unidirectional prediction, prediction block A can be obtained from previously encoded reference images (T-1, T-2) (forward prediction), or prediction block B can be obtained from later encoded reference images (T+1, T+2) (backward prediction). In bidirectional prediction, prediction blocks (C and D) can be generated from multiple previously encoded reference images (T-2 to T+2). Typically, P-image type supports unidirectional prediction, and B-image type supports bidirectional prediction.

As in the example above, the images referenced for the encoding of the current image can be obtained from memory, and a list of reference images can be constructed based on the current image (T) in chronological or display order, including reference images before and after the current image.

Inter-frame prediction (E) can be performed on the current image and on previous or subsequent images based on the current image. Performing inter-frame prediction on the current image can be referred to as non-directional prediction. This can be supported by either I-image type or P/B-image type, and the supported image type can be determined based on the coding settings. Performing inter-frame prediction in the current image uses spatial correlation to generate prediction blocks. Performing inter-frame prediction in other images for the purpose of using temporal correlation is different, but the prediction method (e.g., reference image, motion vector, etc.) may be the same.

Here, it is assumed that P-image and B-image are image types capable of performing inter-frame prediction, but this can also be applied to various image types that are added or replaced. For example, a particular image type may not support intra-frame prediction, but may only support inter-frame prediction, or may only support inter-frame prediction in a predetermined direction (backward direction), or may only support inter-frame prediction in a predetermined direction.

The reference image construction unit can construct and manage reference images for encoding the current image using a reference image list. At least one reference image list can be constructed based on encoding settings (e.g., image type, prediction orientation, etc.), and prediction blocks can be generated from the reference images included in the reference image list.

In the case of unidirectional prediction, inter-frame prediction can be performed on at least one reference image included in reference image list 0 (L0) or reference image list 1 (L1). Additionally, in the case of bidirectional prediction, inter-frame prediction can be performed on at least one reference image included in a combined list (LC) generated by combining L0 and L1.

For example, one-way prediction can be classified as forward prediction (Pred_L0) using a forward reference image list (L0) and backward prediction (Pred_L1) using a backward reference image list (L1). Two-way prediction (Pred_BI) can use both a forward reference image list (L0) and a backward reference image list (L1).

Alternatively, two or more forward predictions performed by copying the forward reference image list (L0) to the backward reference image list (L1) can be included in the bidirectional prediction, and two or more backward predictions performed by copying the backward reference image list (L0) to the forward reference image list (L1) can also be included in the bidirectional prediction.

The prediction direction can be indicated by flag information indicating the corresponding direction (e.g., inter_pred_idc, assuming this value can be adjusted by predFlagL0, predFlagL1, and predFlagBI). predFlagL0 indicates whether forward prediction is performed, and predFlagL1 indicates whether backward prediction is performed. Bidirectional prediction can be indicated by whether the prediction is performed via predFlagBI or by activating both predFlagL0 and predFlagL1 simultaneously (e.g., when each flag is 1).

This invention primarily describes the case of omnidirectional prediction using a list of omnidirectional reference images, but the same or modified application can be applied to other situations.

Typically, the following approach can be used: determine the optimal reference image for the image to be encoded by the encoder, and explicitly send information about the reference image to the decoder. To this end, the reference image construction unit can manage the list of images referenced for inter-frame prediction of the current image, and can set rules for reference image management considering limited memory size.

The transmitted information can be defined as a Reference Image Set (RPS), where images selected in the RPS are classified as reference images and stored in memory (or a Data Point Block (DPB), while images not selected in the RPS are classified as non-reference images. Images may be deleted from memory after one hour. A predetermined number of images (e.g., 14, 15, 16, or more) can be stored in memory, and the size of the memory can be set according to the image level and resolution.

Figure 6 is an example diagram of constructing a list of reference pictures according to an embodiment of the present invention.

Referring to Figure 6, typically, reference images (T-1, T-2) existing before the current image can be assigned to L0 and managed, and reference images (T+1, T+2) existing after the current image can be assigned to L1 and managed. When constructing L0, if the number of reference images for L0 is insufficient, reference images from L1 can be assigned. Similarly, when constructing L1, if the number of reference images for L1 is insufficient, reference images from L0 can be assigned.

Furthermore, the current image can be included in at least one list of reference images. For example, L0 or L1 can include the current image, and L0 can be constructed by adding a reference image (or the current image) with a chronological order of T to a list of reference images preceding the current image, while L1 can be constructed by adding a reference image with a chronological order of T to a list of reference images following the current image.

The construction of the reference image list can be determined based on the encoding settings.

The current image may not be included in the reference image list, but may be managed through a separate memory separate from the reference image list, or the current image may be included in at least one reference image list and managed thereafter.

For example, this can be determined using a signal (curr_pic_ref_enabled_flag) that indicates whether the current image is included in the list of reference images. Here, the signal can be implicitly determined or explicitly generated.

Specifically, when the signal is deactivated (e.g., curr_pic_ref_enabled_flag = 0), the current image may not be included as a reference image in any list of reference images. When the signal is activated (e.g., curr_pic_ref_enabled_flag = 1), whether to include the current image in a predetermined list of reference images can be implicitly determined (e.g., added only to L0, added only to L1, or added to both L0 and L1) or the relevant signal can be explicitly generated and determined (e.g., curr_pic_ref_from_l0_flag, curr_pic_ref_from_l1_flag). This signal can be supported in units such as sequences, images, sub-images, slices, tiles, and blocks.

Here, the current image can be positioned either first or last in the list of reference images shown in Figure 6, and the arrangement order in the list can be determined according to encoding settings (e.g., image type information). For example, type I can be positioned first, and type P/B can be positioned last, but the invention is not limited thereto, and other modified examples are also possible.

Alternatively, a separate reference image store can be supported based on a signal (ibc_enabled_flag) indicating whether block matching (or template matching) is supported in the current image. Here, the signal can be implicitly determined information or explicitly generated information.

Specifically, when the signal is deactivated (e.g., ibc_enabled_flag = 0), this may mean that block matching is not supported in the current image, and when the signal is activated (e.g., ibc_enabled_flag = 1), block matching can be supported in the current image, and reference image memory for this purpose can be supported. In this example, it is assumed that additional memory is provided, but it is also possible to configure block matching to be supported directly from existing memory supported by the current image without providing additional memory.

The reference image construction unit may include a reference image interpolation unit, and whether to perform interpolation processing on fractional-unit pixels can be determined based on the interpolation precision of the inter-frame prediction. For example, when the interpolation precision is in integer units, reference image interpolation processing can be omitted, and when the interpolation precision is in fractional units, reference image interpolation processing can be performed.

The interpolation filters used in the reference image interpolation process can be implicitly determined based on the encoding settings, or explicitly determined from among multiple interpolation filters. Depending on the encoding settings, the configuration of multiple interpolation filters can support fixed candidates or adaptive candidates, and the number of candidates can be an integer of 2, 3, 4 or more. The explicitly determined unit can be determined from a sequence, image, sub-image, slice, tile, block, or chunk.

Here, the encoding settings can be determined based on image type, color components, target block state information (e.g., block size, shape, horizontal/vertical length ratio, etc.), and inter-frame prediction settings (e.g., motion information encoding mode, motion model selection information, motion vector precision selection information, reference image, reference orientation, etc.). Motion vector precision can mean the precision of the motion vectors (i.e., pmv + mvd), but it can be replaced by the precision of the motion vector prediction values (pmv) or the motion vector difference (mvd).

Here, the interpolation filter can have a filter length with k taps, and k can be an integer of 2, 3, 4, 5, 6, 7, 8, or more. The filter coefficients can be derived from equations with various coefficient characteristics, such as Wiener and Kalman filters. The filter information used for interpolation (e.g., filter coefficients, tap information, etc.) can be implicitly determined or derived, or the relevant information can be explicitly generated. In this case, the filter coefficients can be configured to include 0.

Interpolation filters can be applied to a predetermined pixel unit, and the predetermined pixel unit can be restricted to an integer or fractional unit, or can be applied to both integer and fractional units.

For example, an interpolation filter can be applied to k integer unit pixels that are adjacent to the interpolation target pixel (i.e., fractional unit pixels) in the horizontal or vertical direction.

Alternatively, the interpolation filter can be applied to p integer units and q fractional units (p+q=k) adjacent to the interpolation target pixel in the horizontal or vertical direction. In this case, the precision referred to as the fractional unit used for interpolation (e.g., in 1/4 units) can be expressed as the same as or lower than the interpolation target pixel (e.g., 2/4 -> 1/2).

When only one of the x-component and y-component of the interpolated target pixel lies within a fractional unit, interpolation can be performed based on k pixels adjacent in the direction of the fractional component (e.g., the x-axis is horizontal and the y-axis is vertical). When both the x-component and y-component of the interpolated target pixel lie within a fractional unit, a first interpolation can be performed based on x pixels adjacent in either the horizontal or vertical direction, and interpolation can be performed based on y pixels adjacent in the other direction. In this case, the case where x and y are the same as k will be described, but the case is not limited to this, and it is also possible for x and y to be different.

The interpolated target pixel can be obtained through interpolation precision (1/m), where m can be an integer of 1, 2, 4, 8, 16, 32, or higher. The interpolation precision can be implicitly determined based on encoding settings, or it can be explicitly generated based on relevant information. Encoding settings can be limited based on image type, color components, reference image, motion information encoding mode, motion model selection information, etc. The explicitly determined unit can be determined from a sequence, sub-image, slice, tile, or block.

Through the above processing, the interpolation filter settings for the target block can be obtained, and reference image interpolation can be performed based on them. Furthermore, detailed interpolation filter settings based on these settings can be obtained, and reference image interpolation can be performed based on them. That is, while it's possible to obtain the interpolation filter settings for the target block using a fixed candidate, it's also possible to use multiple candidates. In the example described later, it's assumed that the interpolation precision is 1/16 (e.g., 15 pixels to be interpolated).

As an example of setting up detailed interpolation filters, one of a plurality of candidates can be adaptively used for the first pixel unit at a predetermined position, and a predetermined interpolation filter can be used for the second pixel unit at the predetermined position.

The first pixel unit can be determined among all fractional units of pixels supported by interpolation precision (1/16 to 15/16 in this example), and the number of pixels included in the first pixel unit can be a, and a can be determined between 0, 1, 2, ..., (m-1).

The second pixel unit may include pixels excluded from the first pixel unit from the total number of pixels in the fractional units, and the number of pixels included in the second pixel unit can be obtained by subtracting the number of pixels in the first pixel unit from the total number of pixels to be interpolated. In this example, an example of dividing the pixel unit into two will be described, but the invention is not limited thereto and may divide it into three or more.

For example, the first pixel unit can be a unit represented by multiple units such as 1/2, 1/4, 1/8. For example, in the case of a 1/2 unit, it can be a pixel located at 8/16, and in the case of a 1/4 unit, it can be a pixel located at {4/16, 8/16, 12/16}, and in the case of a 1/8 unit, it can be a pixel located at {2/16, 4/16, 6/16, 8/16, 10/16, 12/16, 14/16}.

Detailed interpolation filter settings can be determined based on the coding settings, which can be limited by image type, color components, target block state information, inter-frame prediction settings, etc. Below are examples of detailed interpolation filter settings based on various coding elements. For ease of explanation, assume a is 0 and c is or greater (c is an integer greater than or equal to 1).

For example, in the case of color components (luminance, chrominance), a can be (1,0), (0,1), (1,1). Alternatively, in the case of motion information encoding mode (merging mode, competing mode), a can be (1,0), (0,1), (1,1), (1,3). Or, in the case of motion model selection information (translational motion, non-translational motion A, non-translational motion B), a can be (0,1,1), (1,0,0), (1,1,1), (1,3,7). Alternatively, in the case of motion vector precision (1/2, 1/4, 1/8), a can be (0,0,1), (0,1,0), (1,0,0), (0,1,1), (1,0,1), (1,1,0), (1,1,1). Alternatively, in the case of reference images (current image, other images), a can be (0,1), (1,0), (1,1).

As described above, interpolation can be performed by selecting one of several interpolation precisions, and when interpolation based on adaptive interpolation precision is supported (e.g., adaptive_ref_resolution_enabled_flag. If it is 0, a predetermined interpolation precision is used, and if it is 1, one of several interpolation precisions is used), precision selection information (e.g., ref_resolution_idx) can be generated.

Motion estimation and compensation can be performed based on interpolation accuracy, and the representation and storage units for motion vectors can also be determined based on interpolation accuracy.

For example, when the interpolation precision is 1/2 unit, motion estimation and compensation processing can be performed at 1/2 unit, and motion vectors can be represented at 1/2 unit and used in the encoding process. Furthermore, motion vectors can be stored at 1/2 unit and can be referenced in the encoding process of another block of motion information.

Alternatively, when the interpolation precision is 1/8, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/8 units and used in the encoding process and stored in 1/8 units.

Additionally, motion estimation and compensation processing, as well as motion vectors, can be performed, represented, and stored in units different from interpolation precision, such as integers, 1/2 units, and 1/4 units. These units can be adaptively determined based on inter-frame prediction methods/settings (e.g., motion estimation/compensation methods, motion model selection information, motion information coding modes, etc.).

As an example, assuming an interpolation precision of 1/8, in the case of a translational motion model, motion estimation and compensation can be performed in 1/4 units, and motion vectors can be represented in 1/4 units (assuming units in the encoding process in this example), and stored in 1/8 units. In the case of a non-translational motion model, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/4 units, and stored in 1/8 units.

For example, assuming an interpolation precision of 1/8 unit, in the case of block matching, motion estimation and compensation can be performed at 1/4 unit, and motion vectors can be represented at 1/4 unit and stored at 1/8 unit. In the case of template matching, motion estimation and compensation processing can be performed at 1/8 unit, and motion vectors can be represented at 1/8 unit and stored at 1/8 unit.

For example, assuming an interpolation precision of 1/16, in race mode, motion estimation and compensation processing can be performed in 1/4 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units. In merge mode, motion estimation and compensation processing can be performed in 1/8 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units. In skip mode, motion estimation and compensation can be performed in 1/16 units, and motion vectors can be represented in 1/4 units and stored in 1/16 units.

In summary, the units for motion estimation, motion compensation, motion vector representation, and storage can be adaptively determined based on the inter-frame prediction method or settings and the interpolation accuracy. Specifically, the units for motion estimation, motion compensation, and motion vector representation can be adaptively determined based on the inter-frame prediction method or settings, and the storage unit for motion vectors is typically determined based on the interpolation accuracy, but is not limited to this, and various modified examples are feasible. Furthermore, the examples above have illustrated an example based on one category (e.g., motion model selection information, motion estimation/compensation method, etc.), but settings can also be determined when two or more categories are mixed.

Furthermore, as mentioned above, the interpolation accuracy information can have a predetermined value or can be selected as one of multiple accuracies, and the reference image interpolation accuracy is determined based on the motion estimation and compensation settings supported by the inter-frame prediction method or settings. For example, when the translational motion model supports up to 1/8 unit and the non-translational motion model supports up to 1/16 unit, interpolation can be performed based on the accuracy unit of the non-translational motion model with the highest accuracy.

In other words, reference image interpolation can be performed based on settings for the accuracy information used to support it—such as translational motion model, non-translational motion model, competition mode, merging mode, and skip mode. In this case, the accuracy information can be determined implicitly or explicitly, and when the relevant information is explicitly generated, it can be included in units such as sequences, images, sub-images, slices, tiles, and blocks.

A motion estimation unit refers to the process of estimating (or searching) whether a target block has a high correlation with a predetermined block of a predetermined reference image. The size and shape (M×N) of the target block—on which prediction is performed—can be obtained from the block partitioning unit. As an example, the target block can be determined in the range of 4×4 to 128×128. Typically, inter-frame prediction can be performed on a block-by-block basis, but it can be performed on a block-by-block basis, such as a coded block or a transform block, depending on the block partitioning unit settings. Estimation can be performed within an estimable range of the reference region, and at least one motion estimation method can be used. In the motion estimation method, the estimation order and conditions can be defined for each pixel.

Motion estimation can be performed based on motion estimation methods. For example, in the case of block matching, the region to be compared for motion estimation processing can be the target block, and in the case of template matching, it can be a predetermined region (template) set around the target block. In the former case, the block with the highest relevance can be found within the estimable range of the target block and the reference region, and in the latter case, the region with the highest relevance can be found within the estimable range of the template defined according to the encoding settings and the reference region.

Motion estimation can be performed based on motion models. Additional motion models, besides translational motion models that only consider parallel motion, can be used to perform motion estimation and compensation. For example, motion models that consider motions such as rotation, perspective, zoom in/out, and parallel motion can be used to perform motion estimation and compensation. This can be supported to improve coding performance by generating prediction blocks that reflect the various types of motions occurring based on the regional features of the image.

Figure 7 is a conceptual diagram illustrating a non-translational motion model according to an embodiment of the present invention.

Referring to Figure 7, as an example of an affine model, an example is shown where motion information is represented based on motion vectors _V0 and _V1 at predetermined positions. Since motion can be represented based on multiple motion vectors, accurate motion estimation and compensation are possible.

As in the example above, inter-frame prediction is performed based on a predefined motion model, but inter-frame prediction based on an additional motion model can also be supported. Here, it is assumed that the predefined motion model is a translational motion model and the additional motion model is an affine model, but the invention is not limited to this, and various modifications are possible.

In the case of a translational motion model, motion information (assuming unidirectional prediction) can be represented based on a motion vector, and the control point (reference point) used to indicate the motion information is assumed to be the upper left coordinate, but is not limited to this.

In the case of a non-translational motion model, it can be represented as motion information for various configurations. In this example, it is assumed that the configuration is represented as additional information in a motion vector (relative to the top-left corner coordinates). Some motion estimation and compensation mentioned in the examples described later can be performed not on a block-by-block basis, but on a predetermined sub-block basis. In this case, the size and position of the predetermined sub-blocks can be determined based on each motion model.

Figure 8 is an example diagram illustrating motion estimation on a sub-block basis according to an embodiment of the present invention. Specifically, it illustrates motion estimation on a sub-block basis based on an affine model (two motion vectors).

In the case of a translational motion model, the motion vectors of the pixel units included in the target block can be the same. That is, the motion vectors can be applied to the pixel units together, and a single motion vector ( _V0 ) can be used to perform motion estimation and compensation.

In the case of a non-translational motion model (affine model), the motion vectors of the pixel units included in the target block may not be the same, and a separate motion vector may be required for each pixel. In this case, the motion vectors of the pixel units or sub-block units can be derived based on the motion vectors ( _V0 , _V1 ) at the positions of predetermined control points of the target block, and the derived motion vectors can be used to perform motion estimation and compensation.

For example, the motion vector of a sub-block or pixel unit within the target block (e.g., ( _Vx , _Vy )) can be derived from the equations _Vx = ( _V1x - _V0x )×x/M - ( _V1y - _V0y )×y/N + _V0x , _Vy = ( _V1y - _V0y )×x/M + ( _V1x - _V0x )×y/N + _V0y . In these equations, _V0 (in this example, ( _V0x , _V0y )) refers to the motion vector at the top left of the target block, and _V1 (in this example, ( _V1x , _V1y )) refers to the motion vector at the top right of the target block. Considering the complexity, motion estimation and compensation for a non-translational motion model can be performed on a sub-block basis.

Here, the size of the sub-block (M×N) can be determined according to the encoding settings and can have a fixed size or can be set to an adaptive size. M and N can be integers of 2, 4, 8, 16, or more, and M and N can be the same or different. The size of the sub-block can be explicitly generated in units such as sequences, pictures, sub-pictures, slices, tiles, and blocks. Alternatively, it can be implicitly determined by a mutual commitment between the encoder and decoder, or it can be determined by the encoding settings.

Here, the encoding settings can be defined by one or more elements, including state information, image type, color components, and inter-frame prediction settings for the target block (motion information encoding mode, reference image information, interpolation accuracy, motion model selection information, etc.).

In the examples above, the process of deriving the size of the sub-blocks based on a predetermined non-translational motion model and performing motion estimation and compensation accordingly has been described. As in the examples above, motion estimation and compensation can be performed on a sub-block or pixel-by-pixel basis according to the motion model, and a detailed description of this will be omitted.

The following are various examples of motion information constructed based on motion models.

For example, in the case of a motion model representing rotational motion, the translational motion of a block can be represented by a motion vector, and the rotational motion can be represented by rotation angle information. The rotation angle information can be measured relative to a predetermined position (e.g., the top-left coordinate) and can be represented as k candidates (k is an integer of 1, 2, 3 or more) with predetermined intervals (e.g., angle differences of 0 degrees, 11.25 degrees, 22.25 degrees, etc.) within a predetermined angle range (e.g., between -90 degrees and 90 degrees).

Here, the rotation angle information can be encoded as is during motion information encoding processing, or it can be encoded based on the motion information of adjacent blocks (e.g., motion vectors, rotation angle information) (e.g., prediction + difference information).

Alternatively, the translational motion of a block can be represented by a single motion vector, and the rotational motion of a block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vector can be encoded as is during motion information encoding processing, or encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational motion models or non-translational motion models) (e.g., prediction + difference information), or encoded based on another motion vector in a block representing rotational motion (e.g., prediction + difference information).

For example, in the case of a motion model representing sizing or scaling movements such as zooming in/out, the translational motion of a block can be represented by a single motion vector, and the scaling motion can be represented by scaling information. Scaling information can be represented as scaling information indicating expansion or reduction in the horizontal or vertical direction based on a predetermined position (e.g., top-left coordinate).

Here, scaling can be applied in at least one of the horizontal or vertical directions. Alternatively, it can support individual scaling information applied in the horizontal or vertical directions, or scaling information applied in a common direction. The width and height of the scaling block can be added to a predetermined position (top-left coordinate) to determine the location used for motion estimation and compensation.

Here, scaling information can be encoded as is during motion information encoding processing, or it can be encoded based on motion information of neighboring blocks (e.g., motion vectors, scaling information) (e.g., prediction + difference information).

Alternatively, the translation of a block can be represented by a single motion vector, and the size adjustment of the block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vectors can be encoded as is during motion information encoding processing, or they can be encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational motion models or non-translational motion models) (e.g., prediction + difference information), or they can be encoded based on predetermined coordinates within the block (e.g., lower right coordinates) (e.g., prediction + difference).

In the examples above, the representation of some motions has been described, and it can be represented as motion information for multiple motions.

For example, in the case of motion models representing various or complex motions, the translational motion of a block can be represented by a motion vector, the rotational motion can be represented by rotation angle information, and the sizing adjustment can be represented by scaling information. Since the description of each motion can be derived from the above examples, detailed descriptions will be omitted.

Alternatively, the translational motion of a block can be represented by a single motion vector, and other motions of the block can be represented by one or more additional motion vectors. In this case, the number of additional motion vectors can be an integer of 1, 2, or more, and the control points of the additional motion vectors can be determined from the upper right, lower left, and lower right coordinates, or other coordinates within the block can be set as control points.

Here, the additional motion vectors can be encoded as is during motion information encoding processing, or they can be encoded based on motion information of adjacent blocks (e.g., motion vectors based on translational or non-translational motion models) (e.g., prediction + difference information), or they can be encoded based on other motion vectors within blocks representing various motions (e.g., prediction + difference information).

The above description may pertain to affine models and will be based on the presence of one or two additional motion vectors. In summary, it is assumed that the number of motion vectors used according to the motion model can be 1, 2, or 3, and that a single motion model can be considered based on the number of motion vectors used to represent motion information. Furthermore, when only one motion vector exists, it is assumed to be a predefined motion model.

Multiple motion models can be supported for inter-frame prediction, and this can be determined by a signal indicating support for additional motion models (e.g., `adaptive_motion_mode_enabled_flag`). Here, a signal of 0 indicates support for predefined motion models, and a signal of 1 indicates support for multiple motion models. This signal can be generated in units such as sequences, images, sub-images, slices, tiles, blocks, chunks, etc., but if it is not possible to check them individually, the signal value can be assigned according to predefined settings. Alternatively, whether support is implicit can be determined based on encoding settings. Alternatively, the implicit or explicit case can be determined based on encoding settings. Here, encoding settings can be limited by one or more elements of image type, image class (e.g., a normal image if 0, a 360-degree image if 1), or color components.

Whether multiple motion models are supported can be determined through the process described above. The following assumes that two or more additional motion models are supported, and that multiple motion models are supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., but some configurations may be excluded. In the example described later, it is assumed that motion models A, B, and C are supported, where A is the fundamentally supported motion model, and B and C are additionally supported motion models.

Configuration information about supported motion models can be generated within the unit. That is, configurations for supported motion models such as {A, B}, {A, C}, and {A, B, C} are feasible.

For example, indices (0 to 2) can be assigned to candidates for the above configurations and selected. If index 2 is selected, a motion model configuration supporting {A, C} can be determined, and if index 3 is selected, a motion model configuration supporting {A, B, C} can be determined.

Alternatively, information indicating whether a predetermined motion model is supported can be supported individually. That is, a flag indicating whether B is supported or a flag indicating whether C is supported can be generated. If both flags are 0, then only A is supported. This example could be an instance where information indicating whether multiple motion models are supported is not generated and processed.

When candidate groups of supported motion models are configured as in the example above, the motion model of one of the candidate groups can be explicitly identified and used on a block-by-block basis, or it can be used implicitly.

Typically, a motion estimation unit can be a component present in the encoding device, but it can also be a component that can be included in the decoding device depending on the prediction method (e.g., template matching). For example, in the case of template matching, the decoder can obtain the motion information of the target block by performing motion estimation via a template adjacent to the target block. In this case, motion estimation related information (e.g., motion estimation range, motion estimation method <scan order>, etc.) can be implicitly determined or explicitly generated and included in units such as sequences, pictures, sub-pictures, slices, tiles, blocks, etc.

The motion compensation unit refers to the process of obtaining data of some blocks of a predetermined reference image determined by motion estimation processing as predicted blocks of a target block. Specifically, the predicted blocks of the target block can be generated from at least one region (or block) of at least one reference image based on motion information (e.g., reference image information, motion vector information, etc.) obtained through motion estimation processing.

Motion compensation can be performed based on the following motion compensation methods.

In the case of block matching, the predicted block of the target block can be compensated with data of the regions corresponding to the right M and down N relative to (P _{x + V x} , P _y + V _y ), which are explicitly obtained from the reference image and are obtained through the motion vector (V _x , V _y ) of the target block (M _{× N} ) and the top-left coordinates (P _x , P _y ) of the target block.

In the case of template matching, the predicted block of the target block can be compensated with data of the regions corresponding to the right M and down N relative to (P _{x + V x} , P _y + V _y ), which are implicitly obtained from the reference image and are obtained through the motion vector (V _x , V _y ) of the target block (M _{× N} ) and the top-left coordinates (P _x , P _y ) of the target block.

Alternatively, motion compensation can be performed based on the following motion model.

In the case of the translational motion model, the predicted block of the target block can be compensated with data of the regions corresponding to the _{right M} and down N relative to ( _Px + _Vx , _Py + _Vy ), which are explicitly obtained from the reference image and are obtained by a motion vector ( _Vx , _Vy ) of the target block (M×N ₎ and the upper left coordinates ( _Px , _{Py )} of the target block.

In the case of a non-translational motion model, the predicted sub-blocks can be compensated using data from regions M to the right and N downwards relative to ( _Pmx + _Vnx , _Pmy + _Vny ). This ( _Pmx + _Vnx , _Pmy + _Vny ) is obtained by implicitly acquiring the motion vectors (Vmx, Vny) of m×n sub-blocks via multiple motion vectors ( _V0x , _V0y ) and ( _V1x , _V1y ) of the target block (M×N) explicitly obtained from the reference image _{, along with the top-left coordinates (Pmx , Pny} ) of each sub-block. In other words, the predicted sub-blocks can be collected and compensated for by _the predicted sub-blocks of the target block.

The motion information determination unit can perform processing for selecting the optimal motion information for the target block. Typically, the optimal motion information can be determined based on coding cost and rate distortion techniques using block distortion (e.g., distortion of the target block and the reconstructed block, such as SAD (sum of absolute differences), SSD (sum of squared differences), etc.), where the rate distortion technique considers the bit amount generated based on the corresponding motion information. The predicted block generated based on the motion information determined through the above processing can be sent to the subtraction and addition units. Alternatively, it can be configured according to some prediction methods (e.g., template matching, etc.) that can be included in the decoding device, and in this case, it can be determined based on the block distortion.

In the motion information determination unit, settings related to inter-frame prediction can be considered, such as motion compensation methods and motion models. For example, when multiple motion compensation methods are supported, the motion compensation method selection information, the corresponding motion vectors, and reference image information can be the optimal motion information. Alternatively, when multiple motion models are supported, the motion model selection information, its corresponding motion vectors, and reference image information can be the optimal motion information.

The motion information encoding unit can encode the motion information of the target block obtained through motion information determination processing. In this case, the motion information can consist of information about the region and image referenced for the prediction of the target block. Specifically, it can consist of information about the reference image (e.g., reference image information) and information about the reference region (e.g., motion vector information).

Additionally, settings related to inter-frame prediction (or selection information, such as motion estimation/compensation methods, motion model selection information, etc.) can be included in the motion information of the target block. Information about the reference image and reference region (e.g., the number of motion vectors, etc.) can be configured based on settings related to inter-frame prediction.

Motion information can be encoded by configuring a reference image and information about the reference region into a combination, and the combination of the reference image and information about the reference region can be configured as a motion information encoding mode.

Here, information about the reference image and reference region can be obtained based on adjacent blocks or predefined information (e.g., images encoded before or after the current image, zero motion vectors, etc.).

Neighboring blocks can be classified into blocks <inter_blk_A> that belong to the same space as the target block and are closest to the target block, blocks <inter_blk_B> that belong to the same space and are far from the target block, and blocks <inter_blk_C> that belong to a different space from the target block. Information about the reference image and reference region can be obtained based on neighboring blocks (candidate blocks) that belong to at least one of the categories.

For example, the motion information of the target block can be encoded based on the motion information of the candidate block or the reference image information, and the motion information of the target block can also be encoded based on the information derived from the motion information of the candidate block or the reference image information (or information median, transformation processing, etc.). In other words, the motion information of the target block can be predicted from the candidate block, and information about the motion information can be encoded.

In this invention, the motion information of the target block can be encoded based on one or more motion information encoding modes. Here, the motion information encoding modes can be defined in various ways and can include one or more of skip modes, merge modes, and competition modes.

Based on the template matching (tmp) described above, it can be combined with motion information coding modes, or it can be supported as a standalone motion information coding mode, or it can be included in the detailed configuration of all or some motion information coding modes. This presupposes that template matching is supported at a higher unit (e.g., picture, sub-picture, slice, etc.), but the flags regarding support can be considered as part of the inter-frame prediction settings.

Based on the method described above for performing block matching (ibc) within the current image, it can be combined with motion information coding modes, or it can be supported as a standalone motion information coding mode, or it can be included in the detailed configuration of all or some motion information coding modes. This presupposes that block matching is supported in a higher unit within the current image, but the flag regarding support can be considered part of the inter-frame prediction settings.

Based on the aforementioned motion model (affine), motion information coding modes can be combined with other motion information coding modes, or they can be supported as standalone motion information coding modes, or they can be included in the detailed configuration of all or some motion information coding modes. This presupposes that higher units support non-translational motion models, but the flags regarding support can be considered part of the inter-frame prediction settings.

For example, single motion information encoding patterns such as temp_inter, temp_tmp, temp_ibc, and temp_affine can be supported. Alternatively, combined motion information encoding patterns such as temp_inter_tmp, temp_inter_ibc, temp_inter_affine, and temp_inter_tmp_ibc can be supported. Alternatively, the motion information prediction candidate group for constructing temp can include template-based candidates, candidates based on methods for performing block matching in the current image, and affine-based candidates.

Here, temp can refer to skip mode, merge mode, and comp mode. For example, it is the same as the motion information encoding modes such as skip_inter, skip_tmp, skip_ibc, and skip_affine in skip mode; merge_inter, merge_tmp, merge_ibc, and merge_affine in merge mode; and comp_inter, comp_tmp, comp_ibc, and comp_affine in comp mode.

When skip mode, merge mode, and competition mode are supported, and candidates considering the above factors are included in the motion information prediction candidate group for each mode, a mode can be selected by distinguishing between the flags for skip mode, merge mode, and competition mode. For example, a flag indicates whether skip mode is supported; if the value is 1, skip mode is selected. A flag indicates whether merge mode is supported; if the value is 1, merge mode is selected, and if the value is 0, competition mode can be selected. Additionally, candidates based on inter, tmp, ibc, and affine can be included in the motion information prediction candidate group for each mode.

Alternatively, when multiple motion information encoding modes are supported under a common mode, in addition to the flags used to select one of the skip mode, merge mode, and competition mode, additional flags can be supported to distinguish the detailed mode of the selected mode. For example, when the merge mode is selected, this means additionally supporting flags to select a detailed mode among merge_inter, merge_tmp, merge_ibc, merge_affine, etc.—which are detailed modes associated with the merge mode. Alternatively, flags indicating whether it is merge_inter can be supported, and if it is not merge_inter, additional flags for selection among merge_tmp, merge_ibc, merge_affine, etc. can be supported.

The encoding settings can support all or part of the motion information encoding mode candidates. Here, the encoding settings can be limited by more than one of the following elements: target block state information, image type, image category, color components, and inter-frame prediction support settings (e.g., whether template matching is supported in the current image, non-translation motion model support elements, etc., and whether block matching is supported).

For example, the supported motion information encoding patterns can be determined based on the block size. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H of the block's width (W) and height (H). With the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. With the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by either the first or second threshold size, or both.

In this scenario, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this case, a first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and a second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

For example, supported motion information encoding modes can be determined based on the image type. In this case, the image type can include at least one of skip mode, merge mode, and competition mode. In this case, a method for performing block matching (or template matching) in the current image, a single motion information encoding mode for an affine model (hereinafter referred to as an element) can be supported, or motion information encoding modes can be supported by combining two or more elements. Alternatively, elements other than the predetermined motion information encoding modes can be configured as motion information prediction candidate groups.

P/B image types may include at least one of skip mode, merge mode, and competition mode. In this case, methods for performing general inter-frame prediction, template matching, block matching on the current image can be supported, as well as a single motion information coding mode for an affine model (hereinafter referred to as an element), or motion information coding modes can be supported by combining two or more elements. Alternatively, elements other than the predetermined motion information coding mode can be configured as motion information prediction candidate groups.

Figure 9 is a flowchart illustrating the encoding of motion information according to an embodiment of the present invention.

Referring to Figure 9, the motion information encoding mode S900 of the target block can be checked.

Motion information coding modes can be defined by combining and setting predetermined information (e.g., motion information) used for inter-frame prediction. Predetermined information may include one or more of the following: predicted motion vector, motion vector difference, motion vector difference accuracy, reference image information, reference orientation, motion model information, information about the presence or absence of residual components, etc. Motion information coding modes may include at least one of skip mode, merge mode, and contention mode, and may include other additional modes.

The configuration and settings of explicitly generated information and implicitly determined information can be determined based on the motion information encoding pattern. In this case, in the explicit case, each piece of information can be generated individually or in combination (e.g., in the form of an index).

For example, in skip or merge modes, the predicted motion vector, reference image information, and/or reference orientation can be defined based on (a) predetermined index information. In this case, the index may include one or more candidates, and each candidate can be set based on the motion information of a predetermined block (e.g., the predicted motion vector, reference image information, and reference orientation of the corresponding block are composed of a combination). Additionally, motion vector differences (e.g., zero vectors) can be implicitly handled.

For example, in competition mode, the predicted motion vector, reference image information, and/or reference orientation can be defined based on (one or more) predetermined index information. In this case, the index can support each information segment individually, or a combination of two or more information segments can be supported. In this case, the index can include one or more candidates, and each candidate can be set based on the motion information of a predetermined block (e.g., predicted motion vector, etc.), or can be configured with predetermined values (e.g., the interval between current images is processed as 1, 2, 3, etc., and the reference orientation information is processed as L0, L1, etc.) (e.g., reference image information, reference orientation information, etc.). In addition, motion vector differences can be explicitly generated, and motion vector difference accuracy information can be additionally generated based on the motion vector difference (e.g., if it is not a zero vector).

As an example, in skip mode, information about the presence or absence of residual components can be implicitly processed (e.g., processed as <cbf_flag=0>), and the motion model can be implicitly processed as a predetermined value (e.g., supporting parallel motion models).

For example, in merge or competition modes, information about the presence or absence of residual components can be explicitly generated, and the motion model can be explicitly selected. In this case, information for classifying the motion model can be generated under a single motion information encoding mode, or separate motion information encoding modes can be supported.

In the above description, when supporting one index, index selection information may not be generated, while when supporting two or more indexes, index selection information may be generated.

In this invention, a candidate group comprising two or more indices is referred to as a motion information prediction candidate group. Furthermore, existing motion information encoding patterns can be modified or new motion information encoding patterns can be supported through various predetermined information and settings.

Referring to Figure 9, the predicted motion vector S910 of the target block can be obtained.

The predicted motion vector can be determined as a predetermined value, or a candidate set of predicted motion vectors can be constructed and selected from the candidate set. When the predicted motion vector can be determined as a predetermined value, it implies an implicit determination; and when a candidate set of predicted motion vectors can be constructed and selected from the candidate set, index information for selecting the predicted motion vector can be explicitly generated.

Here, the predetermined value can be set based on the motion vector of a block at a predetermined position (e.g., left, top, top left, top right, bottom left, etc.), or it can be set based on the motion vectors of two or more blocks, or it can be set to a default value (e.g., zero vector).

Depending on the motion information encoding mode, the predicted motion vectors can have the same or different candidate group construction settings. For example, skip mode/merge mode/competition mode can support a, b, and c prediction candidates respectively, and the number of each candidate can be the same or different. In this case, a, b, and c can include integers greater than or equal to 1, such as 2, 3, 5, 6, 7, etc. The order, settings, etc., of constructing candidate groups will be described through other implementations. In the examples described later, the construction of the skip mode candidate group will be described based on the same assumptions as the merge mode, but it is not limited to this, and some can have different constructions.

The predicted motion vectors obtained based on the index information can be used as is to reconstruct the motion vectors of the target block, or they can be adjusted based on the distance between reference images and the reference orientation.

For example, in competitive mode, reference image information can be generated separately. When the distance between the current image and the reference image of the target block differs from the distance between the image containing the predicted motion vector candidate block and the reference image of the corresponding block, adjustments can be made based on the distance between the current image and the reference image of the target block.

Furthermore, the number of motion vectors derived from the motion model can be different. That is, in addition to the motion vector at the upper left control point, motion vectors for the upper right and lower left control points can also be derived from the motion model.

Referring to Figure 9, the motion vector difference S920 of the target block can be reconstructed.

In skip mode and merge mode, the motion vector difference can be obtained as a zero vector value, and in competition mode, the difference information of each component can be reconstructed.

Motion vectors can be represented according to a predetermined precision (e.g., based on interpolation precision settings or motion model selection information settings). Alternatively, they can be represented based on one of several precisions, and predetermined information for this purpose can be generated. In this case, the predetermined information can relate to the selection of motion vector precision.

For example, when one component of the motion vector is 32/16, motion component data for 32 can be obtained based on an implicit precision setting in units of 1/16 pixels. Alternatively, the motion component can be converted to 2/1 based on an explicit precision setting in units of 2 pixels, and motion component data for 2 can be obtained.

The above configuration can be a method for accurately processing motion vectors with the same precision, or it can be obtained by adding the motion vector difference to a predicted motion vector obtained through the same precision transformation process.

Alternatively, the motion vector difference can be represented with a predetermined precision, or it can be represented with one of multiple precisions. In the case where the motion vector difference can be represented with one of multiple precisions, precision selection information can be generated.

Precision can include at least one of 1/16, 1/8, 1/4, 1/2, 1, 2, 4 pixel units, and the number of candidates can be an integer greater than or equal to 1, 2, 3, 4, 5. Here, the supported precision candidate group configuration (e.g., division by number, candidate, etc.) can be explicitly determined in units such as sequences, pictures, sub-pictures, slices, tiles, and blocks. Alternatively, it can be implicitly determined based on encoding settings, which can be defined by at least one or more elements of image type, reference image, color components, and motion model selection information.

The following describes a case where precision candidates are formed based on various coding elements and precision is described by assuming a difference in motion vectors, which can be similar to or applicable to motion vectors.

For example, in the case of a translational motion model, candidate groups such as {1/4, 1}, {1/4, 1/2}, {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} can be configured, and in the case of a non-translational motion model, precision candidate groups such as {1/16, 1/4, 1}, {1/16, 1/8, 1}, {1/16, 1, 4}, {1/16, 1/4, 1, 2}, and {1/16, 1/4, 1, 4} can be configured. This assumes that the minimum precision supported in the case of the translational motion model is 1/4 pixel unit, while the minimum precision supported in the case of the non-translational motion model is 1/16 pixel unit, and additional precision beyond the minimum precision can be included in the candidate groups.

Alternatively, when the reference image is a different image, candidate groups such as {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} can be configured. Additionally, when the reference image is the current image, candidate groups such as {1, 2}, {1, 4}, and {1, 2, 4} can be configured. In this case, the configuration for the reference image being the current image can be as follows: for block matching, interpolation is not performed in fractional units, and if interpolation is performed in fractional units, it can be configured to include fractional units in the precision candidates.

Alternatively, when the color component is a luminance component, candidate groups such as {1/4, 1/2, 1}, {1/4, 1, 4}, and {1/4, 1/2, 1, 4} are feasible. And when the color component is a chrominance component, candidate groups such as {1/8, 1/4}, {1/8, 1/2}, {1/8, 1}, {1/8, 1/4, 1/2}, {1/8, 1/2, 2}, and {1/8, 1/4, 1/2, 2} are feasible. In this case, when the color component is a chrominance component, candidate groups proportional to the luminance component can be formed based on the color component composition ratio (e.g., 4:2:0, 4:2:2, 4:4:4, etc.), or a single candidate group can be configured.

Although the above examples describe the case where there are multiple precision candidates for each coded element, it is possible for there to be a single candidate configuration (i.e., represented with a predetermined minimum precision).

In summary, the motion vector difference with minimum accuracy can be reconstructed based on the accuracy of the motion vector difference in the competition mode, and the motion vector difference with zero vector value can be obtained in the skip mode and the merge mode.

Here, the number of motion vector differences reconstructed from the motion model can be different. That is, in addition to the motion vector difference of the position of the upper left control point, the motion vector differences of the positions of the upper right and lower left control points can also be derived from the motion model.

Alternatively, a single motion vector difference precision can be applied to multiple motion vector differences, or a single motion vector difference precision can be applied to each motion vector difference, depending on the encoding settings.

Additionally, information about the accuracy of the motion vector difference can be generated when at least one motion vector difference is not zero, and otherwise it can be omitted, but is not limited to this.

Referring to Figure 9, the motion vector S930 of the target block can be reconstructed.

The motion vector of the target block can be reconstructed by obtaining the predicted motion vector and the difference between the motion vectors—which are obtained through processing prior to this step—and summing them. In this case, when reconstructing by selecting one of multiple accuracies, accuracy unification processing can be performed.

For example, a motion vector obtained by adding the predicted motion vector (related to the accuracy of the motion vector) and the motion vector difference can be reconstructed based on accuracy selection information. Alternatively, the motion vector difference (related to the accuracy of the differential motion vector) can be reconstructed based on accuracy selection information, and the motion vector can be obtained by adding the reconstructed motion vector difference to the predicted motion vector.

Referring to Figure 9, motion compensation S940 can be performed based on the motion information of the target block.

In addition to the predicted motion vector and motion vector difference mentioned above, motion information can also include reference image information, reference orientation, motion model information, etc. In skip mode and merge mode, some information (e.g., reference image information, reference orientation, etc.) can be implicitly determined, while in competition mode, relevant information can be explicitly processed.

The predicted block can be obtained by performing motion compensation based on the motion information obtained through the above processing.

Referring to Figure 9, decoding S950 can be performed on the residual components of the target block.

The target block can be reconstructed by adding the residual components obtained through the above processing to the predicted block. In this case, explicit or implicit processing can be performed on the information regarding the presence or absence of the residual components, depending on the motion information encoding pattern.

Figure 10 is a layout diagram of the target block and its adjacent blocks according to an embodiment of the present invention.

Referring to Figure 10, blocks <inter_blk_A> adjacent to the target block in the left, top, top left, top right, and bottom left directions, and blocks <inter_blk_B> adjacent to the block corresponding to the target block in the spatial Col_Pic that are different in time in the center, left, right, top, bottom, top left, top right, bottom left, and bottom right directions, can be configured as candidate blocks for predicting the motion information (e.g., motion vectors) of the target block.

In the case of inter_blk_A, the orientation of adjacent blocks can be determined based on the encoding order such as raster scan or z scan, and existing orientations can be removed, or adjacent blocks in the right, bottom, and lower right directions can additionally include candidate blocks.

Referring to Figure 10, Col_Pic can be an adjacent image before or after the current image (e.g., when the interval between images is 1), and the corresponding block in the image can have the same position as the target block.

Alternatively, Col_Pic can be an image in which the interval between images is predefined relative to the current image (e.g., the interval between images is z, where z is an integer of 1, 2, or 3), and the corresponding block can be set to a position moved from the predetermined coordinates (e.g., top left) of the target block by a predetermined disparity vector, and the disparity vector can be set to a predefined value.

Alternatively, Col_Pic can be set based on motion information of adjacent blocks (e.g., a reference image), and disparity vectors can be set based on motion information of adjacent blocks (e.g., motion vectors) to determine the position of the corresponding block.

In this case, k neighboring blocks can be referenced, and k can be an integer of 1, 2, or greater. If k is 2 or greater, Col_Pic and the disparity vector can be obtained based on calculations such as maximum, minimum, median, or weighted average of motion information of neighboring blocks (e.g., reference images or motion vectors). For example, the disparity vector can be set as the motion vector of the left or top block, and can be set as the median or average of the motion vectors of the left and bottom blocks.

The settings for time candidates can be determined based on motion information configuration settings. For example, the position of Col_Pic and the position of the corresponding block can be determined based on whether motion information is configured on a block-by-block or sub-block-by-sub-block basis. For this motion information, motion information configured on a block-by-block or sub-block-by-sub-block basis should be included in the motion information prediction candidate group. For example, when obtaining motion information on a sub-block basis, the block at the position where the predetermined disparity vector has been moved can be set as the position of the corresponding block.

The above examples illustrate the following: information about the location of the block corresponding to Col_Pic is implicitly determined, and related information can also be explicitly generated in units such as sequences, pictures, slices, tile groups, tiles, and blocks.

On the other hand, when partitioning units such as sub-images, slices, and tiles are not limited to a single image (e.g., the current image) and are shared between images, and if the position of the block corresponding to Col_Pic differs from the partitioning unit to which the target block belongs (i.e., if it extends or deviates from the boundary of the partitioning unit to which the target block belongs), it can be determined based on motion information of subsequent candidates according to a predetermined priority. Alternatively, it can be set to the same position in the image as the target block.

The motion information of spatially adjacent blocks and temporally adjacent blocks can be included in the motion information prediction candidate group of the target block, which is called the spatial candidate and the temporal candidate. A_a spatial candidates and b_b temporal candidates can be supported, and a and b can be integers from 1 to 6 or greater. In this case, a can be greater than or equal to b.

Based on the configuration of the predicted candidates according to the previous motion information, a fixed number of spatial and temporal candidates can be supported, or a variable number (e.g., including 0) can be supported.

Additionally, blocks <inter_blk_C> that are not adjacent to the target block can be considered during the candidate group construction.

For example, motion information of blocks located relative to a target block (e.g., top-left coordinates) at predetermined intervals can be the target to be included in the candidate group. The intervals can be p and q (depending on each component of the motion vector), where p and q can be integers of 2 or greater, such as 0, 2, 4, 8, 16, 32, and p and q can be the same or different. That is, assuming the top-left coordinates of the target block are (m, n), motion information of blocks at positions (m ± p, n ± q) can be included in the candidate group.

For example, motion information of blocks that have been encoded before the target block can be used as targets to be included in the candidate group. In this case, based on a predetermined encoding order relative to the target block (e.g., raster scan, Z-scan, etc.), a predetermined number of recently encoded blocks can be considered as candidates, and since encoding is performed using a first-in-first-out method such as FIFO, blocks with older encoding orders can be removed from the candidates.

Because the above example involves a case where motion information of a non-nearest neighbor block that has already been encoded relative to the target block is included in the candidate group, this is called a statistical candidate.

In this case, the reference block used to obtain statistical candidates can be set as the target block, but a higher block of the target block can be set (e.g., if the target block is in coded block <1>/predicted block <2>, the higher block is the maximum coded block <1>/coded block <2>, etc.). k statistical candidates can be supported, and k can be an integer from 1 to 6 or greater.

Motion information obtained by combining spatial, temporal, and statistical candidates can be included in a candidate group. For example, candidates can be obtained by applying a weighted average to each component of multiple (2, 3, or more) motion information previously included in the candidate group, and candidates can be obtained by processing each component of multiple motion information using median, maximum, minimum, etc. This is called a combined candidate, and r combined candidates can be supported, where r can be an integer of 1, 2, 3, or greater.

When the candidates used for combination do not have the same information about the reference image, they can be set as the reference image information for the combined candidates based on the reference image information about one of the candidates, or they can be set to a predefined value.

Based on the configuration of the predicted candidates according to the previous motion information, it can support a fixed number of statistical candidates or combined candidates, or it can support a variable number.

Additionally, default candidates with predetermined values such as (s, t) can be included in the candidate group, and the configuration of candidate prediction based on previous motion information can support a variable number (0, 1, or larger integers). In this case, s and t can be set to include 0, and can be set based on various block sizes (e.g., maximum encoding/prediction/transform block, minimum encoding/prediction/transform block horizontal or vertical length, etc.).

Depending on the motion information encoding mode, motion information prediction candidate groups can be constructed from the candidates, and other additional candidates can be included. Furthermore, the candidate group configuration settings can be the same or different depending on the motion information encoding mode.

For example, skip mode and merge mode can jointly construct candidate groups, while competition mode can construct a single candidate group.

Here, the candidate group configuration settings can be limited by the following: the category and position of the candidate block (e.g., determined from the position of the sub-block used to obtain motion information in the determined direction among the left/top/top-left/top-right/bottom-left directions), the number of candidates (e.g., total number, maximum number of each category), and the candidate construction method (e.g., priority of each category, priority within each category, etc.).

The number (total) of motion information prediction candidate groups can be k, and k can be an integer from 1 to 6 or greater. In this case, when the number of candidates is one, it may mean that no candidate group selection information is generated, and the motion information of the predefined candidate block is set as the predicted motion information. When there are two or more candidate groups, candidate group selection information can be generated.

The candidate block can be one or more of inter_blk_A, inter_blk_B, and inter_blk_C. In this case, inter_blk_A can be the default included category, and the other category can be an additional supported category, but is not limited to this.

The above description can be applied to the candidate construction for motion information prediction in translational motion models, and the same or similar candidate blocks can be used/referenced in the candidate group construction of non-translational motion models (affine models). On the other hand, the candidate group construction of affine models can differ from that of translational motion models because the motion features and the number of motion vectors are different.

For example, when the motion model of a candidate block is an affine model, the configuration of the motion vector set of the candidate block can be included in the candidate group as is. For example, when the top-left and top-right coordinates are used as control points, the motion vector set of the top-left and top-right coordinates of the candidate block can be included in the candidate group as a candidate.

Alternatively, when the motion model of the candidate block is a translational motion model, the combination of motion vectors of the candidate blocks set based on the positions of the control points can be constructed as a set and included in the candidate group. For example, when the upper-left, upper-right, and lower-left coordinates are used as control points, the motion vector of the upper-left control point can be predicted based on the left, upper, and upper-left motion vectors of the target block (e.g., in the case of a translational motion model), and the motion vector of the upper-right control point can be predicted based on the motion vectors of the upper and upper-right blocks of the target block (e.g., in the case of a translational motion model), and the motion vector of the lower-left control point can be predicted based on the left and right motion models of the target block. In the above examples, the case where the motion model of the candidate block set based on the positions of the control points is a translational motion model has been described, but even in the case of an affine model, the motion vectors of the control point positions can be obtained or derived only by acquiring or deriving them. That is, for the motion vectors of the upper-left, upper-right, and lower-left control points of the target block, the motion vectors can be obtained or derived from the upper-left, upper-right, lower-left, and lower-right control points of the candidate block.

In summary, when the motion model of the candidate block is an affine model, the motion vector set of the corresponding block can be included in the candidate group (A), and when the motion model of the candidate block is a translational motion model, the motion vector set obtained by combining the control points of the motion vectors of the predetermined control points of the target block can be included in the candidate group (B).

In this case, candidate groups can be constructed using either method A or method B, or both methods A and B can be used. Furthermore, method A can be used first for construction, and method B can be used subsequently, but this is not a limitation.

Figure 11 shows an example diagram of statistical candidates according to an embodiment of the present invention.

Referring to Figure 11, the blocks corresponding to A to L in Figure 11 refer to the blocks that have been encoded and are separated from the target block by a predetermined interval.

When candidate blocks for motion information of a target block are limited to blocks adjacent to the target block, they may not reflect various characteristics of the image. Therefore, blocks encoded before the current image can also be considered candidate blocks. This category is called inter_blk_B, and this is referred to as statistical candidate.

Since the number of candidate blocks included in the motion information prediction candidate group is limited, effective statistical candidate management may be required.

(1) A block at a predetermined position that is at a predetermined distance from the target block can be regarded as a candidate block.

As an example, by restricting the x-component of a predetermined coordinate relative to the target block (e.g., the top-left coordinate, etc.), blocks with predetermined intervals can be considered as candidate blocks, and examples such as (-4, 0), (-8, 0), (-16, 0), (-32, 0), etc. are feasible.

As an example, by restricting the y-component of a predetermined coordinate relative to the target block, blocks with a predetermined interval can be considered as candidate blocks, and examples such as (0, -4), (0, -8), (0, -16), (0, -32), etc. are feasible.

As an example, a block with predetermined intervals of x and y components (excluding 0) relative to predetermined coordinates of the target block can be considered a candidate block. Examples such as (-4, -4), (-4, -8), (-8, -4), (-16, 16), (16, -16), etc., are feasible. In this example, the x component may depend on the y component having a positive sign.

Candidate blocks that are considered as statistical candidates can be determined in a limited way based on the encoding settings. For an example described later, refer to Figure 11.

Candidate blocks can be categorized into statistical candidates based on whether they belong to a predetermined unit. Here, the predetermined unit can be determined based on the largest coded block, block, tile, slice, sub-image, image, etc.

For example, when selecting candidate blocks by restricting them to the largest coded block that is the same as the target block, A through C can be the target.

Alternatively, when selecting candidate blocks by restricting the blocks belonging to the largest coded block to which the target block belongs and the left largest coded block, A through C, H, and I can be targets.

Alternatively, when selecting candidate blocks by restricting the blocks belonging to the largest coded block to which the target block belongs and the largest coded block above, A through C, E, and F can be targets.

Alternatively, when selecting candidate blocks by restricting them to blocks belonging to the same slice as the target block, A through I can be the target.

Candidate blocks can be classified as statistical candidates based on whether they are located in a predetermined direction relative to the target block. Here, the predetermined direction can be determined from left, top, upper left, upper right, etc.

For example, when selecting candidate blocks by restricting blocks located in the left and top directions relative to the target block, B, C, F, I, and L can be targets.

Alternatively, when selecting candidate blocks by restricting them to blocks located in the left, top, upper left, or upper right directions relative to the target block, A through L can be the target.

As in the example above, even if candidate blocks considered as statistical candidates are selected, there can be a priority order for including them in the motion information prediction candidate group. That is, as many statistical candidates as there are supported can be selected according to priority.

Priority can be determined by a predetermined order or by various coding elements. Coding elements can be qualified based on the distance between the candidate block and the target block (e.g., whether it is a short distance, the distance between blocks can be checked based on the x and y components) and the relative direction of the candidate block to the target block (e.g., left, top, top-left, top-right direction, left->top->top-right->left, etc.).

(2) A block that has been encoded according to a predetermined encoding order relative to the target block can be regarded as a candidate block.

The following examples assume that raster scan (e.g., maximum coding block) and Z scan (e.g., coding block, prediction block, etc.) are followed, but it should be understood that the following descriptions may vary depending on the scan order.

In the case of example (1) above, priority can be supported for constructing candidate groups for motion information prediction, and similarly, predetermined priority can be supported in this embodiment. Priority can be determined by various coding elements, but for ease of description, it is assumed that priority is determined according to the coding order. In the examples described later, candidate blocks and priorities will be considered together.

Candidate blocks that are considered as statistical candidates can be determined in a limited way based on the encoding settings.

Candidate blocks can be categorized into statistical candidates based on whether they belong to a predetermined unit. Here, the predetermined unit can be determined based on the largest coded block, block, tile, slice, sub-image, image, etc.

For example, when a block belonging to the same image as the target block is selected as a candidate block, the candidate block and priority, such as J-D-E-F-G-K-L-H-I-A-B-C, can be the target.

Alternatively, when a block belonging to the same slice as the target block is selected as a candidate block, the candidate block and its priority, such as D-E-F-G-H-I-A-B-C, can be the target.

Candidate blocks can be classified as statistical candidates based on whether they are located in a predetermined direction relative to the target block. Here, the predetermined direction can be determined based on the left or top direction.

For example, when selecting candidate blocks by restricting blocks located to the left of the target block, the candidate blocks and priorities such as K-L-H-I-A-B-C can be the target.

Alternatively, when selecting candidate blocks by restricting blocks located in the direction above the target block, the candidate block and priority, such as E-F-A-B-C, can be the target.

The above description is based on the following assumptions: candidate blocks and priorities are combined, but not limited to this, and various candidate blocks and priorities can be set.

As mentioned above, the priorities included in the candidate group for motion information prediction can be supported by the descriptions in (1) and (2) above. One or more priorities can be supported, and a single priority can be supported for the entire candidate block. Alternatively, candidate blocks can be classified into two or more categories, and priorities based on the categories can be supported. In the former case, an example could be selecting k candidate blocks based on a single priority, while in the latter case, an example could be selecting p and q (p+q=k) candidate blocks based on each priority (e.g., two categories). In this case, categories can be classified based on predetermined encoding elements, and these encoding elements can include whether the block belongs to a predetermined unit, whether the candidate block is located in a predetermined direction relative to the target block, etc.

An example of selecting candidate blocks based on whether a candidate block belongs to a predetermined unit (e.g., a partitioning unit) described by (1) and (2) has been described. In addition to partitioning units, candidate blocks can be selected by restricting blocks within a predetermined range relative to the target block. For example, it can be identified by boundary points that define the ranges of (x1, y1), (x2, y2), (x3, y3), etc., and by the minimum and maximum values of the x or y components. Values such as x1 to y3 can be integers such as 0, 4, and 8 (based on absolute values).

Statistical candidates can be supported based on either (1) or (2) above, or they can be supported in a combination of (1) and (2). The above description has described how to select candidate blocks for statistical candidates, and the management and updating of statistical candidates and their inclusion in the motion information prediction candidate group will be described later.

The construction of motion information prediction candidate groups can usually be performed on a block-by-block basis, because the motion information of blocks adjacent to the target block may be the same or similar. That is, spatial or temporal candidates can be constructed based on the target block.

Furthermore, statistical candidates can be constructed based on a predetermined unit, since the candidate block is not adjacent to the target block. The predetermined unit can be a higher block that includes the target block.

For example, when the target block is a prediction block, statistical candidates can be constructed on a prediction block basis or on a coding block basis.

Alternatively, when the target block is a coded block, statistical candidates can be constructed on a coded block basis, or statistical candidates can be constructed on a predecessor block basis (e.g., depth information is at least one difference from the target block). In this case, the predecessor block can include the largest coded block, or it can be a unit obtained based on the largest coded block (e.g., an integer multiple of the largest coded block).

As in the examples above, statistical candidates can be constructed based on the target block or higher units, and in the examples described later, it is assumed that statistical candidates are constructed based on the target block.

Furthermore, memory initialization for statistical candidates can be performed based on predetermined units. The predetermined unit can be determined based on an image, sub-image, slice, tile, block, or chunk, and in the case of a chunk, it can be set based on the largest coded block. For example, memory initialization for statistical candidates can be performed based on an integer (1, 2, or greater) of the largest coded block, the row or column units of the largest coded block, etc.

The management and updating of statistical candidates are described below. A memory can be prepared for managing statistical candidates, and it can store motion information for up to k blocks. In this case, k can be an integer from 1 to 6 or larger. The motion information stored in the memory can be determined based on the motion vectors, reference images, and reference directions. Here, the number of motion vectors (e.g., 1 to 3, etc.) can be determined based on motion model selection information. For ease of explanation, the case where a block that has completed its encoding according to a predetermined encoding order based on the target block is considered a candidate block will be described later.

(Case 1) Motion information of blocks pre-coded according to the coding order can be included as candidates in the previous ranking. Additionally, when the maximum number is filled and updated by other candidates, candidates in the previous ranking can be removed, and the order can be shifted forward one by one. In the following example, x represents a blank space that has not yet been constructed.

(Example)

1 order - [a,x,x,x,x,x,x,x,x,x]

2 order - [a,b,x,x,x,x,x,x,x,x]

...

9 order - [a, b, c, d, e, f, g, h, i, x]

10. Sequence: [a, b, c, d, e, f, g, h, i, j] -> Add j. (Full)

11. Sequence - [b, c, d, e, f, g, h, i, j, k] -> Add k. Remove leading a. Shift sequence.

(Case 2) When duplicate motion information exists, remove pre-existing candidates and adjust the order of existing candidates for forwarding.

In this context, duplication implies identical motion information, which can be defined by the motion information encoding mode. As an example, in the merge mode, duplication can be determined based on motion vectors, a reference image, and a reference direction, while in the competition mode, duplication can be determined based on motion vectors and a reference image. In the competition mode, if the reference images for each candidate are identical, duplication can be determined by comparing motion vectors; and if the reference images for each candidate are different, duplication can be determined by comparing motion vectors scaled based on the reference images for each candidate, but this is not the only possibility.

(Example)

1. Sequence - [a, b, c, d, e, f, g, x, x, x]

2. The sequence - [a, b, c, d, e, f, g, d, x, x] -> d overlaps.

-[a, b, c, e, f, g, d, x, x, x] -> Remove the existing d, shift order

(Case 3) When there is duplicate motion information, the corresponding candidate can be marked and managed as a long-term candidate (long-term candidate) according to predetermined conditions.

A separate memory can be supported for long-term candidates, and additional information such as occurrence frequency, in addition to motion information, can be stored and updated. In the example below, the memory for long-term candidate storage can be represented as ().

(Example)

1. Sequence - [(), b, c, d, e, f, g, h, i, j, k]

2. Sequence - [(), b, c, d, e, f, g, h, i, j, k] -> e is the next sequence. Overlapping - [(e), b, c, d, f, g, h, i, j, k] -> Move to the front. Long-term marker.

3. Sequence - [(e), c, d, f, g, h, i, j, k, l] -> Add l. Remove b.

...

10. The sequence - [(e), l, m, n, o, p, q, r, s, t] -> l is the next sequence. Overlapping.

-[(e, l), m, n, o, p, q, r, s, t]->long-term marker

11. Order - [(e, l), m, n, o, p, q, r, s, t] -> l is the next order. 3. Overlap - [(l, e), m, n, o, p, q, r, s, t] -> Changes the long-term candidate order.

The above example illustrates the following situation: existing statistical candidates (short-term candidates) are managed in an integrated manner, but can be managed separately from short-term candidates, and the number of long-term candidates can be 0, 1, 2 or a larger integer.

Short-term candidates can be managed in a first-in, first-out (FIFO) manner based on encoding order, but long-term candidates can be managed based on frequency, considering the redundancy among short-term candidates. In the case of long-term candidates, the order of candidates can be changed according to frequency during memory updates, but this is not a limitation.

The above example could be a case where candidate blocks are classified into two or more categories and priority is supported based on that category. When constructing motion information prediction candidate groups to include statistical candidates, short-term and long-term candidates a and b can be constructed separately based on priority within each candidate. Here, a and b can be integers 0, 1, 2, or larger.

Cases 1 to 3 above are examples based on statistical candidates, and the configuration of (case 1 + case 2) or (case 1 + case 3) is feasible, and statistical candidates can be managed in various modified and added forms.

Figure 12 is a conceptual diagram of statistical candidates based on a non-translational motion model according to an embodiment of the present invention.

Statistical candidates can be used not only for translational motion models but also for supporting non-translational motion models. In the case of translational motion models, due to the high frequency of occurrence, there may be many blocks that can be referenced from spatial or temporal candidates, etc. However, in the case of non-translational motion models, because of the low frequency of occurrence, statistical candidates in addition to spatial or temporal candidates may be needed.

Referring to Figure 12, the motion information { _TV0 , _TV1 } of the target block is predicted based on the motion information { _PV0 , _PV1 } of the block <inter_blk_B> that belongs to the same space as the target block and is far from the target block (hmvp).

Compared to the statistical candidates for the translational motion model described above, the number of motion vectors in the motion information stored in memory can be increased. Therefore, the basic concept of statistical candidates can be the same as or similar to that of the translational motion model, and can be explained by different configurations of the number of motion vectors. The following will describe the differences based on the non-translational motion model. In the example described later, it is assumed that the affine model has three control points (e.g., v0, v1, and v2).

In the case of an affine model, the motion information stored in memory can be determined based on a reference image, a reference direction, and motion vectors v0, v1, and v2. Here, the stored motion vectors can be stored in various forms. For example, the motion vectors can be stored as is, or predetermined difference information can be stored.

Figure 13 is an example diagram of the construction of motion information for storing the position of each control point as a statistical candidate according to an embodiment of the present invention.

Referring to Figure 13, (a) the motion vectors v0, v1 and v2 of the candidate block are represented as v0, v1 and v2, while in Figure 13, (b) the motion vector v0 of the candidate block is represented as v0, and the motion vectors v1 and v2 are represented as v*1 and v*2, which is different from the motion vector v0.

In other words, the motion vector of each control point location can be stored as is, or the difference between the motion vector of another control point location and the original motion vector can be stored. This can be an example of a configuration considered in terms of memory management, and various modification examples are possible.

Whether statistical candidates are supported can be explicitly generated in units such as sequences, images, slices, tiles, and blocks, or implicitly determined based on encoding settings. Since encoding settings can be limited by the various encoding elements described above, a detailed description is omitted.

Here, the support of a statistical candidate can be determined based on the motion information encoding pattern. Alternatively, it can be determined based on the motion model selection information. For example, statistical candidates can be supported among merge_inter, merge_ibc, merge_affine, comp_inter, comp_ibc, and comp_affine.

If statistical candidates are supported for both translational and non-translational motion models, then memory for multiple statistical candidates can be supported.

Next, we will describe a method for constructing motion information prediction candidate groups based on motion information encoding patterns.

The candidate set for motion information prediction in the competitive mode (hereinafter referred to as the competitive mode candidate set) may include k candidates, and k may be an integer of 2, 3, 4 or greater. The competitive mode candidate set may include at least one of spatial candidates or temporal candidates.

Spatial candidates can be derived from at least one of the blocks adjacent to the reference block in the left, top, top-left, top-right, and bottom-left directions. Alternatively, at least one candidate can be derived from the left-adjacent block (left block, bottom-left block) and the top-adjacent block (top-left block, top block, top-right block), under the assumption of this setting, which will be described later.

There can be two or more priorities for constructing candidate groups. In the region adjacent to the left, the priority order can be set to bottom left-left, and in the region adjacent to the top, the priority order can be set to top right-top-top left.

In the examples above, spatial candidates can be derived solely from blocks whose reference images are identical to the target block's. Furthermore, spatial candidates can be derived by scaling the target block's reference image (hereinafter marked *). In this case, within regions adjacent to the left, the priority order can be set to left-bottom-left*-bottom-left* or left-bottom-left*-left*, and within regions adjacent to the top, the priority order can be set to top-right-top-top-left-top-right*-top*-top-left* or top-right-top-left-top-left*-top*-top*-top-right*.

Time candidates can be derived from at least one of the adjacent blocks such as center, left, right, top, bottom, top left, top right, bottom left, and bottom right, based on the block corresponding to the reference block. Priorities can be configured for candidate groups, and priorities such as center-bottom left-right-bottom-bottom left-center-top left can be set.

When the sum of the maximum allowed number of spatial candidates and the maximum allowed number of temporal candidates is less than the number of candidates in the competitive mode, temporal candidates can be included in the candidate group, regardless of the construction of the candidate group for spatial candidates.

Candidates can be included in a candidate group based on priority, availability of each candidate block, and the maximum number of time-limited candidates (the number of q, an integer between 1 and the number of contested mode candidates).

Here, when the maximum allowed number of spatial candidates is set equal to the number of merge pattern candidates, temporal candidates may not be included in the candidate group, and when the maximum allowed number of spatial candidates is not filled, temporal candidates may be included in the candidate group. This example assumes the latter case.

Here, the motion vector of the temporal candidate can be obtained based on the motion vector of the candidate block and the distance interval between the current image and the reference image of the target block, and the reference image of the temporal candidate can be obtained based on the distance interval between the current image and the reference image of the target block, or the reference image of the temporal candidate can be obtained as a predefined reference image (e.g., the reference image index is 0).

If the candidate group for the competitive mode is not filled with spatial candidates, temporal candidates, etc., the candidate group can be constructed by including the default candidate with the zero vector.

The competition mode focuses on the description of comp_inter, and in the case of comp_ibc or comp_affine, can construct candidate groups for similar or different candidates.

For example, in the case of comp_ibc, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. In this case, candidate groups can be constructed by prioritizing spatial candidates, and then candidate groups can be constructed in the order of statistical candidates - combined candidates - default candidates, etc., but the order is not limited to this.

Alternatively, in the case of comp_affine, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. Specifically, a set of motion vectors for a candidate (e.g., a block) can be constructed in candidate group <1>, or a candidate group <2> can be constructed where the motion vectors of candidates (e.g., two or more blocks) are combined based on control point locations. It is possible, but not limited to, to first include the candidates for <1> in the candidate group and then include the candidates for <2> in the candidate group.

Since a detailed description of the above construction of various competitive mode candidate groups can be obtained from comp_inter, a detailed description is omitted.

In the process of constructing a candidate group for a competitive mode, if there is duplicate motion information among the candidates initially included, the candidates initially included can be maintained, and candidates with the next priority can be included in the candidate group.

Here, before constructing candidate groups, the motion vectors of the candidate groups can be scaled based on the distance between the reference image of the target block and the current image. For example, when the distance between the reference image of the target block and the current image and the distance between the reference image of the candidate block and the image to which the candidate block belongs are the same, the motion vector can be included in the candidate group; otherwise, the motion vector scaled based on the distance between the reference image of the target block and the current image can be included in the candidate group.

In this context, redundancy can imply identical motion information, which can be defined by the motion information encoding pattern. In a contention-based mode, duplication can be determined based on motion vectors, a reference image, and a reference direction. For example, if at least one component of a motion vector is different, it can be determined that no duplication exists. Redundancy checks are typically performed when a new candidate is included in the candidate group, but this can be modified to omit the redundancy check.

The motion information prediction candidate set for the merging pattern (hereinafter referred to as the merging pattern candidate set) may include k candidates, and k may be an integer of 2, 3, 4, 5, 6 or greater. The merging pattern candidate set may include at least one of spatial candidates or temporal candidates.

Spatial candidates can be derived from at least one of the blocks adjacent to the reference block in the left, top, top-left, top-right, and bottom-left directions. Priorities can exist for constructing candidate groups, and priorities can be set such as left-top-bottom-top-right-top-left, left-top-top-right-bottom-left-top, and top-left-bottom-left-top-right.

Candidates can be included in a candidate group based on priority, availability of each candidate block (e.g., determined based on encoding mode, block position, etc.), and the maximum allowed number of spatial candidates (the number of p, an integer between 1 and the number of candidate groups for the merge mode). Candidates may not be included in the candidate group in the order tl-tr-bl-t3-13, depending on the maximum allowed number and availability. If the maximum allowed number is 4 and the availability of all candidate blocks is true, the motion information of tl may not be included in the candidate group; conversely, if the availability of some candidate blocks is false, the motion information of tl may be included in the candidate group.

Time candidates can be derived from at least one of the blocks adjacent to the center, left, right, top, bottom, top left, top right, bottom left, and bottom right, based on the block corresponding to the reference block. Priorities can exist for constructing candidate groups, and priorities such as center-bottom left-right-bottom, bottom left-center-top left, etc., can be set.

Candidates can be included in a candidate group based on priority, availability of each candidate block, and the maximum number of time candidates allowed (q, an integer between 1 and the number of merge pattern candidates).

Here, the motion vector of the temporal candidate can be obtained based on the motion vector of the candidate block, and the reference image of the temporal candidate can be obtained based on the reference image of the candidate block, or a predefined reference image can be obtained (e.g., reference image index 0).

For priorities included in the merge pattern candidate group, spatial candidate-temporal candidate can be set, or vice versa, and priorities that combine spatial and temporal candidates can be supported. In this example, spatial candidate-temporal candidate is assumed.

In addition, statistical candidates or combined candidates can also be included in the merged pattern candidate group. Statistical candidates and combined candidates can be constructed after spatial and temporal candidates, but are not limited to this, and various priorities can be set.

For statistical candidates, up to n motion information segments can be managed, and among these, z motion information segments can be included in the merged pattern candidate group as statistical candidates. z can vary depending on the candidate construction already included in the merged pattern candidate group, and can be an integer of 0, 1, 2 or greater, and can be less than or equal to n.

A combined candidate can be derived by merging n candidates already included in the combined pattern candidate group, where n can be an integer of 2, 3, 4, or greater. The number of combined candidates (n) can be information explicitly generated in units such as sequences, pictures, sub-pictures, slices, tiles, blocks, or chunks. Alternatively, it can be implicitly determined based on encoding settings. In this case, the encoding settings can be limited based on one or more factors such as the size, shape, position, image type, and color components of the reference block.

Furthermore, the number of combined candidates can be determined based on the number of unfilled candidates in the merged candidate group. In this case, the number of unfilled candidates in the merged candidate group can be the difference between the number of merged candidate groups and the number of already filled candidates. That is, if the merged candidate group construction is complete, combined candidates may not be added. If the merged candidate group construction is incomplete, combined candidates may be added, but no combined candidates are added when the number of filled candidates in the merged candidate group is less than or equal to one.

If the candidate group for the merge pattern is not filled with spatial candidates, temporal candidates, statistical candidates, or combined candidates, the candidate group can be constructed by including the default candidate with a zero vector.

The merge mode focuses on the description of merge_inter, and in the case of merge_ibc or merge_affine, it can construct candidate groups for similar or different candidates.

For example, in the case of merge_ibc, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, statistical candidates, combined candidates, and default candidates. In this case, candidate groups can be constructed by prioritizing spatial candidates, and then candidate groups can be constructed in the order of statistical candidates - combined candidates - default candidates, etc., but the order is not limited to this.

Alternatively, in the case of merge_affine, candidate groups can be constructed based on predetermined candidates selected from spatial candidates, temporal candidates, statistical candidates, combined candidates, and default candidates. Specifically, a set of motion vectors for a candidate (e.g., a block) can be constructed in candidate group <1>, or a candidate group <2> can be constructed where the motion vectors of candidates (e.g., two or more blocks) are combined based on control point locations. It is possible, but not limited to, to first include the candidates for <1> in the candidate group and then include the candidates for <2>.

Since a detailed description of the above construction of various merge pattern candidate groups can be derived from merge_inter, a detailed description will be omitted.

In the process of constructing a candidate group for merging patterns, if there is overlapping motion information among the first included candidates, the first included candidates can be maintained, and candidates with the next priority can be included in the candidate group.

In this context, redundancy implies identical motion information, which can be defined by the motion information encoding pattern. In the case of the merging pattern, overlap can be determined based on motion vectors, a reference image, and a reference direction. For example, if at least one motion vector component is different, it can be determined that no overlap exists. Redundancy checks are typically performed when new candidates are included in the candidate group, but this can be modified to omit them.

Figure 14 is a flowchart illustrating motion information encoding according to an embodiment of the present invention.

A candidate list of motion vector predictions for the target block can be generated S1400. The aforementioned competition mode candidate group may mean a candidate list of motion vectors, and its detailed description will be omitted.

The motion vector difference S1410 of the target block can be reconstructed. The difference between the x and y components of the motion vector can be reconstructed separately, and the difference of each component can have a value of 0 or greater.

The prediction candidate index for the target block can be selected from the motion vector prediction candidate list S1420. The predicted motion vector value for the target block can be derived based on the motion vector obtained from the candidate index in the candidate list. If a predetermined motion vector prediction value can be obtained, the process of selecting the prediction candidate index and index information can be omitted.

The accuracy information S1430 for the motion vector difference can be obtained. This can be the accuracy information typically applied to the x and y components of the motion vector, or the accuracy information applied to each component. If the motion vector difference is 0, the accuracy information can be omitted, and this processing can also be omitted.

An adjustment offset S1440 for the motion vector prediction value can be derived. The offset can be a value added to or subtracted from the x or y component of the motion vector prediction value. The offset can support only one of the x and y components, or it can support both x and y components.

Assuming the predicted motion vector value is (pmv_x, pmv_y), and the offsets are adjusted to offset_x and offset_y, then the predicted motion vector value can be adjusted (or obtained) to (pmv_x + offset_x, pmv_y + offset_y).

Here, the absolute values of offset_x and offset_y can be integers of 0, 1, 2, or larger, and can have values (+1, -1, +2, -2, etc.), where sign information is considered together. Furthermore, offset_x and offset_y can be determined based on a predetermined precision. The predetermined precision can be determined in units of 1/16, 1/8, 1/4, 1/2, and 1 pixel, and can be based on interpolation precision, motion vector precision, etc.

For example, if the interpolation precision is in units of 1/4 pixel, then the interpolation precision can be 0, 1/4, -1/4, 2/4, -2/4, 3/4, or -3/4, which are combinations of absolute value and sign information.

Here, offset_x and offset_y can be backed by a and b respectively, and a and b can be integers of 0, 1, 2 or greater. a and b can have fixed values or variable values. Furthermore, a and b can have the same or different values.

Whether adjustment offsets for motion vector prediction values are supported can be explicitly supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., or implicitly determined based on encoding settings. Additionally, the settings for adjustment offsets (e.g., the range and number of values) can be determined based on encoding settings.

The encoding settings can be determined by taking into account at least one of the encoding elements such as image type, color components, target block state information, motion model selection information (e.g., whether it is a translational motion model), reference image (e.g., whether it is the current image), and motion vector precision difference selection information (e.g., whether it is a predetermined unit among 1/4, 1/2, 1, 2, and 4 units).

For example, the size of the block can be used to determine whether offset and offset settings are supported. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H, which have the width (W) and height (H) of the block. In the case of the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. In the case of the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by one of the first threshold size or the second threshold size, or both.

In this case, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this scenario, the first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and the second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

Furthermore, the adjustment offset can be applied to all candidates included in the motion information prediction candidate group, or it can be applied to only some candidates. In the example described later, it is assumed that all candidates included in the candidate group can be applied, but the candidates to which the adjustment offset is applied can be selected from 0, 1, 2 to the maximum number of candidates.

If offset adjustment is not supported, this processing and offset adjustment information can be omitted.

The motion vector of the target block can be reconstructed by adding the predicted motion vector value and the motion vector difference in S1450. In this case, the motion vector accuracy can be prioritized to unify the processing of the predicted motion vector value or the motion vector difference, and the aforementioned motion vector scaling process can be performed first or during this process.

The configuration and order are not limited to this and can be changed in various ways.

Figure 15 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention.

For ease of explanation, two motion vector prediction candidates are supported, and it is assumed that only one component (x or y component) is compared. Furthermore, it is assumed that the interpolation precision or motion vector precision is 1/4 pixel unit. Additionally, it is assumed that 1/4, 1, and 4 pixel units are supported for the motion vector difference precision (e.g., if 1/4, it is assumed to be binaryized as <0>, if 1, it is assumed to be binaryized as <10>, and if 4, it is assumed to be binaryized as <11>). Furthermore, it is assumed that sign information is omitted for the motion vector difference, and the motion vector difference is processed through univariate binarization (e.g., 0: <0>, 1: <10>, 2: <110>, etc.).

Referring to Figure 15, the value of the actual motion vector (X) is 2, the value of candidate 1 (A) is 0, and the value of candidate 2 (B) is 1/4.

(In cases where differential motion vector precision is not supported)

Since the distance (da) between A and X is 8/4 (9 bits) and the distance (db) between B and X is 7/4 (8 bits), B can be selected as the prediction candidate in terms of bit generation.

(When motion vector difference accuracy is supported)

Since da is 8/4, it can generate 1 pixel unit precision (2 bits) and 2/1 difference information (3 bits), thus generating a total of 5 bits. On the other hand, since db is 7/4, it can generate 1/4 pixel unit precision (1 bit) and 7/4 difference information (8 bits), thus generating a total of 9 bits. In terms of bit generation, the prediction candidate can be chosen as A.

As in the example above, if motion vector difference accuracy is not supported, it may be advantageous to select candidates with short distance intervals from the motion vectors of the target block as prediction candidates. If it is supported, then in addition to the distance interval from the motion vectors of the target block, the amount of information generated based on the accuracy information may be important in selecting prediction candidates.

Figure 16 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention. In the following, it is assumed that motion vector difference accuracy is supported.

Since da is 33/4, it can generate 1/4 pixel unit precision (1 bit) and 33/4 difference information (34 bits), thus generating a total of 35 bits. On the other hand, since db is 21/4, it can generate 1/4 pixel unit precision (1 bit) and 21/4 difference information (22 bits), thus generating a total of 23 bits. Regarding bit generation, prediction candidate B can be selected.

Figure 17 is an example diagram of motion vector prediction candidates and motion vectors for a target block according to an embodiment of the present invention. In the following, it is assumed that motion vector difference accuracy is supported and that offset information is adjusted.

In this example, it is assumed that the adjustment offset has 0 and +1 candidates, and a flag (1 bit) indicating whether the adjustment offset has been applied and offset selection information (1 bit) are generated.

Referring to Figure 17, A1 and B1 can be motion vector predictions obtained based on the prediction candidate indices, while A2 and B2 can be new motion vector predictions obtained by modifying A1 and B1 with adjusted offsets. Assume that the distances between A1, A2, B1, B2 and X are da1, da2, db1, and db2, respectively.

(1) Since da1 is 33/4, it can generate 1/4 pixel unit precision (1 bit), offset application flag (1 bit), offset selection information (1 bit) and 33/4 difference information (34 bits), so a total of 37 bits can be generated.

(2) Since da2 is 32/4, it can generate 4-pixel unit precision (2 bits), offset application flag (1 bit), offset selection information (1 bit) and 2/1 difference information (3 bits), so a total of 7 bits can be generated.

(3) Since db1 is 21/4, it can generate 1/4 pixel unit precision (1 bit), offset application flag (1 bit), offset selection information (1 bit) and 21/4 difference information (22 bits), so a total of 25 bits can be generated.

(4) Since db2 is 20/4, it can generate 1 pixel unit precision (2 bits), offset application flag (1 bit), offset selection information (1 bit) and 5/1 difference information (6 bits), so a total of 10 bits can be generated.

For in-situ quantity generation, the prediction candidate can be selected as A, and the offset selection information can be selected as index 1 (+1 in this example).

In the above example, when deriving the motion vector difference based on the existing motion vector prediction values, there may be a situation where many bits are generated due to the small difference between the vector values. This problem can be solved by adjusting the motion vector prediction values.

Predefined flags can be supported to apply adjustment offsets to motion vector predictions. Predefined flags can be configured via offset application flags, offset selection information, etc.

(If only offset application flags are supported)

When the offset application flag is 0, the offset is not applied to the motion vector prediction value, and when the offset application flag is 1, the predetermined offset can be added to or subtracted from the motion vector prediction value.

(If only offset selection information is supported)

The offset set based on offset selection information can be added to or subtracted from the motion vector prediction value.

(If offset application flags and offset selection information are supported)

When the offset application flag is 0, the offset is not applied to the motion vector prediction value, and when the offset application flag is 1, the offset set based on the offset selection information can be added to or subtracted from the motion vector prediction value. This setting is assumed in the example described later.

Meanwhile, offset application flags and offset selection information can be used as information that is unnecessarily generated in some cases. That is, even if the offset is not applied, the offset-related information may be quite inefficient if the information is already zero or the amount of information is reduced based on the maximum precision information. Therefore, it is necessary to support settings that implicitly generate information according to predetermined conditions, rather than always explicitly generating offset-related information.

Next, we assume a motion-related coding sequence (motion vector difference reconstruction -> motion vector difference precision acquisition). In this example, we assume that motion vector difference precision is supported when at least one of the x and y components is not zero.

When the motion vector difference is not 0, one of the following pixel units can be selected: {1/4, 1/2, 1, 4} for the motion vector difference precision.

If the selected information belongs to a predetermined category, information about the adjustment offset can be implicitly omitted; otherwise, information about the adjustment offset can be explicitly generated.

Here, the category can include one of the differential motion vector precision candidates, which can be configured with various categories such as {1/4}, {1/4, 1/2}, {1/4, 1/2, 1}. The minimum precision can also be included in the category.

For example, if the motion vector difference precision is 1/4 pixel (e.g., minimum precision), the offset application flag is implicitly set to 0 (i.e., not applied), and if the motion vector difference precision is not 1/4 pixel, the offset application flag is explicitly generated, and if the offset application flag is 1 (i.e., offset), offset selection information can be generated.

Alternatively, when the motion vector difference precision is 4 pixels (e.g., maximum precision), an offset application flag can be explicitly generated, and when the offset application flag is 1, offset selection information can be generated. Alternatively, when the motion vector difference precision is not 4 pixels, the offset information can be implicitly set to 0.

The above examples assume that offset-related information is implicitly omitted when the motion vector difference indicates minimum precision, and offset-related information is explicitly generated when the motion vector difference indicates maximum precision, but are not limited to this.

Figure 18 is an example diagram illustrating the arrangement of multiple motion vector prediction values according to an embodiment of the present invention.

The above example has described the part about redundancy checks when constructing candidate groups for motion information prediction. Here, redundancy means that the motion information is the same, and as described above, if at least one component of the motion vector is different, it can be determined that there is no redundancy.

Through redundancy checks, multiple candidates for motion vector predictions can be made to not overlap. However, when multiple candidate component elements are very similar (i.e., each candidate's x or y component exists within a predetermined range, and the width or height of that predetermined range is an integer of 1, 2, or greater; or the range can be set based on offset information), motion vector predictions and offset-modified motion vector predictions can overlap. Various settings are feasible for this purpose.

As an example (C1), in the step of constructing a candidate group for motion information prediction, a new candidate can be included in the candidate group if it does not overlap with already included candidates. That is, if there is no overlap only by comparing with the same candidate as previously described, it can be a configuration that can be included in the candidate group.

As an example (C2), a new candidate can be included in a candidate group if it does not overlap with a predetermined number of candidates (group_A) obtained by adding an offset to it, and with a predetermined number of overlaps between already included candidates and candidates (group_B) obtained by adding an offset to it. The predetermined number can be an integer of 0, 1, or greater. If the predetermined number is 0, even if one overlaps, the corresponding new candidate may not be included in the candidate group. Alternatively, (C3), a predetermined offset (a concept distinct from adjusting the offset) can be added to the new candidate to be included in the candidate group, and this offset can be a value that allows group_A to be configured not to overlap with group_B.

Referring to Figure 18, it can be the case that multiple motion vectors belonging to categories A, B, C, and D (in this example, AX, BX, CX, and DX. X is 1 and 2. For example, A1 is a candidate included in the candidate group preceding A2) satisfy the non-overlapping condition (if any component of the candidate motion vector is different).

In this example, it is assumed that -1, 0, and 1 are supported for the x and y components, respectively, and the offset-modified motion vector predictions can be represented as * in the figure. Furthermore, dashed lines (rectangles) represent ranges (e.g., group_A, group_B, etc.) that can be obtained by adding an offset to a predetermined motion vector prediction.

In the case of category A, it can correspond to the case where A1 and A2 do not overlap and group_A1 and group_A2 do not overlap.

In the case of categories B, C, and D, it can correspond to the situation where B1/C1/D1 and B2/C2/D2 do not overlap, but group_B1/C1/D1 and group_B2/C2/D2 partially overlap.

Here, in the case of category B, it could be an example of constructing candidate groups without taking any special actions (C1).

Here, in the case of category C, the following configuration can be used: In this configuration, C2 is determined to be redundant in the redundancy check step, and C3, which has the next higher priority, is included in the candidate group (C2).

Here, in the case of category D, it can be the following: D2 is determined to be redundant in the redundancy check step, and D2 is modified so that group_D2 does not overlap with group_D1 (i.e., D3. D3 is not a motion vector present in the candidate group construction priority) (C3).

Among various categories, it can be applied to the setting of constructing candidate groups of prediction patterns, or various methods other than those mentioned above can be applied.

Figure 19 is a flowchart illustrating motion information encoding in merging mode according to an embodiment of the present invention.

A candidate list S1900 for predicting motion information of the target block can be generated. The above-mentioned candidate group for merging modes may refer to the candidate list for predicting motion information, and its detailed description will be omitted.

A prediction candidate index S1910 for the target block can be selected from the motion information prediction candidate list. The motion vector prediction value of the target block can be derived based on the motion information obtained from the candidate indices in the candidate list. If a predetermined motion information prediction value can be obtained, the process of selecting the prediction candidate index and index information can be omitted.

An adjustment offset S1920 for the motion vector prediction value can be derived. The offset can be a value added to or subtracted from the x or y component of the motion vector prediction value. The offset can support only one of the x or y components, or it can support both x and y components.

Since the adjustment offset in this process can be the same as or similar to the adjustment offsets above, a detailed description is omitted, and a portion of the differences will be described later.

Assuming the predicted motion vector is (pmv_x, pmv_y), and the offsets are adjusted to offset_x and offset_y, the predicted motion vector can be adjusted (or obtained) to (pmv_x + offset_x, pmv_y + offset_y).

Here, the absolute values of offset_x and offset_y can be integers such as 0, 1, 2, 4, 8, 16, 32, 64, 128, etc., and can have sign information considered together. Furthermore, offset_x and offset_y can be determined based on a predetermined precision. The predetermined precision can be determined in units of 1/16, 1/8, 1/4, 1/2, or 1 pixel.

For example, if the motion vector precision is 1/4 pixel unit, then the motion vector precision can be 0, 1/4, -1/4, 1/2, -1/2, 1, -1, 2, -2, etc., which combine absolute value and sign information.

Here, offset_x and offset_y can be resources a and b, respectively, and a and b can be integers such as 0, 1, 2, 4, 8, 16, 32, etc. a and b can have fixed values or variable values. Furthermore, a and b can have the same or different values.

Whether adjustment offsets for motion vector prediction values are supported can be explicitly supported in units such as sequences, images, sub-images, slices, tiles, blocks, etc., or implicitly determined based on encoding settings. Additionally, the settings for adjustment offsets (e.g., the range and number of values) can be determined based on encoding settings.

The encoding settings can be determined by taking into account at least one of the encoding elements such as image type, color components, target block state information, motion model selection information (e.g., whether it is a translational motion model), and reference image (e.g., whether it is the current image).

For example, the size of the block can be used to determine whether offset and offset settings are supported. In this case, the block size can be determined by a first threshold size (minimum) or a second threshold size (maximum), and each threshold size can be represented as W, H, W x H, and W*H, which have the width (W) and height (H) of the block. In the case of the first threshold size, W and H can be integers of 4, 8, 16, or greater, and W*H can be integers of 16, 32, 64, or greater. In the case of the second threshold size, W and H can be integers of 16, 32, 64, or greater, and W*H can be integers of 64, 128, 256, or greater. This range can be determined by one of the first threshold size or the second threshold size, or both.

In this case, the threshold size can be fixed or adaptive based on the image (e.g., image type, etc.). In this scenario, the first threshold size can be set based on the sizes of the minimum coding block, minimum prediction block, and minimum transform block, and the second threshold size can be set based on the sizes of the maximum coding block, maximum prediction block, and maximum transform block.

Furthermore, the adjustment offset can be applied to all candidates included in the motion information prediction candidate group, or it can be applied to only some candidates. In the example described later, it is assumed that all candidates included in the candidate group can be applied, but the candidates to which the adjustment offset is applied can be selected from a number between 0, 1, or 2 and the maximum number of candidates.

Predefined flags can be used to apply adjustment offsets to motion vector prediction values. These flags can be configured via offset application flags, absolute offset value information, offset sign information, and more.

If offset adjustment is not supported, this processing and offset adjustment information can be omitted.

The motion vector S1930 of the target block can be reconstructed using the predicted motion vector values. Motion information other than the motion vector (e.g., reference image, reference direction, etc.) can be obtained based on the predicted candidate index.

The configuration and order are not limited to this and can be changed in various ways. The background description for supporting offset adjustment in merge mode has been described above with various examples of competition mode, and therefore its detailed description will be omitted.

The methods of this disclosure can be implemented as program instructions executable by various computer devices and stored in a computer-readable medium. The computer-readable medium may include program instructions, data files, and data structures, individually or in combination. The program instructions recorded on the computer-readable medium may be specifically designed for this disclosure or are known to and therefore usable by one of skill in the art of computer software.

The computer-readable medium may include hardware devices particularly suitable for storing and executing program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, etc. The program instructions may include machine language code generated by a compiler or high-level language code that can be executed by an interpreter in a computer. The aforementioned hardware devices may be configured to operate as one or more software modules to perform operations according to this disclosure, and vice versa.

Furthermore, the above-described methods or devices can be implemented by combining or separating all or part of their configurations or functions.

Although the present disclosure has been described above with reference to preferred embodiments thereof, those skilled in the art will understand that various modifications and variations may be made to the present disclosure without departing from the scope and spirit of the present disclosure.

Industrial applicability

This invention can be used to encode/decode video signals.

Claims (7)

1. An image decoding method performed by an image decoding device, the method comprising:

Construct a list of candidate blocks to merge;

Determine whether the predicted motion vector values of the target blocks from the merged candidate list can be adjusted by motion vector offset;

In response to determining that the motion vector prediction value needs to be adjusted, the motion vector prediction value of the target block is selected from the merge candidate list based on merge candidate information indicating one of the two merge candidates included in the merge candidate list, wherein only two merge candidates in the merge candidate list can be selected as the motion vector prediction value and adjusted by the motion vector offset, and any remaining merge candidates in the merge candidate list that are not one of the two merge candidates cannot be selected;

Obtain motion vector offset information, which includes first information about the sign of the motion vector offset and second information about the absolute value of the motion vector offset;

Determine the motion vector accuracy of the motion vector offset of the target block;

The sign of the motion vector offset is determined based on the first information, and the absolute value of the motion vector offset is determined based on the second information and the motion vector precision.

The motion vector offset is determined based on the sign and the absolute value of the motion vector offset; and

The motion vector of the target block is determined by adjusting the motion vector offset according to the predicted motion vector value of the target block.

The merge candidate list includes time merge candidates derived from time-adjacent blocks of the target block in the juxtaposed image, the time-adjacent blocks being located to the lower right or center of the target block.

2. The method of claim 1, wherein the motion vector offset is applied to one of the x-component and the y-component of the motion vector prediction value.

3. The method according to claim 1, further comprising:

If the list of merge candidates includes more than one merge candidate, the merge candidate information is obtained.

4. An image encoding method performed by an image encoding device, the method comprising:

Construct a list of candidate blocks to merge;

Determine whether the predicted motion vector values of the target blocks from the merged candidate list can be adjusted by motion vector offset;

In response to determining that the motion vector prediction value needs to be adjusted, the motion vector prediction value of the target block is selected from one of the two merge candidates included in the merge candidate list, wherein only two merge candidates in the merge candidate list can be selected as the motion vector prediction value and adjusted by the motion vector offset, and any remaining merge candidates in the merge candidate list that are not one of the two merge candidates cannot be selected;

Determine the motion vector accuracy of the motion vector offset of the target block;

The motion vector offset is determined based on the motion vector accuracy.

The motion vector of the target block is determined by adjusting the motion vector offset according to the predicted motion vector value of the target block;

Encoding the merging candidate information indicating the motion vector prediction values from the two merging candidates included in the merging candidate list; and

The motion vector offset information is encoded, including first information about the sign of the motion vector offset and second information about the absolute value of the motion vector offset, wherein the second information is encoded using the motion vector precision.

The merge candidate list includes time merge candidates derived from time-adjacent blocks of the target block in the juxtaposed image, the time-adjacent blocks being located to the lower right or center of the target block.

5. The method of claim 4, wherein the motion vector offset is applied to one of the x-component and the y-component of the motion vector prediction value.

6. The method according to claim 4, wherein,

If the list of merge candidates includes more than one merge candidate, the merge candidate information is encoded.

7. A data transmission method for transmitting a bit stream generated by encoding image data using an image encoding method.

The bitstream mentioned therein includes:

Indicates merge candidate information derived from the motion vector predictions of two merge candidates included in the merge candidate list; and

Motion vector offset information, which includes first information about the sign of the motion vector offset and second information about the absolute value of the motion vector offset, wherein the second information is encoded using motion vector precision.

The image encoding method mentioned above includes:

Construct the list of candidate merges for the target block;

Determine whether the predicted motion vector values of the target blocks from the merged candidate list can be adjusted by motion vector offset;

In response to determining that the motion vector prediction value needs to be adjusted, the motion vector prediction value of the target block is selected from one of the two merge candidates included in the merge candidate list, wherein only two merge candidates in the merge candidate list can be selected as the motion vector prediction value and adjusted by the motion vector offset, and any remaining merge candidates in the merge candidate list that are not one of the two merge candidates cannot be selected;

Determine the motion vector accuracy of the motion vector offset of the target block;

The motion vector offset is determined based on the motion vector accuracy.

The motion vector of the target block is determined by adjusting the motion vector offset according to the predicted motion vector value of the target block; and

The merged candidate information and the motion vector offset information are encoded.

The merge candidate list includes time merge candidates derived from time-adjacent blocks of the target block in the juxtaposed image, the time-adjacent blocks being located to the lower right or center of the target block.