KR20240035359A - Method, apparatus and recording medium for encoding/decoding image - Google Patents

Abstract

A method, device, and recording medium for video encoding/decoding are disclosed. Selective compression learning of hidden representations for variable-rate video compression is used for a method, apparatus, and recording medium. In embodiments, a selective compression method is disclosed that partially encodes a hidden representation in a fully generalized manner for deep learning-based variable rate image compression. The methods of the embodiments adaptively determine essential presentation elements for compression of different target quality levels.

Description

Method, device, and recording medium for video encoding/decoding {METHOD, APPARATUS AND RECORDING MEDIUM FOR ENCODING/DECODING IMAGE}

The present invention relates to a method, device, and recording medium for video encoding/decoding. Specifically, the present invention provides a method, device, and recording medium for video encoding/decoding that provides variable-rate video compression.

Through the continued development of the information and communications industry, broadcasting services with HD (High Definition) resolution have spread globally. Through this proliferation, many users have become accustomed to high-resolution, high-definition images and/or videos.

In order to satisfy users' demand for high image quality, many organizations are accelerating the development of next-generation imaging devices. User interest in not only High Definition TV (HDTV) and Full HD (FHD) TV, but also Ultra High Definition (UHD) TV, which has a resolution more than four times that of FHD TV. has increased, and with this increase in interest, image encoding/decoding technology for images with higher resolution and image quality is required.

Using this video compression technology, video data can be effectively compressed, transmitted, and stored.

One embodiment may provide an apparatus, method, and recording medium for variable-rate video compression.

One embodiment may provide an apparatus, method, and recording medium using selective compression learning of hidden representations.

In one aspect, generating a hidden representation using an input image; generating a quantized hidden representation by performing adaptive quantization on the hidden representation; deriving a set of selected elements of the quantized hidden representation; and generating encoded information of the selected elements by performing entropy encoding on the set of selected elements.

The quantized hidden representation can be generated for a specific target quality level.

The set of selected elements can be determined using a three-dimensional binary mask.

The 3D binary mask can be generated using the output of a specific layer of the hyper decoder.

A hyperprior may be input into the hyper decoder.

The encoded information of the selected elements may be generated using parameters for a specific target quality level.

The parameters may include scale parameters for the specific target quality level or intermediate parameters for the specific target quality level.

On the other hand, generating a set of selected elements by performing decoding on encoded information of selected elements of the quantized hidden representation; converting the selected set of elements into elements of a three-dimensional-shaped hidden representation; generating de-quantized elements by performing de-quantization on elements of the 3D-shaped hidden representation; and generating a restored image by performing decoding on the dequantized elements.

The de-quantization may be performed for a specific target quality level.

Elements of the 3D-shaped hidden representation may be determined using a 3D binary mask.

The 3D binary mask can be generated using the output of a specific layer of the hyper decoder.

A hyperprior may be input into the hyper decoder.

The selected set of elements can be used to generate information using parameters for a specific target quality level.

The parameters may include scale parameters for the specific target quality level or intermediate parameters for the specific target quality level.

On another side, in a computer-readable recording medium storing a bitstream for video decoding, the bitstream includes encoded information of selected elements of a quantized hidden representation, and By performing decoding, a set of selected elements is created, the set of selected elements is converted into elements of a 3D-shaped hidden representation, and de-quantization of the elements of the 3D-shaped hidden representation is performed. A computer-readable recording medium is provided in which de-quantized elements are generated by performing decoding on the de-quantized elements, and a restored image is generated by decoding the de-quantized elements.

The de-quantization may be performed for a specific target quality level.

Elements of the 3D-shaped hidden representation may be determined using a 3D binary mask.

The 3D binary mask can be generated using the output of a specific layer of the hyper decoder.

The selected set of elements can be used to generate information using parameters for a specific target quality level.

The parameters may include scale parameters for the specific target quality level or intermediate parameters for the specific target quality level.

An apparatus, method, and recording medium for variable-rate video compression are provided.

An apparatus, method, and recording medium using selective compression learning of hidden representations are provided.

1 shows the overall architecture of the SCR method of the embodiment.
Figure 2 shows a 3D binary mask generation process according to one example.
3 shows importance adjustment curves within eight target quality levels according to an example.
4 shows generated masks for different target quality levels according to an example.
Figure 5 shows average ratios of selected presentation elements versus average BPP according to an example.
Figure 6 shows the average percentages of reused presentation elements from a low quality level to a high quality level according to an example.
Figure 7 shows the code of an expression selection operator and a reformulation operator according to an example.
Figure 8 shows the structure of an encoding device according to an embodiment.
Figure 9 is a signal flow diagram of an encoding method according to an embodiment.
Figure 10 is a structural diagram of a decoding device according to an embodiment.
Figure 11 is a flowchart of a decryption method according to an embodiment.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention.

For a detailed description of the exemplary embodiments described below, refer to the accompanying drawings, which illustrate specific embodiments by way of example. These embodiments are described in sufficient detail to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein with respect to one embodiment may be implemented in other embodiments without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the embodiment. Accordingly, the detailed description that follows is not to be taken in a limiting sense, and the scope of the exemplary embodiments is limited only by the appended claims, together with all equivalents to what those claims assert if properly described.

Similar reference numbers in the drawings refer to identical or similar functions across various aspects. The shapes and sizes of elements in the drawings may be exaggerated for clearer explanation.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component without departing from the scope of the present invention. The term “and/or” may include any of a plurality of related stated items or a combination of a plurality of related stated items.

When a component is said to be “connected” or “connected” to another component, the two components may be directly connected or connected to each other, but It should be understood that other components may exist in the middle of the components. On the other hand, when a component is said to be “directly connected” or “directly connected” to another component, it should be understood that no other component exists in between the two components. something to do.

Components appearing in the embodiments are shown independently to represent different characteristic functions, and do not mean that each component consists of separate hardware or a single software component. In other words, each component is listed and included as a separate component for convenience of explanation, and at least two of each component are combined to form one component, or one component is divided into multiple components to function. It can be performed, and integrated embodiments and separate embodiments of each of these components are included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

The terms used in the examples are only used to describe specific examples and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In embodiments, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are intended to indicate the presence of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. It should be understood that this does not exclude in advance the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof. In other words, the description of “including” a specific configuration in the embodiments does not exclude configurations other than the configuration, and means that additional configurations may also be included in the practice of the present invention or the scope of the technical idea of the present invention. .

In embodiments, the term “at least one” may mean one of one or more numbers, such as 1, 2, 3, and 4. In embodiments, the term “a plurality of” may mean one of two or more numbers, such as 2, 3, and 4.

Some of the components of the embodiments may not be essential components that perform essential functions in the present invention, but may simply be optional components to improve performance. Embodiments may be implemented by including only components essential for implementing the essence of the embodiments, excluding components used only to improve performance. Structures that include only essential components excluding optional components used to improve performance are also included in the scope of the embodiments.

Hereinafter, embodiments will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the embodiments. In describing the embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present specification, the detailed description will be omitted. In addition, the same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

Hereinafter, an image may refer to a picture constituting a video, and may also represent the video itself. For example, “encoding and/or decoding of an image” may mean “encoding and/or decoding of a video,” and may mean “encoding and/or decoding of one of the images that constitute a video.” It may be possible.

In embodiments, each of the specified information, data, flag, index, element, attribute, etc. may have a value. The value "0" of information, data, flags, indexes, elements, and attributes may represent false, logical false, or a first predefined value. That is, the values “0”, false, logical false and the first predefined value can be used interchangeably. The value "1" in information, data, flags, indexes, elements, and attributes may represent true, logical true, or a second predefined value. That is, the value “1”, true, logical true and the second predefined value can be used interchangeably.

When a variable such as i or j is used to represent a row, column, or index, the value of i may be an integer greater than or equal to 0, or an integer greater than or equal to 1. That is, in embodiments rows, columns, indices, etc. may be counted from 0, and may be counted from 1.

In embodiments, the term “one or more” or the term “at least one” may mean the term “plural.” “One or more” or “at least one” can be used interchangeably with “plural.”

Recently, many neural network-based image compression methods are showing superior results than existing tool-based codecs.

However, most of the neural network-based image compression methods are often trained as separate models according to different target bit rates, which may increase model complexity.

Therefore, several studies have been conducted on learned compression that supports various bitrates with a single model. However, these studies require additional network modules, layers or inputs and often result in complexity overhead or may not provide sufficient encoding/decoding efficiency.

In embodiments, a selective compression method may be disclosed that first partially encodes a latent representation in a fully generalized manner for deep learning-based variable rate image compression.

In embodiments, “expression” may refer to “hidden expression.”

The methods of embodiments may adaptively determine representation elements necessary for compression of different target quality levels.

For this determination, a 3Dimension (3D) importance map can first be created as a nature of the input content to indicate the underlying importance of the presentation elements. Next, the 3D importance map can be adjusted to various target quality levels using an importance adjustment curve. Finally, the adjusted 3D importance map can be converted to a 3D binary mask to determine the representation elements essential for compression.

The methods of the embodiments can be easily integrated with existing compression models with a negligible amount of increased overhead. Additionally, the methods of embodiments may enable continuously variable rate compression through simple interpolation of importance adjustment curves between various quality levels.

The methods of the embodiments can achieve compression efficiencies comparable to those of individually trained reference compression models and can shorten decoding time due to selective compression.

Neural Network (NN)-based image compression methods are being actively researched, and compared to existing tool-based compression methods such as BPG and JPEG2000, Vizontegard has a higher Peak Signal-to-Noise Ratio (PSNR). Delta (Bjontegaard Delta; BD) can show superior performance in terms of rate.

Some methods can achieve comparable results compared to a state-of-the-art codec called H.266 intra coding.

However, because most of the existing deep learning-based models are individually trained according to different target compression levels, multiple models with a large number of parameters may be required to support various compression levels.

To deal with this issue, several methods using conditional transformation or adaptive quantization can be proposed.

However, most of these methods may require additional network modules, layers or inputs, thus incurring complexity overhead.

In embodiments, a new 'Selective Compression of Representations (SCR)' method may be presented, which performs entropy encoding/decoding only on partially selected hidden representations.

The choice of representation may be determined through a 3D binary mask generation process in a targeted quality-adaptive manner.

In the 3D binary mask generation of the SCR method, (i) for multi-channel feature maps (3D representations), 3D importance maps of the same size can be generated, independent of the target quality level, and (ii) for a given target quality. A 3D importance map may be adjusted via a channel-specific importance adjustment curve for levels, and (iii) a 3D binary mask may be generated by rounding-off the adjusted 3D importance map.

A 3D importance map that is target-quality-independent can become target-quality-dependent after channel-specific importance adjustment.

The methods of the embodiments may be integrated with an adaptive quantization scheme, where the overall elements may be jointly optimized in an end-to-end manner with selective compression and adaptive quantization of the hidden representation of the embodiments.

In terms of architecture, the SCR method of embodiments can minimize overhead by utilizing only a single 1×1 convolutional layer to generate 3D importance maps and importance adjustment curves for a limited number of target quality levels.

Additionally, the SCR method can continuously support variable-rate compression through simple non-linear interpolation of importance scaling curves between two discrete target quality levels.

Furthermore, the SCR method can reduce decoding time compared to both the reference compression model and the very lightweight adaptive quantization-based variable-rate method by omitting the entropy decoding process for a significant amount of unselected representations. there is.

The encoding/decoding efficiency of the SCR method of embodiments may be superior to or similar to the encoding/decoding efficiencies of reference compression models trained individually for various target quality levels, and the encoding/decoding efficiency of the adaptive quantization-based method It can be better than that.

The methods of the embodiments may have the following features:

- The SCR method of the embodiments may be the first NN-based variable rate image compression method that selectively compresses representations in a fully generalized and target quality-adaptive manner. The SCR method of embodiments can provide compression efficiencies comparable to those of individually trained reference compression models.

- The SCR method of the embodiments can be applied to other image compression models without modifying the architecture of the image compression model. Therefore, the SCR method of the embodiments can have high applicability. Ultra-lightweight modules containing only one 1×1 convolutional layer and a few importance tuning curves can be integrated into the compression model. The SCR method of the embodiments can shorten the decoding time compared to the decoding time of the lightweight variable-rate model and the reference compression model due to selective compression.

- The SCR method of embodiments may continuously enable variable-rate compression by simple interpolation of importance tuning curves between discrete quality levels on which selective compression is trained.

overall architecture

1 shows the overall architecture of the SCR method of the embodiment.

In Figure 1, the SCR scheme can be integrated into a hyperprior model.

In Figure 1, elements for variable-rate compression can be indicated as dotted boxes.

In particular, elements for selective compression may be highlighted with a bold line.

As shown in Figure 1, the SCR method of embodiments can be combined with adaptive quantization on compression architectures with a hyper-encoder and hyper-decoder.

In embodiments, the SCR method can be applied to reference compression models such as hyperprior, mean-scale, and context to have generality and demonstrate its effectiveness.

In embodiments, the hyperprior may indicate the name of the model and may indicate collateral information. Models such as the hyperprior model can use hyperprior side information.

가 표현

로 변환될 수 있다. 하이퍼 부호화기 및 하이퍼 복호화기는

의 양자화된 표현

에 대한 분포 파라미터를 하이퍼프라이어로 명명된 부수 정보로서 부호화/복호화하기 위해 사용될 수 있고, 이러한 분포 파라미터를 통해

가 엔트로피-부호화 및 엔트로피-복호화될 수 있다.In an architecture with a hyper-encoder and a hyper-decoder, the input image is encoded using an encoder network.

expression

can be converted to Hyper encoder and hyper decoder are

Quantized representation of

It can be used to encode/decode the distribution parameters for as side information named hyperprior, and through these distribution parameters,

can be entropy-encoded and entropy-decoded.

는 복호화기 네트워크를 통해 이미지

로 복원될 수 있다.Next, the quantized representation

image through a decryptor network

can be restored.

On this base compression architecture, variable rate compression can be made possible by utilizing two additional elements: adaptive quantization and selective compression.

Selection of expression elements on the encoder side can be expressed as Equation 1 below.

는 목적 품질 수준

에서의 양자화된 표현일 수 있다.From here,

is the objective quality level

It may be a quantized expression in .

에 대하서 아래의 수학식 2가 성립할 수 있다. 즉, 수학식 2의 조건/정의 하에서 수학식 1이 성립할 수 있다.From here,

For this reason, Equation 2 below can be established. In other words, Equation 1 can be established under the conditions/definition of Equation 2.

는 주어진 목표 품질 수준

에 대한

의 선택된 요소들의 집합일 수 있다.From here,

is the given target quality level

for

It may be a set of selected elements of .

는 타겟 품질-적응적 양자화 연산자일 수 있다. 여기에서,

에 대하여 양자화 벡터

를 갖고, 아래의 수학식 3이 성립할 수 있다.

may be a target quality-adaptive quantization operator. From here,

With respect to the quantization vector

With this, Equation 3 below can be established.

은

에 대한 요소 선택 연산자일 수 있다.

silver

It can be an element selection operator for .

은

및 하이퍼프라이어

에 대하여 생성된 3D 이진 마스크를 나타낼 수 있다.

silver

and hyperfryer

It can represent the 3D binary mask generated for .

는 도 1에서 도시된 입력 영상

에 대한 부호화기 네트워크

의 출력

일 수 있다.expression

is the input image shown in Figure 1

Encoder network for

output of

It can be.

는 목표 품질 의존 분포

에 기반하는 엔트로피 모델을 사용하여 엔트로피-부호화 및 엔트로피-복호화될 수 있다.

is the target quality dependent distribution

It can be entropy-encoded and entropy-decoded using an entropy model based on .

는 아래의 수식 4와 같을 수 있다.Reconstructed image on the decoder side

may be the same as Equation 4 below.

Here, Equation 5 and Equation 6 can be established. In other words, Equation 4 can be established under the conditions/definition of Equation 5 and Equation 6.

는, 복호화기 네트워크

의 출력으로서의, 주어진 타겟 품질 수준

에 대한 복원된 영상(reconstructed image)일 수 있다.From here,

, the decoder network

As the output of, given the target quality level

It may be a reconstructed image for .

는 역-양자화 벡터

를 입력

에 곱하는 적응적 역-양자화 연산자일 수 있다.

is the inverse-quantized vector

Enter

It may be an adaptive inverse-quantization operator that multiplies with .

는 3D 이진 마스크

를 사용함으로써 1D 형태를 갖는 선택된 요소들

을 3D-형태된(3D-shaped) 표현의 요소들로 변형(convert)하는 재형태(reshaping) 연산자일 수 있다. 이 때, 1D 형태의

을 구성하는 각각의 요소는 부호화기의

요소 선택 과정을 거치기 이전 위치에 재배치될 수 있다.

is a 3D binary mask

Selected elements have a 1D shape by using

It may be a reshaping operator that converts into elements of a 3D-shaped representation. At this time, 1D form

Each element that constitutes the encoder's

It can be relocated to the location before going through the element selection process.

는 0들을 대응하는 위치들에 위치시킬 수 있다.For unselected elements, reformat operator

can place zeros in corresponding positions.

및

의 예시적인 코드들은 아래에서 개시된다.

and

Exemplary codes are disclosed below.

및

의 벡터 차원수들(dimensionalities)은

일 수 있으며,

는

내의 채널들의 개수일 수 있다. 이로서,

의 양자화 및

의 역-양자화는 대응하는(respective) 요소들

및

에 의해 채널-별로 각각 수행될 수 있다.In Equation 3 and Equation 5,

and

The vector dimensionality of is

It can be,

Is

It may be the number of channels within. With this,

Quantization of and

The inverse quantization of the corresponding (respective) elements

and

It can be performed individually on a channel-by-channel basis.

Generate 3D binary mask

Figure 2 shows a 3D binary mask generation process according to one example.

The 3D binary mask creation process can consist of the following three steps:

(1) Generating 3D importance maps;

(2) Importance adjustment

(3) Binarization

The 3D binary mask creation process can be defined as Equation 7 below.

는 입력으로서 사용되는 하이퍼프라이어

에 대한 하이퍼-복호화기를 통해 생성된 3D 중요도 맵일 수 있다.From here,

is the hyperfryer used as input.

It may be a 3D importance map generated through a hyper-decoder for .

는 차원수

의 파라미터 벡터일 수 있다.

는 아래의 수식 8과 같이 정의될 수 있다.

is the number of dimensions

It may be a parameter vector of .

Can be defined as Equation 8 below.

은

와 같을 수 있다.

silver

It may be the same as

의 파라미터들은 주어진 목표 품질

에 대한 채널-별 중요도 조정 곡선들을 결정하기 위해 학습될 수 있다.

The parameters of are given the target quality

Can be learned to determine channel-specific importance adjustment curves for .

는 라운딩-오프(rounding-off)를 갖는 이진화(binarization) 연산자일 수 있다.

may be a binarization operator with rounding-off.

Create 3D importance map

은

내의 각 요소의 기저를 이루는(underlying) 중요도를 나타낼 수 있다.3D importance map

silver

It can indicate the underlying importance of each element within.

은 0 및 1의 사이의 범위 내의 값들을 가질 수 있다.3D importance map

can have values in the range between 0 and 1.

를 생성하는 전용의(dedicated) 복잡한(complex) 네트워크를 활용하지 않은 채, 하이퍼 복호화기 내의 (활성화 이후의) 끝에서 두 번째(penultimate) 컨볼루션 레이어의 출력이 마스크 생성 모듈 내의 단일한 1×1 컨볼루션 레이어로 입력될 수 있다. 그 뒤를 이어, 0 및 1의 사이의 중요도 값들을 획득하기 위해 클리핑 함수(function)가 적용될 수 있다. 즉, 1×1 컨볼루션 레이어의 출력에 대하여 클리핑 함수가 적용될 수 있다. 마스크 생성 모듈 내의 단일한 1×1 컨볼루션 레이어의 입력은 하이퍼복호화기 내의 다른 레이어의 출력일 수 있다. 일 예로, 마스크 생성 모듈 내의 단일한 1×1 컨볼루션 레이어의 입력은 하이퍼 복호화기의 최종 출력 또는 두 번째(penultimate) 레이어보다 더 이전 레이어의 출력일 수 있다.

Without utilizing a complex network dedicated to generating It can be input into a convolutional layer. Subsequently, a clipping function can be applied to obtain importance values between 0 and 1. That is, a clipping function can be applied to the output of the 1×1 convolution layer. The input of a single 1×1 convolutional layer in the mask generation module may be the output of another layer in the hyperdecoder. As an example, the input of a single 1×1 convolutional layer in the mask generation module may be the final output of a hyper decoder or the output of a layer earlier than the penultimate layer.

의 특성을 나타낼 수 있다.Here, the 3D importance map can be generated depending on the input images, without depending on the target quality levels. Therefore, the 3D importance map is

characteristics can be expressed.

Adjust importance

3 shows importance adjustment curves within eight target quality levels according to an example.

The SCR method can be implemented on the hyperprior model.

The actual importance of each presentation element may vary according to various target quality levels. For example, some presentation elements that correspond to high complexity textures in images may not be essential in low-quality compression.

Therefore, it may be natural to adjust the 3D importance map, which is commonly used for all quality levels, according to the specific target quality level.

이 조정되는 방식이 제공될 수 있다.For this adjustment, a 3D importance map is created using importance adjustment curves for various target quality levels.

This coordinated method may be provided.

의 요소 값들을 채널-별로 변경할 수 있다. The importance adjustment curves are

Element values can be changed on a channel-by-channel basis.

로서 학습될 수 있다. 여기에서,

는 1 보다 더 크고,

보다 더 작을 수 있다.

는 학습에 이용되는 목표 품질 수준들의 총 개수일 수 있다.The curvatures of the importance adjustment curves are parameter vectors.

It can be learned as. From here,

is greater than 1,

It can be smaller than

may be the total number of target quality levels used for learning.

가 증가함에 따라 향상될 수 있다.The target quality is

It can be improved as .

Figure 3 shows some examples of importance adjustment curves.

값을 나타낼 수 있다. 수직 축은 입력

값의 조정된 결과를 나타낼 수 있다.In Figure 3, the horizontal axis is the input to be adjusted

It can represent a value. The vertical axis is the input

The adjusted result of the value can be displayed.

개의 목표 품질 레벨들에 대한 훈련된

벡터들의 평균 값들을 가리킬 수 있다.Additionally, the numbers attached to the importance adjustment curves are

Dogs trained to target quality levels

It can refer to the average values of vectors.

에 대한 중요도 조정 곡선들은 입력

의 요소들을 증폭하는(amplify) 경향을 가질 수 있다. 반면, 6보다 더 작은

에 대한 중요도 조정 곡선들은 입력

의 요소들을 감쇠시킬(attenuating) 수 있다.

가 6인 경우에 대하여, 중요도 조정의 이전 및 이후에서 변동(variation)이 거의 없을 수 있다. 이 때, 평균

는 0.9897일 수 있다.According to Figure 3, in terms of average, greater than 6

The importance adjustment curves for the input

It may have a tendency to amplify elements of . On the other hand, smaller than 6

The importance adjustment curves for the input

It is possible to attenuate the elements of

For the case where is 6, there may be little variation before and after the importance adjustment. At this time, average

may be 0.9897.

는 더 높은 목표 품질 수준들에 대하여 전반적으로(overall) 더 강하게 증폭될 수 있다.As a result,

can be amplified more strongly overall for higher target quality levels.

는 크게 감쇄될 수 있다. 따라서,

요소들 중 1에 가까운 값을 갖는 소수의 일부만이

요소들의 중요도들을 유지할 수 있다.On the other hand, in general, for lower target quality levels,

can be greatly attenuated. thus,

Only a small fraction of the elements have values close to 1.

The importance of elements can be maintained.

벡터들의 총 개수는

일 수 있다. 따라서, 총

×

개의 파라미터들이 모든

벡터들에 대하여 학습될 수 있다.

The total number of vectors is

It can be. Therefore, total

×

All parameters are

Vectors can be learned.

는 8로 설정될 수 있다.

는 참조 모델에서의

로 설정될 수 있다.In embodiments,

can be set to 8.

is in the reference model

It can be set to .

Binization

와 같이 표시될 수 있다.The 3D binary mask can be finally determined by the rounding operator. The rounding operator is

It can be displayed as follows.

내 동일한 위치에서의 대응하는 요소들이 선택되었음을 나타낼 수 있다.Here, the "1" values in the output 3D binary mask are

It can indicate that the corresponding elements at the same location have been selected.

4 shows generated masks for different target quality levels according to an example.

At the top of Figure 4, sample masks of eight target quality levels are shown. The dark parts of the sample masks indicate the representation elements selected by the 3D binary masks.

At the bottom of Figure 4, masks averaged along the channel axis are shown.

The higher the target quality, the more presentation elements can be selected, especially in more complex regions.

는 1.0 부터 8.0까지일 수 있다.For example,

can be from 1.0 to 8.0.

The SCR method of the embodiments can be implemented on a hyperfryer model, and Kodim12 images from the Kodim12 image set can be used as input samples. In the input sample, components marked in bold may indicate “1” values.

가 SCR 방법에서 가장 낮은 품질 수준인 1.0으로 설정되었을 때, 단지 총 요소들의 3.22% 만이 선택될 수 있다. 또한,

값이 증가함에 따라 선택 비율은 점진적으로(gradually) 증가할 수 있다.For example,

When is set to 1.0, which is the lowest quality level in the SCR method, only 3.22% of the total elements can be selected. also,

As the value increases, the selection ratio can increase gradually.

가 8.0인 경우, 표현 요소들의 43.39%가 선택될 수 있다.For example,

When is 8.0, 43.39% of the expression elements can be selected.

Additionally, the SCR method of embodiments can use more representations within the high-complexity region, as shown in the averaged masts along the channel axis.

Figure 5 shows average ratios of selected presentation elements versus average BPP according to an example.

In the embodiment referring to Figure 5, the test set may be a Kodak image set. The base model may be a hyperfryer.

For example, for the entire Kodak image set, the average proportions of selected elements for target quality levels from 1.0 to 8.0 were 6.41%, 9.66%, 14.17%, 19.90%, respectively. It could be 27.00%, 35.68%, 46.20% and 55.81%. Here, as shown in Figure 5, the average ratios may be approximately linearly proportional to the average Bits Per Pixel (BPP) values.

Figure 6 shows the average percentages of reused presentation elements from a low quality level to a high quality level according to an example.

In the embodiment referring to Figure 6, the test set may be a Kodak image set. The base model may be a hyperfryer.

Figure 6 may indicate how many representations at a lower target quality level are typically used (or selected) for higher target quality levels.

가 2일 때의 선은 표현 요소들의 100%, 99.8%, 99.6%, 99.0%, 98.3% 및 98.2%를 가리킬 수 있다. 목표 품질 레벨

가 2일 때 선택된 표현 요소들은 3.0에서 8.0까지의 목표 품질 레벨

들에 대해서도 각각 재사용될 수 있다.For example, in Figure 6, the target quality level

When is 2, the line can indicate 100%, 99.8%, 99.6%, 99.0%, 98.3%, and 98.2% of the expression elements. target quality level

When is 2, the selected expression elements have target quality levels from 3.0 to 8.0.

Each can also be reused.

According to Figure 6, The case where is 8.0 is For the case where is 1.0, 97.6% of the selected expression elements can be significantly reused.

This reuse means that the SCR method of the embodiments does not select representation elements separately for different target quality levels, but actively takes a large portion of representation elements as common components for various target quality levels. It can be expressed.

training

The SCR model can be trained in an end-to-end manner using the total loss formulated according to Equation 9 below.

에 대하서 아래의 수학식 10이 성립할 수 있다. 즉, 수학식 10의 조건/정의 하에서 수학식 9가 성립할 수 있다.From here,

For this reason, Equation 10 below can be established. In other words, Equation 9 can be established under the conditions/definition of Equation 10.

는 타겟 품질 수준

에 대한 레이트 항(term)을 나타낸다.

는 타겟 품질 수준

에 대한 왜곡(distortion) 항을 나타낸다.From here,

is the target quality level

Indicates the rate term for .

is the target quality level

It represents the distortion term for .

는 레이트 및 왜곡 간의 균형(balance)를 조정하기 위한 파라미터를 나타낸다.

는 아래의 수학식 11과 같이 정의될 수 있다.

represents a parameter for adjusting the balance between rate and distortion.

Can be defined as Equation 11 below.

는 입력 영상

및 복원된 영상

간의 중간 제곱된 오차(Mean Squared Error; MSE) 또는 멀티 스케일-구조적 유사성(Multi Scale-Structural SIMilarity; MS-SSIM)일 수 있다.

is the input image

and restored video

It may be Mean Squared Error (MSE) or Multi Scale-Structural SIMilarity (MS-SSIM).

은 3000(1 - MS-SSIM(

,

))일 수 있다.In MS-SSIM-based optimization, distortion terms used

is 3000(1 - MS-SSIM (

,

)) can be.

는

및

의 양자화된 표현들에 대한 계산된 크로스-엔트로피일 수 있다.

Is

and

It may be the calculated cross-entropy for quantized representations of .

의 경우에서, 양자화 및 마스크 생성 프로세스들은 각 목표 품질 수준

에 대하여 다르기 때문에, 크로스 엔트로피

가 아래의 수학식 12와 같이 목적 품질 수준

에 대하여 사용될 수 있다.

In this case, the quantization and mask generation processes are performed at each target quality level.

Since it is different with respect to

The objective quality level is as shown in Equation 12 below:

It can be used against.

에 대하서 아래의 수학식 13이 성립할 수 있다. 즉, 수학식 13의 조건/정의 하에서 수학식 12가 성립할 수 있다.From here,

For this reason, Equation 13 below can be established. In other words, Equation 12 can be established under the conditions/definition of Equation 13.

는 입력 영상

내의 픽셀들의 개수일 수 있다.

is the input image

It may be the number of pixels within.

는

의 선택된 요소들

의 총 개수일 수 있다.

Is

selected elements of

It may be the total number of .

는, 서로 다른 목표 품질 수준들에 대하여 변하는

의 분포를 다루기 위하여, 근사 확률 질량 함수(Probability Mass Function; PMF)

에 기반하여 계산될 수 있다.Cross entropy of selected expression elements

varies for different target quality levels.

To deal with the distribution of , an approximate Probability Mass Function (PMF) is used.

It can be calculated based on .

의 추산된(estimated) 분포 파라미터들

및

은

및

로서 각각 결정될 수 있다.especially,

The estimated distribution parameters of

and

silver

and

Each can be determined as .

값 및

값은 베이스 압축 모델들로부터 획득될 수 있다.From here,

value and

Values can be obtained from base compression models.

는 중간(mean) 파라미터일 수 있다.

는 양자화된 표현

의 엔트로피 모델에 대한 중건 파라미터일 수 있다.

may be a mean parameter.

is a quantized representation

It may be a neutral parameter for the entropy model of .

는 스케일 파라미터일 수 있다.

는 양자화된 표현

의 엔트로피 모델에 대한 스케일 파라미터일 수 있다.

may be a scale parameter.

is a quantized representation

It may be a scale parameter for the entropy model of .

및

는 각 공간적(spatial) 좌표

에 대하여

및

를 통해 각각 획득될 수 있다. For context-based models, position-wise parameters

and

is each spatial coordinate

about

and

Each can be obtained through .

에 대하여 제로-평균(zero-mean) 가우시안-기반 모델이 사용될 때,

는 무시될 수 있다.

When a zero-mean Gaussian-based model is used,

can be ignored.

로서 균일(uniform) 분포를 가지고 컨볼루션된(convolved with) 가우시안 분포 모델이 채용될 수 있다.As in entropy minimization-based compression models, the approximate PMF

A Gaussian distribution model convolved with a uniform distribution can be adopted.

의 대신에, 추가적인 균일 노이즈

(-0.5, 0.5)가 더해진 표현이 훈련을 위해 사용될 수 있다. 추가적인 균일 노이즈

(-0.5, 0.5)가 더해진 표현은

로 표시될 수 있다.Also, for inference, the rounded expression

Instead of , additional uniform noise

Expressions with (-0.5, 0.5) added can be used for training. Additional uniform noise

The expression with (-0.5, 0.5) added is

It can be displayed as .

보다는 확률적으로(stochastically) 생성된 마스크

가 테스트 페이즈 내에서 사용될 수 있다.To deal with instability within the training phase due to learning the binary representation of the mask,

A mask generated stochastically rather than

Can be used within the test phase.

에 대하여 단순한 라운드-오프(rounded-off)가 되지만, 조정된 3D 중요도 맵

의 각 요소 값을 출력 마스크의 대응하는 구성요소가 "1"일 확률로 간주함으로써 랜덤으로 샘플링된 이진 표현들을 가지고

가 구축될 수 있다.The adjusted 3D importance map is

is a simple rounded-off, but adjusted 3D importance map.

with randomly sampled binary representations by considering the value of each element of

can be built.

는 아래의 수학식 14와 같이 생성될 수 있다.

Can be generated as in Equation 14 below.

에 의해 야기되는 불연속성은 그래디언트들(gradients)을 역방향(backward)으로 바이패싱(bypassing)함으로써 다루어질 수 있다.Rounding-off operator

The discontinuity caused by can be handled by bypassing the gradients backward.

및

를 사용하지 않고 수행될 수 있다. 왜냐하면, 비선택된 표현들은

를 계산하기 위한 아래의 수학식 15를 사용함으로써 제외될 수 있고,

를 계산하기 위해

를 통해

를 획득할 수 있기 때문이다.In practical implementation, training

and

It can be done without using . Because the non-selected expressions are

It can be excluded by using Equation 15 below to calculate

to calculate

Through the

This is because you can obtain .

에 대하서 아래의 수학식 16이 성립할 수 있다. 즉, 수학식 16의 조건/정의 하에서 수학식 15가 성립할 수 있다.From here,

For this reason, Equation 16 below can be established. In other words, Equation 15 can be established under the conditions/definition of Equation 16.

Other training details are also described below.

Continuous variable-rate compression

가 결정될 수 있다. 여기에서, q는 2 개의 이산 목표 품질 레벨들 사이의 값일 수 있다.To support continuous variable-rate compression during testing, by interpolation, as defined in Equation 17 below:

can be decided. Here, q can be a value between two discrete target quality levels.

가 3.8일 때,

는

및

의 요소-별 곱(multiplication)에 의해 결정될 수 있다.For example,

When is 3.8,

Is

and

It can be determined by element-wise multiplication of .

및

벡터들 또한 상기에서 설명된 것과 동일한 방식으로 보간될 수 있다. 상기의 보간은 비-선형 보간일 수 있다.

and

Vectors can also be interpolated in the same way as described above. The above interpolation may be non-linear interpolation.

operator's code

Figure 7 shows the code of an expression selection operator and a reformulation operator according to an example.

및 재형태 연산자

를 구현하기 위한 코드가 도시되었다.In Figure 7, the selection operator

and reshaping operators

The code to implement is shown.

These two modules can be used in the test phase. These two modules may not be required for training.

Training details of the SCR method

For more reliable and faster training, step-by-step training can be employed, including the three steps below:

는 8.0일 수 있다.(1) In the first step, a fixed-rate compression model can be trained for high quality compression. For example, in high-quality compression, the target quality level of a variable-rate model is

may be 8.0.

(2) In the second step, the trained fixed-rate compression model can be used as a pre-trained model. The second-stage SCR deformation model can be trained without selective compression in an end-to-end manner.

(3) In the third step, the SCR variant model that does not perform the trained selective compression of the second step can be used as the pre-trained model. The SCR full model in the third stage can be trained in an end-to-end manner.

All training steps can be performed using an optimizer until the performance of each step sufficiently converges.

For example, the number of training iterations for these three stages may be 7 million, 1.2 million, and 1.2 million, respectively.

As a training data set, 51,141 patches of size 256×256 cut from the entire training set in a non-overlapping manner can be used. The batch size can be set to 8.

The initial learning rate can be set to 5×10 ^-5 . In the final 100,000 iterations a learning rate of 2×10 ^-6 can be used. This learning rate decay can be performed for all training phases.

Figure 8 shows the structure of an encoding device according to an embodiment.

The encoding device 800 includes an encoder 810, an adaptive quantization unit 820, a hyper encoder 830, a quantization unit 835, a first entropy encoder 840, a hyper decoder 845, and a 3D mask generator. It may include (850), an expression selection unit (855), a scaling and selection unit (860), a second entropy encoder (865), and a communication unit (870).

에 대한 부호화를 수행함으로써 생성된 정보를 포함하는 비트스트림을 생성할 수 있다.The encoding device 800 is an input video

A bitstream containing the generated information can be generated by performing encoding on .

Encoder 810, adaptive quantization unit 820, hyper encoder 830, quantization unit 835, first entropy encoder 840, hyper decoder 845, 3D mask generator 850, representation selection At least some of the unit 855, the scaling and selection unit 860, the second entropy encoder 865, and the communication unit 870 may be program modules and may communicate with an external device or system. Program modules may be included in the encoding device 800 in the form of an operating system, application program module, and other program modules.

Program modules may be physically stored on various known storage devices. Additionally, at least some of these program modules may be stored in a remote memory device capable of communicating with the encoding device 800.

Program modules are routines, subroutines, programs, objects, components, and data that perform a function or operation according to an embodiment or implement an abstract data type according to an embodiment. It may include data structures, etc., but is not limited thereto.

Program modules may be composed of instructions or codes that are executed by at least one processor of the encoding device 800.

The encoding device 800 may be implemented in a computer system that includes a recording medium that can be read by a computer.

The recording medium may store at least one module required for the encoding device 800 to operate.

Functions related to communication of data or information of the encoding device 800 may be performed through the communication unit 870.

For example, the communication unit 870 may transmit a bitstream to the decoding device 1000, which will be described later.

Figure 9 is a signal flow diagram of an encoding method according to an embodiment.

을 사용하여 은닉 표현

를 생성할 수 있다.In step 910, the encoder 810 processes the input image

Hidden expression using

can be created.

에 대한 부호화를 수행함으로써 은닉 표현

를 생성할 수 있다.The encoder 810 is an input video

Hidden representation by performing encoding on

can be created.

가 주어졌을 때, 적응적 양자화부(820)는 은닉 표현

에 대한 적응적 양자화를 수행함으로써 목표 품질 수준

에서의 양자화된 은닉 표현

을 생성할 수 있다.At step 920, target quality level

When given, the adaptive quantization unit 820 uses the hidden expression

Target quality level by performing adaptive quantization for

Quantized hidden representation in

can be created.

가 주어진다는 것은 목표 품질 수준

가 특정 구성요소에 입력된다는 것을 의미할 수 있다. 또는, 특정 구성요소에 목표 품질 수준

가 주어진다는 것은 특정 목표 품질 수준에 대하여 특정 구성요소가 생성된다는 것을 의미할 수 있다.In embodiments, a target quality level for a particular component.

Given a target quality level

This may mean that is input to a specific component. Or, a target quality level for a specific component.

Being given may mean that a specific component is created for a specific target quality level.

은 특정 목표 품질 수준에 대하여 생성될 수 있다.For example, say, quantized hidden representation

can be generated for a specific target quality level.

을 사용하여 하이퍼프라이어 은닉(hyperprior latent)

를 생성할 수 있다.At step 930, hyper-encoder 830 generates the hidden representation

Using hyperprior latent

can be created.

을 사용하여 양자화된 하이퍼프라이어 은닉

를 생성할 수 있다.In step 935, the quantization unit 835 performs hyperprior hiding.

Quantized hyperprior concealment using

can be created.

에 대한 양자화를 수행함으로써 양자화된 하이퍼프라이어 은닉

를 생성할 수 있다.The quantization unit 835 hides the hyperfryer.

Concealing the quantized hyperprior by performing quantization on

can be created.

에 대한 엔트로피 부호화를 수행함으로써 하이퍼프라이어의 부호화된 정보를 생성할 수 있다.At step 940, the first entropy encoder 840 performs the quantized hyperprior concealment.

By performing entropy encoding on , the encoded information of the hyperprior can be generated.

The bitstream may include encoded information of the hyperprior.

를 사용하여 끝에서 두 번째(penultimate) 레이어의 출력을 생성할 수 있다.At step 945, the hyper decoder 845 generates the quantized hyperprior concealment.

You can use to generate the output of the penultimate layer.

를 사용하여 파라미터를 생성할 수 있다. 파라미터는 스케일 파라미터

를 포함할 수 있다. 파라미터는 평균 파라미터

를 포함할 수 있다.The hyper decoder 845 hides the quantized hyperprior.

You can create parameters using . The parameter is a scale parameter

may include. The parameter is the average parameter

may include.

In step 950, the 3D mask generator 850 may generate a 3D binary mask using the output of a specific layer of the hyper decoder.

Certain layers may be penultimate. The 3D mask generator 850 may generate a 3D binary mask using the output of the penultimate layer of the hyper decoder.

In embodiments, 3D mask may refer to a 3D binary mask.

가 입력될 수 있다. 하이퍼 복호화기는 양자화된 하이퍼프라이어 은닉

에 대한 복호화를 수행함으로써 끝에서 두 번째(penultimate) 레이어의 출력을 생성할 수 있다.Quantized hyperprior concealment with hyperdecoder

can be entered. The hyperdecoder hides the quantized hyperprior.

By performing decoding on , the output of the penultimate layer can be generated.

가 주어졌을 때, 3D 마스크 생성부(850)는 끝에서 두 번째(penultimate) 레이어의 출력을 사용하여 목표 품질 수준

에 대한 3D 이진 마스크를 생성할 수 있다.target quality level

Given, the 3D mask generator 850 uses the output of the penultimate layer to achieve the target quality level.

A 3D binary mask can be created.

에서의 양자화된 은닉 표현

및 3D 이진 마스크를 사용하여 목표 품질 수준

에 대한

의 선택된 요소들의 집합

를 유도할 수 있다.At step 955, the representation selection unit 855 selects a target quality level.

Quantized hidden representation in

and target quality level using a 3D binary mask.

for

A set of selected elements of

can be derived.

가 주어졌을 때, 스케일링 및 선택부(860)는 3D 이진 마스크 및 파라미터를 사용하여 목표 품질 수준

에 대한 파라미터를 생성할 수 있다.At step 960, target quality level

Given, the scaling and selection unit 860 uses the 3D binary mask and parameters to select the target quality level.

You can create parameters for .

를 포함할 수 있다. 파라미터는 평균 파라미터

를 포함할 수 있다.The parameter is a scale parameter

may include. The parameter is the average parameter

may include.

에 대한 파라미터는 목표 품질 수준

에 대한 스케일 파라미터

를 포함할 수 있다.

는

에 기반하여 생성될 수 있다.target quality level

The parameters for the target quality level are

scale parameter for

may include.

Is

It can be created based on .

에 대한 파라미터는 목표 품질 수준

에 대한 평균 파라미터

를 포함할 수 있다.

는

에 기반하여 생성될 수 있다.target quality level

The parameters for the target quality level are

average parameter for

may include.

Is

It can be created based on .

에 대한 파라미터를 사용하여 목표 품질 수준

에서의 양자화된 은닉 표현

의 선택된 요소들의 집합

에 대한 엔트로피 부호화를 수행함으로써 목표 품질 수준

에서의 양자화된 은닉 표현

의 선택된 요소들의 부호화된 정보를 생성할 수 있다.At step 965, the second entropy encoder 865 sets the target quality level

Target quality level using parameters for

Quantized hidden representation in

A set of selected elements of

Target quality level by performing entropy encoding for

Quantized hidden representation in

Encoded information of selected elements can be generated.

에서의 양자화된 은닉 표현

의 선택된 요소들의 부호화된 정보를 포함할 수 있다.Bitstream is at target quality level

Quantized hidden representation in

It may contain encoded information of selected elements of .

In step 970, the communication unit 870 may transmit a bitstream to the decoding device 1000.

The description and processing of information described above in the embodiments may also be applied to the information of the steps described with reference to FIG. 9 .

Figure 10 is a structural diagram of a decoding device according to an embodiment.

The decoding device 1000 includes a communication unit 1005, a first entropy decoder 1040, a hyper decoder 1045, a 3D mask generator 1050, a scaling and selection unit 1060, and a second entropy decoder 1065. ), a reshaping unit 1080, an adaptive inverse-quantization unit 1085, and a decoder 1090.

를 생성할 수 있다.The decoding device 1000 produces a reconstructed image by performing decoding on the encoded information of the bitstream.

can be created.

Communication unit 1005, first entropy decoder 1040, hyper decoder 1045, 3D mask generator 0150, scaling and selection unit 1060, second entropy decoder 1065, reshaping unit ( 1080), at least some of the adaptive dequantization unit 1085 and the decoder 1090 may be program modules and may communicate with an external device or system. Program modules may be included in the decryption device 1000 in the form of an operating system, application program module, and other program modules.

Program modules may be physically stored on various known storage devices. Additionally, at least some of these program modules may be stored in a remote memory device capable of communicating with the decoding device 1000.

Program modules are routines, subroutines, programs, objects, components, and data that perform a function or operation according to an embodiment or implement an abstract data type according to an embodiment. It may include data structures, etc., but is not limited thereto.

Program modules may be composed of instructions or codes that are executed by at least one processor of the decoding device 1000.

The decryption device 1000 may be implemented in a computer system that includes a recording medium that can be read by a computer.

The recording medium may store at least one module required for the decoding device 1000 to operate.

Functions related to communication of data or information of the decoding device 1000 may be performed through the communication unit 1005.

For example, the communication unit 1005 may receive a bitstream from the encoding device 800.

Figure 11 is a flowchart of a decryption method according to an embodiment.

In step 1105, the communication unit 1005 may receive a bitstream from the encoding device 800.

The bitstream may contain encoded information of the hyperprior.

에서의 양자화된 은닉 표현

의 선택된 요소들의 부호화된 정보를 포함할 수 있다.Bitstream is at target quality level

Quantized hidden representation in

It may contain encoded information of selected elements of .

를 생성할 수 있다.In step 1140, the first entropy decoder 1040 performs decoding on the encoded information of the hyperprior to hide the quantized hyperprior.

can be created.

를 사용하여 끝에서 두 번째(penultimate) 레이어의 출력을 생성할 수 있다.At step 1145, the hyper decoder 1045 generates the quantized hyperprior concealment.

You can use to generate the output of the penultimate layer.

를 사용하여 파라미터를 생성할 수 있다. 파라미터는 스케일 파라미터

를 포함할 수 있다. 파라미터는 중간 파라미터

를 포함할 수 있다.The hyper decoder 1045 hides the quantized hyperprior.

You can create parameters using . The parameter is a scale parameter

may include. Parameter is an intermediate parameter

may include.

In step 1150, the 3D mask generator 1050 may generate a 3D binary mask using the output of a specific layer of the hyper decoder.

Certain layers may be penultimate. The 3D mask generator 1050 may generate a 3D binary mask using the output of the penultimate layer of the hyper decoder.

In embodiments, 3D mask may refer to a 3D binary mask.

가 입력될 수 있다. 하이퍼 복호화기는 양자화된 하이퍼프라이어 은닉

에 대한 복호화를 수행함으로써 끝에서 두 번째(penultimate) 레이어의 출력을 생성할 수 있다.Quantized hyperprior concealment with hyperdecoder

can be entered. The hyperdecoder hides the quantized hyperprior.

By performing decoding on , the output of the penultimate layer can be generated.

가 주어졌을 때, 3D 마스크 생성부(1050)는 끝에서 두 번째(penultimate) 레이어의 출력을 사용하여 목표 품질 수준

에 대한 3D 이진 마스크를 생성할 수 있다.target quality level

Given, the 3D mask generator 1050 uses the output of the penultimate layer to achieve the target quality level.

A 3D binary mask can be created.

가 주어졌을 때, 스케일링 및 선택부(1060)는 3D 이진 마스크 및 파라미터를 사용하여 목표 품질 수준

에 대한 파라미터를 생성할 수 있다.At step 1160, target quality level

Given, the scaling and selection unit 1060 uses the 3D binary mask and parameters to select the target quality level.

You can create parameters for .

를 포함할 수 있다. 파라미터는 중간 파라미터

를 포함할 수 있다.The parameter is a scale parameter

may include. Parameter is an intermediate parameter

may include.

에 대한 파라미터는 목표 품질 수준

에 대한 스케일 파라미터

를 포함할 수 있다.

는

에 기반하여 생성될 수 있다.target quality level

The parameters for the target quality level are

scale parameter for

may include.

Is

It can be created based on .

에 대한 파라미터는 목표 품질 수준

에 대한 중간 파라미터

를 포함할 수 있다.

는

에 기반하여 생성될 수 있다.target quality level

The parameters for the target quality level are

intermediate parameters for

may include.

Is

It can be created based on .

에 대한 파라미터를 사용하여 목표 품질 수준

에서의 양자화된 은닉 표현

의 선택된 요소들의 부호화된 정보에 대한 복호화를 수행함으로써 목표 품질 수준

에서의 양자화된 은닉 표현

의 선택된 요소들의 집합

을 생성할 수 있다.At step 1165, the second entropy decoder 1065 determines the target quality level.

Target quality level using parameters for

Quantized hidden representation in

Target quality level by performing decoding on the encoded information of the selected elements

Quantized hidden representation in

A set of selected elements of

can be created.

에서의 양자화된 은닉 표현

의 선택된 요소들의 집합

을 목표 품질 수준

에서의 3D-형태된(3D-shaped) 은닉 표현의 요소들

로 변환할 수 있다.At step 1180, reshaping unit 1180 uses the 3D binary mask to determine the target quality level.

Quantized hidden representation in

A set of selected elements of

target quality level

Elements of a 3D-shaped hidden representation in

It can be converted to .

에서의 양자화된 은닉 표현

의 선택된 요소들의 집합

은 1D 형태를 가질 수 있다.Here, the target quality level

Quantized hidden representation in

A set of selected elements of

may have a 1D shape.

에서의 3D-형태된(3D-shaped) 은닉 표현의 요소들

에 대한 역-양자화를 수행함으로써 3D-형태된 은닉 표현의 역-양자화된 요소들을 생성할 수 있다.At step 1185, the adaptive dequantizer 1085 determines the target quality level.

Elements of a 3D-shaped hidden representation in

By performing de-quantization on , the de-quantized elements of the 3D-shaped hidden representation can be generated.

에 대하여 수행될 수 있다.Inverse-quantization is the target quality level

It can be performed for.

를 생성할 수 있다.At step 1195, the decoder 1090 performs decoding on the de-quantized elements of the 3D-shaped hidden representation to produce a reconstructed image.

can be created.

The description and processing of information described above in the embodiments may also be applied to the information of the steps described with reference to FIG. 11 .

The above embodiments may be performed in the encoding device 800 and the decoding device 1000 using the same method and/or a corresponding method. Additionally, a combination of one or more of the above embodiments may be used in encoding and/or decoding an image.

The order in which the above embodiments are applied may be different in the encoding device 800 and the decoding device 1000. Alternatively, the order in which the above embodiments are applied may be (at least partially) the same in the encoding device 800 and the decoding device 1000.

In the above-described embodiments, the methods are described based on flowcharts as a series of steps or units, but the present invention is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. You can. Additionally, a person of ordinary skill in the art will recognize that the steps shown in the flowchart are not exclusive and that other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You will understand.

The above-described embodiments include examples of various aspects. Although not all possible combinations for representing the various aspects can be described, those skilled in the art will recognize that other combinations are possible in addition to those explicitly described. Accordingly, the present invention is intended to include all other substitutions, modifications and changes falling within the scope of the following claims.

Embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable by those skilled in the computer software field.

A computer-readable recording medium may contain information used in embodiments according to the present invention. For example, a computer-readable recording medium may include a bitstream, and the bitstream may include information described in embodiments according to the present invention.

A bitstream may contain computer-executable code and/or programs. Computer-executable code and/or program may include information described in the embodiments and may include syntax elements described in the embodiments. That is, the information and syntax elements described in the actual example may be considered computer-executable code within the bitstream, and may be considered at least part of the computer-executable code and/or program represented by the bitstream.

Computer-readable recording media may include non-transitory computer-readable medium.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The above hardware devices may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

In the above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. No, those skilled in the art can make various modifications and changes based on this description.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all modifications equivalent to or equivalent to the scope of the claims shall fall within the scope of the spirit of the present invention. It will be said that it belongs.

Claims (20)

generating a hidden representation using the input image;
generating a quantized hidden representation by performing adaptive quantization on the hidden representation;
deriving a set of selected elements of the quantized hidden representation; and
Generating encoded information of the selected elements by performing entropy encoding on the set of selected elements
A video encoding method including. According to paragraph 1,
An image encoding method in which the quantized hidden representation is generated for a specific target quality level. According to paragraph 1,
An image encoding method in which the set of selected elements is determined using a 3D binary mask. According to paragraph 3,
An image encoding method in which the 3D binary mask is generated using the output of a specific layer of a hyper decoder. According to paragraph 4,
A video encoding method in which a hyperprior is input to the hyper decoder. According to paragraph 1,
A video encoding method in which the encoded information of the selected elements is generated using parameters for a specific target quality level. According to clause 6,
The parameter is a video encoding method including a scale parameter for the specific target quality level or an intermediate parameter for the specific target quality level. generating a set of selected elements of the quantized hidden representation by performing decoding on encoded information of the selected elements;
converting the selected set of elements into elements of a three-dimensional-shaped hidden representation;
generating de-quantized elements by performing de-quantization on elements of the 3D-shaped hidden representation; and
Generating a restored image by performing decoding on the dequantized elements.
A video decoding method including. According to clause 8,
An image decoding method in which the inverse quantization is performed for a specific target quality level. According to clause 8,
An image decoding method wherein elements of the 3D-shaped hidden representation are determined using a 3D binary mask. According to clause 10,
An image decoding method in which the 3D binary mask is generated using the output of a specific layer of a hyper decoder. According to clause 11,
A video decoding method in which a hyper fryer is input to the hyper decoder. According to clause 8,
A video decoding method in which the set of selected elements is information generated using parameters for a specific target quality level. According to clause 13,
The video decoding method wherein the parameters include a scale parameter for the specific target quality level or an intermediate parameter for the specific target quality level. In the computer-readable recording medium storing a bitstream for video decoding, the bitstream includes:
Encoded information of selected elements of the quantized hidden representation
Including,
A set of the selected elements is generated by performing decoding on the encoded information,
Converting the set of selected elements into elements of a 3D-shaped hidden representation,
Dequantized elements are generated by performing dequantization on elements of the three-dimensional hidden representation,
A computer-readable recording medium in which a restored image is generated by decoding the dequantized elements. According to clause 15,
A computer-readable recording medium wherein the de-quantization is performed for a specific target quality level. According to clause 15,
A computer-readable recording medium wherein elements of the three-dimensionally-shaped hidden representation are determined using a three-dimensional binary mask. According to clause 17,
A computer-readable recording medium in which the 3D binary mask is generated using the output of a specific layer of a hyper decoder. According to clause 15,
The set of selected elements is a computer-readable recording medium on which information is generated using parameters for a specific target quality level. According to clause 19,
The computer-readable recording medium wherein the parameter includes a scale parameter for the specific target quality level or an intermediate parameter for the specific target quality level.

Images