KR20200035879A - Method and apparatus for image processing using context-adaptive entropy model - Google Patents

Abstract

Provided is a context-adaptive entropy model for end-to-end optimized image compression. According to an embodiment, the model utilizes two types of contexts. The contexts are bit-consuming contexts and bit-free contexts, and the contexts are classified according to whether or not the contexts require additional bit allocation. The model can more accurately estimate the distribution of each hidden expression component while having a more generalized form of approximation models based on the contexts, thereby improving compression performance.

Description

Method and method for image processing using a context-adaptive entropy model {METHOD AND APPARATUS FOR IMAGE PROCESSING USING CONTEXT-ADAPTIVE ENTROPY MODEL}

The following embodiments relate to a video decoding method, a decoding apparatus, a coding method, and a coding apparatus, and related to a decoding method, a decoding apparatus, a coding method, and a coding apparatus using a context-adaptive entropy model.

Recently, Artificial Neural Networks (ANNs) have reached a variety of domains and have achieved many breakthroughs due to their excellent optimization and efficacy learning performance.

In particular, a number of ANN-based studies have been conducted and large progress has been made on various problems that can be solved in a short period of time by a human being because they are intuitive.

However, for image compression, only relatively slow progress has been made due to the complex target problems of image compression.

Furthermore, due to the decades of standardization history of traditional codecs and the tremendous amount of heuristics accumulated through firm structures, most of the work related to image compression is reconstructed. The emphasis is on improving the quality of reconstructed images.

For example, several approaches have proposed a method for reducing image compression artifacts depending on the superior image restoration capabilities of ANNs.

Although there is no doubt that artifact reduction is one of the most promising areas in exploiting the benefits of ANNs, these approaches appear to be a form of post-processing rather than image compression itself. Can lose.

One embodiment may provide an ANN-based encoding apparatus, encoding method, decoding apparatus, and decoding method that exhibits better performance than traditional video codecs.

In one side, performing an entropy encoding using an entropy model on the input image to generate a bitstream; And transmitting or storing the bitstream.

The entropy model may be a context-adaptive entropy model.

The context-adaptive entropy model can utilize multiple different types of contexts.

The plurality of different types of the contexts may include a bit-consumption context and a bit-free context.

The standard deviation parameter and the average value parameter of the entropy model can be estimated from a plurality of different types of the contexts.

The input to the analytic transformation of the context-adaptive entropy model may be uniformly quantized expression components.

The entropy model can be based on a Gaussian model with mean value parameters.

The entropy model may include a context-adaptive entropy model and a lightweight entropy model.

The hidden expression component used in the entropy model may be divided into a first partial hidden expression component and a second partial hidden expression component.

The first partial hidden expression component may be quantized as a first quantized partial hidden expression component.

The second partial hidden expression component may be quantized as a second quantized partial hidden expression component.

The first quantized partial hidden expression component may be coded using the context-adaptive entropy model.

The second quantized partial hidden expression component may be coded using the lightweight entropy model.

The lightweight entropy model can utilize scale estimation.

The lightweight entropy model can detect standard deviations estimated directly from the analytical transformation.

On the other side, the communication unit for obtaining a bitstream; And a processing unit that performs decoding using the entropy model on the bitstream to generate a reconstructed image.

In another aspect, obtaining a bitstream; And generating a reconstructed image by performing decoding using an entropy model on the bitstream.

The entropy model may be a context-adaptive entropy model.

The context-adaptive entropy model can utilize multiple different types of contexts.

The plurality of different types of the contexts may include a bit-consumption context and a bit-free context.

The standard deviation parameter and the average value parameter of the entropy model can be estimated from a plurality of different types of the contexts.

The input to the analytic transformation of the context-adaptive entropy model may be uniformly quantized expression components.

The entropy model can be based on a Gaussian model with mean value parameters.

The entropy model may include a context-adaptive entropy model and a lightweight entropy model.

The lightweight entropy model can utilize scale estimation.

The lightweight entropy model can detect standard deviations estimated directly from the analytical transformation.

An ANN-based encoding apparatus, encoding method, decoding apparatus, and decoding method showing better performance than traditional video codecs are provided.

1 shows an operation of a convolution layer according to an example.
2 shows an operation of a pooling layer according to an example.
3 shows an operation of a deconvolution layer according to an example.
4 shows an operation of an un-pooling layer according to an example.
5 shows an operation of a Relu layer according to an example.
6 shows an automatic encoder according to an example.
7 shows a convolutional encoder and a convolutional decoder according to an example.
8 shows an encoder according to an embodiment.
9 shows a decoder according to the embodiment.
10 shows an implementation of an automatic encoder according to an embodiment.
11 shows the structure of a hybrid network for higher bit-rate environments according to an example.
12 is a structural diagram of an encoding device according to an embodiment.
13 is a structural diagram of a decoding apparatus according to an embodiment.
14 is a flowchart of an encoding method according to an embodiment.
15 is a flowchart of a decoding method according to an embodiment.

The present invention can be applied to various changes and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

For detailed description of exemplary embodiments described below, reference is made to the accompanying drawings showing specific embodiments as examples. These embodiments are described in detail enough to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different, but need not be mutually exclusive. For example, the specific shapes, structures, and properties described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. In addition, it should be understood that the location or placement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the embodiment. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed.

In the drawings, similar reference numerals refer to the same or similar functions across various aspects. The shape and size of elements in the drawings may be exaggerated for a clearer explanation.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component. The term and / or may include a combination of a plurality of related described items or any one of a plurality of related described items.

When it is mentioned that a component is "connected" or "connected" to another component, the two components may be directly connected to each other or may be connected, but the above 2 It should be understood that other components may exist in the middle of the dog components. On the other hand, when it is mentioned that a component is "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle of the two components. something to do.

Components shown in the embodiments of the present invention are shown independently to represent different characteristic functions, and do not mean that each component is composed of separate hardware or one software component unit. That is, each component is included as being listed as each component for convenience of description, and at least two components of each component are combined to form one component, or one component is divided into a plurality of components to function. It can be carried out and the integrated and separated embodiments of each of these components are also included in the scope of the present invention without departing from the essence of the present invention.

In addition, in the exemplary embodiments, the description that "includes" a specific configuration does not exclude configurations other than the specific configuration described above, and additional configurations may be used to implement the exemplary embodiments or the technical spirit of the exemplary embodiments. It means that it can be included in the scope.

The terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present invention, terms such as “include” or “have” are intended to indicate that there are features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance. That is, in the present invention, a description that "includes" a specific configuration does not exclude a configuration other than the configuration, and means that additional configurations may also be included in the scope of the present invention or the technical spirit of the present invention.

Some of the components of the present invention are not essential components for performing essential functions in the present invention, but may be optional components for improving performance. The present invention may be implemented by including only essential components for implementing the essence of the present invention, except for components used for performance improvement. A structure including only essential components excluding optional components used for performance improvement is also included in the scope of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings in order to enable those skilled in the art to easily implement the embodiments. In describing the embodiments, when it is determined that a detailed description of related known configurations or functions may obscure the subject matter of the present specification, the detailed description will be omitted. In addition, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

Hereinafter, an image may mean one picture constituting a video, or may represent the video itself. For example, "encoding and / or decoding of an image" may mean "encoding and / or decoding of a video", and "encoding and / or decoding of one of the images constituting the video". It might be.

Hereinafter, the term "encoding" may be used as a meaning of coding.

`

Convolution layer

1 shows an operation of a convolution layer according to an example.

The convolution layer may perform filtering on the input frame, and output a feature map as a result of filtering. The feature map can be used as input to the next layer. With this structure, the input frame can be continuously processed by a plurality of layers.

In the convolution layer, the kernel may refer to a filter that performs convolution operation or filtering. The size of the kernel can be referred to as the kernel size or filter size. The operation parameters constituting the kernel may also be referred to as weights, kernel parameters, or filter parameters.

In the convolutional layer, different types of filters can be used for one input. At this time, a process in which one filter processes an input may be referred to as a convolution channel.

As shown in FIG. 1, the convolution layer can reduce samples as many as the size of the kernel into one sample. In FIG. 1, the size of the illustrated kernel may be 3x3. In other words, in FIG. 1, a process in which a convolution operation is performed by a filter having a kernel size of 3x3 in one channel is illustrated.

In FIG. 1, an operation may be performed on a rectangle having a dark border in the input image. At this time, the window (window) may mean an operation area such as a rectangle with a dark border. The window can be moved one space from the upper left to the lower right of the frame, and the size of the movement can be adjusted.

Stride and padding can be used for filters of convolution operations.

Stride may mean the size of the movement. The value of the stride illustrated in FIG. 18 may be 1. When the stride value is 2, operations may be performed on windows opened at intervals of 2 columns.

Padding may be to make the input image larger, and may mean filling specific values at the top, bottom, left, and right sides of the input image.

Pulling layer

2 shows an operation of a pooling layer according to an example.

Pulling may refer to subsampling of a feature map obtained by calculation in a convolutional layer.

As shown in FIG. 2, the pooling layer can select a representative sample for samples of a specified size passing through the pooling layer.

For pooling, the size of the stride and the size of the window may generally be the same.

Pooling may include max pooling and average pooling.

Maximum pooling may be to select a sample having a maximum value among samples of a specified size as a representative sample. For example, for samples of 2x2, the maximum value of the samples can be selected as a representative sample.

The average pooling may be to set an average value of samples of a specified size as a representative sample.

The pooling layer illustrated in FIG. 2 may perform maximum pooling. For example, the pooling layer may select one sample from samples of a window having a size of 2x2. With this selection, the horizontal and vertical of the output from the pooling layer can be half the horizontal and vertical of the input to the pooling layer.

As illustrated in FIG. 2, the size of the stride and the window can all be set to 2. For example, when values of [h, w, n] are input to the pooling layer, the size of values output through the pooling layer may be [h / 2, w / 2, n].

Deconvolution layer

3 shows an operation of a deconvolution layer according to an example.

The deconvolution layer may perform an operation in a direction opposite to that of the operation of the convolution layer. Except for the direction, the operation of the convolution layer and the operation of the deconvolution layer can be considered to be the same.

The deconvolution layer may perform a convolution operation on the input feature map, and output a frame through the convolution operation.

The size of the output frame can be changed according to the value of stride. For example, when the value of stride is 1, the horizontal and vertical sizes of the frame may be the same as the horizontal and vertical sizes of the feature map. When the value of the stride is 2, the horizontal and vertical sizes of the frame may be 1/2 of the horizontal and vertical sizes of the feature map.

Unpooling layer

4 shows an operation of an un-pooling layer according to an example.

The un-pooling layer may perform up-sampling in a direction opposite to the direction of pooling in the pooling layer. The un-pooling layer, as opposed to the pooling layer, may perform a function of expanding a dimension. In other words, the un-pooling layer, as opposed to the pooling layer, can enlarge the sample passing through the un-pooling layer to samples of a specified size. For example, a sample passing through an un-pooling layer can be enlarged to samples of a 2x2 window.

For example, when values of the size [h, w, n] are input to the un-pooling layer, the size of values output through the un-pooling layer may be [h * 2, w * 2, n]. have.

Nonlinear operation layer

5 shows an operation of a Relu layer according to an example.

An example of values input to the relu layer is illustrated on the left side of FIG. 5, and an example of values output from the Relu layer is illustrated on the right side of FIG. 5.

The Relu layer may perform a nonlinear operation as illustrated in FIG. 5. In embodiments, the Relu layer may be replaced with a nonlinear computational layer.

The Relu layer may generate output values by applying a transfer function to input values.

The size of values input to the Relu layer and the size of values output from the Relu layer may be the same. In other words, the size of values passing through the Relu layer may not change.

Autoencoder

6 shows an automatic encoder according to an example.

The automatic encoder may have a structure as shown in FIG. 6, and may be widely used for unsupervised learning.

Convolutional encoders and convolutional decoders can be derived from automatic encoders.

According to the structure of the automatic encoder, the dimension of the input and the dimension of the output may be the same. The purpose of the automatic encoder may be to perform learning on f () such that f (X) = X holds. X may be an input value. In other words, the purpose of the automatic encoder may be to approximate the output predicted value X 'to the input value X.

The automatic encoder may include an encoder and a decoder. The encoder can provide a code or latent variable as an output value for the input value X. The code can be used as a feature vector for the input value X. The code can be entered into the decoder. The decoder can output the predicted value X 'formed from the code.

Convolutional encoder and convolutional decoder

7 shows a convolutional encoder and a convolutional decoder according to an example.

The structures of the convolutional encoder and the convolutional decoder may consist of a pair of convolutional layers and deconvolutional layers. Convolutional encoders and convolutional decoders can provide inputs, feature vectors and outputs similar to automatic encoders.

The convolutional encoder may include a convolutional layer and a pooling layer. The input to the convolutional encoder may be a frame, and the output from the convolutional encoder may be a feature-map.

The convolutional decoder may include a deconvolution layer and an un-pooling layer. The input to the convolution decoder can be a feature map, and the output from the convolution decoder can be a (rebuilt) frame.

Convolutional features may be reflected in structures of the convolutional encoder and the convolutional decoder. According to this reflection, the convolutional encoder and the convolutional decoder can have a smaller learning weight. The convolutional encoder and the convolutional decoder can be particularly useful when operations are performed under the purpose of optical flow and counter edge for the output frame.

Convolutional encoders can reduce dimensions by utilizing convolution and pooling, and generate feature vectors from frames. The feature vector can be generated at the output stage of the convolutional encoder.

The feature vector may be a vector representing characteristics of the original signal in a dimension lower than that of the original signal.

The convolution decoder can rebuild the frame from the feature vector using deconvolution and un-pooling.

ANN-based video compression

With regard to ANN-based image compression, the proposed methods can be divided into two streams.

First, as a result of the success of generative models, several image compression approaches have been proposed that target perceptual quality. The basic idea of these approaches is in learning the distribution of natural images, by allowing the creation of image components, such as textures, that do not significantly affect the structure or perceived quality of the reconstructed image, which is serious. It enables very high compression without cognitive loss.

However, although the images generated by this approach are very realistic, the acceptability of machine-created image components may eventually be somewhat application-dependent. .

On the other hand, second, without using generation models, end-to-end optimized ANN-based approaches can be used.

In this approach, unlike traditional codecs made up of individual tools such as prediction, transformation and quantization, a comprehensive solution that covers the entire functionality through end-to-end optimization Can be provided.

For example, one approach can utilize a small amount of binary latent representations to contain compressed information at all stages. Each step can further accumulate additional hidden expression components to achieve progressively improving quality.

Other approaches can improve compression performance by improving the network structure of the above-described approach.

These approaches can provide new frameworks suitable for quality control through a single trained network. In these approaches, increasing the number of iteration steps can be a burden for some applications.

These approaches can extract binary expression components with the highest entropy possible. On the other hand, other approaches regard the image compression problem as how to detect discrete latent representations with as low entropy as possible.

In other words, the target problem of the former approaches can be regarded as how to include as much information as possible within a fixed number of expression elements, whereas the target problem of the latter approaches is only when a sufficient number of expression elements are given. It can be regarded as how to reduce the expected bit-rate. Here, it can be assumed that the low entropy corresponds to the low bit-rate by entropy coding.

To solve the target problem of the latter approaches, the approaches can employ their own entropy models to approximate the actual distribution of discrete concealment representations.

For example, some approaches can propose new frameworks that utilize entropy models, and the performance of entropy models can be demonstrated by comparing the results produced by entropy models with existing codecs such as JPEG2000. .

In these approaches, it can be assumed that each expression component has a fixed distribution. For the approach, an input-adaptive entropy model that estimates the scale of the distribution for each expression component can be used. This approach can be based on the nature of natural images that the scales of the expression components change together in adjacent regions.

One of the main elements of end-to-end optimization image compression can be a trainable entropy model for hidden representations.

Since the actual distributions of the hidden expression components are unknown, entropy models can calculate estimated bits for encoding the hidden expression components by approximating the distributions of the hidden expression components.

가 은닉 표현성분

로 변환(transform)되고, 은닉 표현성분

가 로 양자화된 은닉 표현성분

로 균일하게 양자화될 때, 단순한 엔트로피 모델은

로 표현될 수 있다.Input image

A hidden expression ingredient

Transformed to, and hidden expression component

Hidden quantization component quantized with

When quantized uniformly, a simple entropy model

Can be expressed as

는

의 실제의 한계(marginal) 분포를 나타낼 수 있다. 엔트로피 모델

을 사용하는 교차(cross) 엔트로피를 통해 계산된 율 추정(rate estimation)은 아래의 수학식 1과 같이 표현될 수 있다.

The

Can represent the actual marginal distribution of. Entropy model

The rate estimation calculated through cross entropy using can be expressed as Equation 1 below.

의 실제의 엔트로피 및 추가의 비트들로 분해될 수 있다. 말하자면, 율 추정은

의 실제의 엔트로피 및 추가의 비트들을 포함할 수 있다.Rate estimation

The actual entropy of and can be decomposed into additional bits. In other words, rate estimation

The actual entropy of and may include additional bits.

Additional bits may be due to a mismatch between the actual distributions and estimates for these actual distributions.

이 감소하면, 엔트로피 모델

및 근사

가 가능한 가까워질 수 있으며, 또한

의 실제의 엔트로피가 작게 되도록 다른 파라미터들이

를

로 원활하게 변환할 수 있다.Thus, during the process of training, rate terms

When it decreases, the entropy model

And approximation

Can be as close as possible, and also

Other parameters to make the actual entropy of

To

Can be converted smoothly.

은

가 실제의 분포

와 완벽하게 매치될 때 최소화될 수 있다. 이는, 상기의 방법들의 압축 성능이 본질적으로 엔트로피 모델의 성능에 의존한다는 것을 의미할 수 있다.In terms of Kullback-Leibler (KL) -divergence,

silver

Is the actual distribution

And can be minimized when matched perfectly. This may mean that the compression performance of the above methods essentially depends on the performance of the entropy model.

In an embodiment, a new entropy model can be proposed that utilizes two types of contexts to improve performance. The two types of context can be a bit-consuming context and a bit-free context.

The bit-consumption context and the bit-free context can be divided according to whether the context requires additional bit allocation for transmission.

Using these contexts, the proposed entropy model can more accurately estimate the distribution of each hidden expression component using a more general form of entropy models. In addition, the proposed entropy model can more efficiently reduce spatial dependencies between adjacent hidden expression components through such accurate estimation.

The following effects can be achieved by the embodiments to be described later.

A new context-adaptive entropy model framework can be provided that incorporates two different types of contexts.

-Improving directions of the method of the embodiment in terms of model capacity and level of contexts can be described.

-In the domain of ANN-based image compression, in terms of peak signal-to-noise ratio (PSNR), test results can be provided that outperform conventional image codecs that are widely used in performance.

In addition, the following descriptions of the embodiments may be described later.

1) Key approaches of end-to-end optimized image compression are introduced, and a context-adaptive entropy model can be proposed.

2) The structure of the encoder and decoder models can be described.

3) The setup of the experiment and the results of the experiment can be provided.

4) The current state of the embodiments and directions for improvement can be described.

End-to-end optimization based on context-adaptive entropy model

Entropy models

The entropy models of the embodiment may approximate the distribution of discrete concealment expression components. Through this approximation, entropy models can improve image compression performance.

Some of the entropy models of the embodiment can be assumed to be non-parametric models, and the other is a zero-mean Gaussian model with six weights per expression component. It may be a Gaussian scale mixed model consisting of.

Even if it is assumed that the types of entropy models are different from each other, entropy models may have a common characteristic of focusing on learning distributions of expression components without considering input adaptability. In other words, once the entropy model is trained, the models trained on the expression components can be fixed for any input during the test time.

On the other hand, a specific entropy model may employ input-adaptive scale estimation for expression components. In this entropy model, the assumption that the scales of hidden expression components from natural images tend to move together in adjacent regions can be applied.

To reduce this redundancy, entropy models can use small amounts of additional information. Additional information can be estimated as appropriate scale parameters (eg, standard deviations) of the hidden expression components.

In addition to scale estimation, entropy models are rounded when the Probability Density Function (PDF) for each expression component in a continuous domain is convolved with a standard uniform density function. ) Can be approximated more closely to the dictionary Probability Mass Function (PMF) of the discrete concealment expression component uniformly quantized by.

For training, uniform noise can be added to each hidden expression component. This addition may be to fit the distribution of noisy expression components to the mentioned PMF-approximation functions.

With these approaches, the entropy model can achieve state-of-the-art compression performance similar to BPG.

Spatial dependencies of hidden variables

When the hidden expression components are transformed through a convolutional neural network, the same convolution filters are shared across spatial regions, and natural images share a variety of factors within adjacent regions. Because of this, hidden expression components can essentially contain spatial dependencies.

In the entropy model, these spatial dependencies can be successfully captured and compression performance can be improved by input-adaptive estimation of the standard deviations of the hidden expression components.

Going one step further, in addition to the standard deviation, the form of the distribution estimated through mean (mu) estimation using contexts can be generalized.

For example, assuming that certain expression components tend to have similar values in spatially adjacent regions, when all neighboring expression components have a value of 10, the current expression component has 10 or similar values. It can be intuitively assumed that the likelihood is relatively high. Thus, this simple estimation can reduce entropy.

Similarly, the entropy model according to the method of the embodiment can use a given context to estimate mu and standard deviation of each hidden expression component.

Alternatively, the entropy model can perform context-adaptive entropy coding by estimating the probability of each binary expression component.

However, this context-adaptive entropy coding does not directly contribute to the rate term of the Rate-Distortion (RD) optimization framework, since the probability estimation of entropy coding does not directly contribute to end-to-end. It can be seen as separate components rather than one of the optimization components.

및 이러한 은닉 변수들의 정규화된 버전들이 예시될 수 있다. 앞서 언급된 문맥들의 2 개의 타입들을 가지고, 하나의 접근방식에서는 단지 표준 편차 파라미터들이 추정될 수 있고, 다른 하나의 접근방식에서는 mu 및 표준 편차 파라미터들의 양자가 추정될 수 있다. 이 때, 주어진 문맥들을 가지고 mu가 함께 추정될 때 공간적 의존성은 더 효율적으로 제거될 수 있다.Hidden variables of two different approaches

And normalized versions of these hidden variables. With the two types of contexts mentioned above, only standard deviation parameters can be estimated in one approach, and both mu and standard deviation parameters can be estimated in another approach. At this time, when mu is estimated together with given contexts, spatial dependence can be removed more efficiently.

Context-adaptive entropy model

는 낮은 엔트로피를 갖는 은닉 표현성분

로 변환될 수 있고,

의 공간적 의존성들은

로 포착될 수 있다. 따라서, 4 개의 주요한 파라미터의(parametric) 변환 함수들이 사용될 수 있다. 엔트로피 모델의 4 개의 파라미터의 변환 함수들은 아래의 1) 내지 4)와 같다.In the optimization problem in the embodiment, the input image

Is a secret expression component with low entropy

Can be converted to

The spatial dependencies of

Can be captured by Thus, four major parametric transformation functions can be used. The transformation functions of the four parameters of the entropy model are as follows 1) to 4).

를 은닉 표현성분

로 변환하기 위한 분석 변환

One)

The secret expression component

Analytical transformation to convert to

를 생성하기 위한 합성(synthesis) 변환

2) Reconstructed video

Synthesis transformation to generate

의 공간적 중복성들을 은닉 표현성분

로 포착(capture)하기 위한 분석 변환

2)

Components that hide the spatial redundancies of

Analytic transformation to capture with

4) Composite transformation to generate contexts for model estimation

는 표현성분들의 표준 편자들을 직접적으로 추정하지 않을 수 있다. 대신, 실시예에서,

는 분포를 추정하기 위해 문맥들의 2 개의 타입들 중 하니인 문맥

을 생성할 수 있다. 문맥들의 2 개의 타입들에 대해서는 아래에서 설명된다.In an embodiment,

May not directly estimate the standard deviations of the expression components. Instead, in an embodiment,

Is one of two types of contexts to estimate the distribution

You can create Two types of contexts are described below.

Optimization problems can be analyzed from a viewpoint of a variational autoencoder, and minimization of KL-emission can be regarded as the same problem as R-D optimization of image compression. Basically, the same concept can be employed in the embodiment. However, in training, in embodiments, discrete expression components for conditions can be used instead of the noisy expression components, so the noisy expression components can be used only as inputs to entropy models.

Empirically, using discrete expressions for conditions can yield better results. These results can come from eliminating discrepancies in conditions between training time and testing time, and improving training capacity by eliminating these discrepancies. The training capacity can be improved by limiting the effect of uniform noise to only assist in approximation to probability mass functions.

In an embodiment, a gradient overriding method with an identity function can be used to address discontinuities from uniform quantization. The objective functions that are used in the examples were described in Equation 2 below.

In Equation 2, the total loss includes two terms. The two terms show ratios and distortions. That is, the total loss may include a rate term R and a distortion term D.

는 R-D 최적화 동안의 율 및 왜곡 간의 균형(balance)을 제어할 수 있다.Coefficient

Can control the balance between rate and distortion during RD optimization.

가 변환

의 결과이고,

가 변환

의 결과일 때,

및

의 노이즈가 낀 표현성분은 표준 균일 분포를 따를 수 있다. 여기에서,

의 평균값은

일 수 있고,

의 평균값은

일 수 있다. 또한,

로의 입력은, 노이즈 낀 표현성분

가 아니라,

일 수 있다.

는 라운딩 함수

에 의한

의 균일하게 양자화된 표현성분들일 수 있다.From here,

Fall conversion

Is the result of,

Fall conversion

When the result of

And

The noisy expression component of can follow the standard uniform distribution. From here,

The average value of

Can be,

The average value of

Can be Also,

The input to the furnace is a noisy expression component

Not,

Can be

Is a rounding function

On by

It may be uniformly quantized expression components of.

및

의 엔트로피 모델들을 가지고 계산된 예상되는 비트들을 나타낼 수 있다.

는 궁극적으로

의 근사일 수 있고,

는 궁극적으로

의 근사일 수 있다.Rate terms

And

Can represent the expected bits calculated with the entropy models of.

Ultimately

May be an approximation of

Ultimately

It can be an approximation of

에 대한 요구되는 비트들의 근사를 위한 엔트로피 모델을 나타낼 수 있다. 수학식 4는 엔트로피 모델에 대한 공식적인(formal) 표현성분일 수 있다.Equation 4 below

It may represent an entropy model for approximation of the required bits for. Equation 4 may be a formal expression component for the entropy model.

뿐만 아니라, mu 파라미터

도 갖는 가우시안 모델에 기반할 수 있다.The entropy model is a standard deviation parameter

In addition, the mu parameter

It can also be based on a Gaussian model.

및

는 함수

에 의해 주어진 문맥들의 2 개의 타입들로부터 결정적 방식으로 추정될 수 있다. 함수

는 분포 추정자(estimator)일 수 있다.

And

Is a function

Can be estimated in a deterministic manner from the two types of contexts given by. function

Can be a distribution estimator.

및

로 표시될 수 있다.The two types of contexts can be bit-consumption context and bit-free context. Here, the two types of contexts for estimating the distribution of certain expression components are

And

It may be indicated by.

는

로부터

를 추출할 수 있다.

는 변환

의 결과일 수 있다. Extractor

The

from

Can be extracted.

Convert

May be the result of

와는 대조적으로,

에 대해서는 어떤 추가 비트 할당도 요구되지 않을 수 있다. 대신,

의 알려진(이미 엔트로피-부호화되거나, 엔트로피-복호화된) 서브세트가 활용될 수 있다. 이러한

의 알려진 서브세트는

로 표시될 수 있다.

In contrast to,

For, no additional bit allocation may be required. instead,

A known (already entropy-encoded or entropy-encoded) subset of can be utilized. Such

A known subset of

It may be indicated by.

는

로부터

를 추출할 수 있다.Extractor

The

from

Can be extracted.

를 처리할 수 있다. 따라서, 동일한

를 처리함에 있어서, 엔트로피 부호기 및 엔트로피 복호기에게 주어지는

는 언제나 동일할 수 있다.The entropy encoder and the entropy decoder are sequentially in the same specific order, such as raster scanning.

Can handle Therefore, the same

In processing, it is given to the entropy encoder and the entropy decoder.

Can always be the same.

의 경우에는, 단순한 엔트로피 모델이 사용될 수 있다. 이러한 단순한 엔트로피 모델은 훈련가능한

를 가진 제로-평균값 가우시안 분포들을 따르는 것으로 가정될 수 있다.

In the case of, a simple entropy model can be used. This simple entropy model is trainable

It can be assumed to follow the zero-average Gaussian distributions with.

는 부가 정보(side information)로 간주될 수 있으며,

는 총 비트-레이트의 매우 적은 양에 기여할 수 있다. 따라서, 실시예에서는, 더 복잡한 엔트로피 모델들이 아닌, 엔트로피 모델의 단순화된 버전이 제안된 방법의 전체의 파라미터들 상의 엔드-투-엔드 최적화를 위해 사용될 수 있다.

Can be considered as side information,

Can contribute to a very small amount of total bit-rate. Thus, in an embodiment, rather than more complex entropy models, a simplified version of the entropy model can be used for end-to-end optimization on the overall parameters of the proposed method.

Equation 5 below shows a simplified version of the entropy model.

The rate term is not the actual amount of bits, but may be an estimate calculated from entropy models as mentioned. Therefore, in training or encoding, actual entropy encoding or entropy decoding processes may not necessarily be required.

가 널리-사용되는 왜곡 메트릭스들(metrics)로서 가우시안 분포들을 따른다고 가정될 수 있다. 이러한 가정 하에서, 왜곡 항은 평균 제곱된 에러(Mean Squared Error; MSE)를 사용하여 계산될 수 있다.Regarding the distortion term,

Can be assumed to follow Gaussian distributions as widely-used distortion metrics. Under this assumption, the distortion term can be calculated using Mean Squared Error (MSE).

Encoder-decoder model

8 shows an encoder according to an embodiment.

In FIG. 8, the small icons on the right may indicate an entropy-encoded bitstream.

는 균일 잡음 추가 또는 균일 양자화를 나타낼 수 있다.In FIG. 8, EC may indicate entropy coding (ie, entropy encoding).

Can represent uniform noise addition or uniform quantization.

In addition, in FIG. 8, the noisy expression components are shown as dotted lines. In an embodiment, noisy expression components can be used for training only as input to entropy models.

The operations and interactions of the encoder 800 and decoder 900 are described in more detail below.

9 shows a decoder according to the embodiment.

In FIG. 9, the small icons on the left may represent an entropy-encoded bitstream.

ED may indicate entropy decoding.

The operations and interactions of the encoder 800 and decoder 900 are described in more detail below.

The encoder 800 may convert the input image into hidden expression components. The encoder can generate quantized hidden expression components by quantizing the hidden expression components. Also, the encoder can generate entropy-encoded hidden expression components by performing entropy-encoding using a trained entropy model for quantized hidden expression components, and can output entropy-coded hidden expression components as a bitstream. have.

The trained entropy model can be shared between the encoder 800 and the decoder 900. In other words, a trained entropy model can also be referred to as a shared entropy model.

 On the other hand, the decoder 900 may receive entropy-encoded hidden expression components through a bitstream. The decoder 900 may generate hidden expression components by performing entropy-decoding using a shared entropy model for the entropy-encoded hidden expression components. The decoder 900 may generate a reconstructed image using hidden expression components.

For encoder 800 and decoder 900, all parameters can be assumed to have already been trained.

및

를 포함할 수 있다.

는

의

로의 변환을 담당할 수 있으며,

는

의 변환에 대한 역변환(inverse transform)을 담당할 수 있다.The structure of the encoder-decoder model is basically

And

It may include.

The

of

Can be responsible for conversion to,

The

In charge of the inverse transform (inverse transform) for the transformation.

는 라운딩에 의해

로 균일하게 양자화될 수 있다.Converted

By rounding

It can be quantized uniformly.

Here, unlike existing codecs, in the case of approaches based on entropy models, tuning for quantization steps may generally be unnecessary because the scales of the expression components are optimized together by training.

및

의 사이의 다른 구성요소들은 1) 공유된 엔트로피 모델들 및 2) 기저에 있는(underlying) 문맥 준비(preparation) 프로세스들을 가지고 엔트로피 부호화(또는, 엔트로피 복호화)의 역할을 수행할 수 있다.

And

Other components in between 1) can perform the role of entropy encoding (or entropy decoding) with 1) shared entropy models and 2) underlying context preparation processes.

의 분포를 개별적으로 추정할 수 있다. 각

의 분포의 추정에 있어서,

및

는 주어진 문맥들의 2 개의 타입들인

및

을 가지고 추정될 수 있다.More specifically, each entropy model

The distribution of can be estimated individually. bracket

In the estimation of the distribution of,

And

Is two types of given contexts

And

It can be estimated with.

는 추가의 비트 할당을 요구하는 부가 정보일 수 있다.

를 운반하기 위해 요구되는 비트-레이트를 감소시키기 위해,

로부터 변환된 은닉 표현성분

는

자신의 엔트로피 모델에 의해 양자화 및 엔트로피-부호화될 수 있다.Among these contexts,

May be additional information requiring additional bit allocation.

In order to reduce the bit-rate required to transport,

Expression component transformed from

The

It can be quantized and entropy-coded by its entropy model.

는 어떤 추가의 비트 할당 없이

로부터 추출될 수 있다.On the other hand,

Without any additional bit allocation

It can be extracted from.

는 엔트로피 부호화 또는 엔트로피 복호화 진행함에 따라 변할 수 있다. 그러나,

는 동일한

를 처리함에 있어서 언제나 부호기(800) 및 복호기(900)의 양자 내에서 동일할 수 있다.From here,

May change as entropy encoding or entropy decoding proceeds. But,

Is the same

In processing, it can always be the same within both the encoder 800 and the decoder 900.

의 파라미터들 및 엔트로피 모델들은 부호기(800) 및 복호기(900)의 양자에 의해 단순하게 공유될 수 있다. 도 8의 점선으로 도시된 것과 같이 훈련이 진행되는 동안 엔트로피 모델들로의 입력들은 노이즈 낀 표현성분들일 수 있다. 노이즈 낀 표현성분들은 엔트로피 모델이 이산 표현성분들의 확률 질량 함수들에 근사하도록 할 수 있다.

The parameters and entropy models of can be simply shared by both the encoder 800 and the decoder 900. As illustrated by the dotted line in FIG. 8, inputs to entropy models during training may be noisy expression components. The noisy expression components can cause the entropy model to approximate the probability mass functions of the discrete expression components.

10 shows an implementation of an automatic encoder according to an embodiment.

또한 콘볼루션 신경 네트워크들과 함께 구현될 수 있다.The structures of the above-described encoder 800 and decoder 900 of FIG. 10 may be represented as an automatic encoder 1000. That is, in the embodiment, for the encoder 800 and the decoder 900, a convolution automatic encoder structure may be used, and the distribution estimator

It can also be implemented with convolutional neural networks.

In Figure 10, convolution is abbreviated as "conv". "GDN" may indicate generalized divisive normalization. "IGDN" may indicate inverse generalized divisive normalization.

In FIG. 10, leakyReLU may be a function that is a variation of ReLU, and may be a function in which the degree of leakage is specified. The first set value and the second set value may be set for the leakyReLU function. The leakyReLU function may output the input value and the second setting value without outputting the first setting value when the input value is less than or equal to the first setting value.

필터 높이

필터 폭 (/ 다운스케일 또는 업스케일의 팩터(factor)).In addition, the notations for the convolutional layer used in FIG. 10 may be as follows: number of filters

Filter height

Filter width (/ downscale or upscale factor).

및

는 업스케일링 및 다운스케일링을 각각 나타낼 수 있다. 업스케일링 및 다운스케일링에 대해서, 트랜스포스된(transposed)된 컨볼루션이 사용될 수 있다.Also,

And

May indicate upscaling and downscaling, respectively. For upscaling and downscaling, transposed convolution can be used.

Convolutional neural networks can be used to implement transformation and reconstruction functions.

,

,

및

는 전술된 다른 실시예에서의 설명이 적용될 수 있다. 또한,

의 말단(end)에서는, 절대(absolute) 연산자(operator)가 아닌 자승(exponentiation) 연산자가 사용될 수 있다.Shown in FIG. 10

,

,

And

The description in the above-described other embodiments may be applied. Also,

At the end of, an exponentiation operator, not an absolute operator, can be used.

의 분포를 추정하기 위한 구성요소들이 컨볼루션 자동 부호기에 추가되었다.bracket

The components for estimating the distribution of are added to the convolutional automatic encoder.

"는 분포 추정자를 나타낼 수 있다.In FIG. 10, “Q” may indicate uniform quantization (rounding). "EC" may indicate entropy encoding. "ED" may indicate entropy decoding. "

"May represent a distribution estimator.

및

일 수 있다. 컨볼루션 레이어는 추정된

및 추정된

를 결과들로서 출력할 수 있다.Also, the automatic convolutional encoder can be implemented using convolutional layers. The input to the convolution layer is channel-wisely concatenated

And

Can be The convolution layer is estimated

And estimated

Can be output as results.

및

가 동일한 공간적 위치에 위치하는 모든

들에게 공유될 수 있다.Here, the same

And

Are all located in the same spatial location

Can be shared with others.

는

를 검출하기 위해 채널들을 걸쳐 모든 공간적으로 인접한 요소들을

로부터 추출할 수 있다. 유사하게,

는

를 위하여 모든 인접한 알려진 요소들을

로부터 추출할 수 있다. 이러한

및

에 의한 추출들은 서로 다른 채널들 사이의 남아있는(remaining) 상관관계들(correlations)을 캡춰하는 효과를 가질 수 있다.

The

All spatially adjacent elements across the channels to detect

Can be extracted from Similarly,

The

For all adjacent known elements

Can be extracted from Such

And

The extractions by can have the effect of capturing the remaining correlations between different channels.

는 동일한 공간적 위치에서의 1) 모든

, 2)

의 채널들의 총 개수 및 3)

들의 분포들을 단 하나의 단계에서 추출할 수 있으며, 이러한 추출을 통해 추정들의 총 개수가 감소될 수 있다.

1) All in the same spatial location

, 2)

3) the total number of channels of

The distributions of their can be extracted in just one step, and the total number of estimates can be reduced through this extraction.

의 파라미터들은

의 모든 공간적 위치들에 대하여 공유될 수 있다. 이러한 공유를 통해

당 단지 하나의 훈련된

가 영상들의 임의의 크기를 처리하기 위해 필요할 수 있다.Furthermore

The parameters of

Can be shared for all spatial locations. Through this sharing

Only one trained per

A may be needed to process any size of images.

However, in the case of training, despite the simplifications described above, collecting results from the entire spatial locations to calculate the rate term can be a heavy burden. To reduce this burden, a specified number (eg, 16) of random spatial points can be designated as representatives for every training step for the context adaptive entropy model. This designation can facilitate calculation of the rate term. Here, these random spatial points can only be used for rate terms. On the other hand, the distortion term can still be calculated on the whole image.

는 3-차원의 배열(array)이기 -문에,

에 대한 인덱스 i는 3 개의 인덱스들 k, l 및 m을 포함할 수 있다. k는 수평의 인덱스일 수 있다. l는 수직의 인덱스일 수 있다. m는 채널 인덱스일 수 있다.

Is a 3-dimensional array -because,

The index i for may include three indices k , l and m . k may be a horizontal index. l may be a vertical index. m may be a channel index.

는

을

로서 추출할 수 있다. 또한,

는

를

로서 추출할 수 있다. 여기에서,

는

의 알려진 영역을 나타낼 수 있다.When the current position is ( k , l , m ),

The

of

Can be extracted. Also,

The

To

Can be extracted. From here,

The

It can represent a known area of.

의 알려지지 않은 영역은 0으로 채워질 수 있다.

의 알려지지 않은 영역을 0으로 채움에 따라,

의 차원이

의 차원과 동일성을 갖도록 유지될 수 있다. 따라서,

는 언제나 0으로 채워질 수 있다.

The unknown region of can be filled with zeros.

By filling the unknown area of 0 with,

The dimension of

It can be maintained to have the same identity as the dimension of. therefore,

Can always be filled with zeros.

및

의 마진의(marginal) 영역들 또한 0으로 세트될 수 있다.To keep the dimensions of the estimation results as input,

And

The marginal areas of can also be set to zero.

는 단지 단순한 4

4

윈도우들 및 이진(binary) 마스크들을 사용하여 추출될 수 있다. 이러한 추출은 병렬 처리를 가능하게 할 수 있다. 반면, 복호화에서는, 순차적인(sequential) 재구축이 사용될 수 있다.When training or coding is performed,

Is just simple 4

4

It can be extracted using windows and binary masks. Such extraction may enable parallel processing. On the other hand, in decoding, sequential reconstruction may be used.

As another implementation technique to reduce the implementation cost, a hybrid approach can be used. The entropy model of an embodiment can be combined with a lightweight entropy model. For a lightweight entropy model, the expression components can be assumed to follow a zero-average Gaussian model with estimated standard deviations.

This hybrid approach can be utilized for the top four cases in descending order of bit-rate within nine configurations. In this application, it can be assumed that the number of sparse expression components having a very low spatial dependence for higher quality compression increases, so that direct scale estimation provides sufficient performance for these added expression components. have.

는 2 개의 파트들

및

로 분리될 수 있다. 2 개의 상이한 엔트로피 모델들이

및

에 대해서 적용될 수 있다.

,

,

및

의 파라미터들은 공유될 수 있고, 전체의 파라미터들은 여전히 함께 훈련될 수 있다.In the implementation, the secret expression component

Is 2 parts

And

Can be separated into. Two different entropy models

And

Can be applied against.

,

,

And

The parameters of can be shared, and the overall parameters can still be trained together.

의 개수는 182로 세트될 수 있다. 파라미터들

의 개수는 192로 세트될 수 있다. 약간 더 많은 파라미터들이 더 상위의 구성들에 대해서 사용될 수 있다.For example, parameters for 5 sub-configurations

The number of can be set to 182. Parameters

The number of can be set to 192. Slightly more parameters can be used for higher configurations.

For actual entropy encoding, an arithmetic encoder can be used. The arithmetic encoder can perform the generation and reconstruction of the bitstream as described above with the estimated model parameters.

As described above, based on the ANN-based image compression approach that utilizes the entropy model, the entropy models of the embodiment can be extended to utilize two different types of contexts.

These contexts allow the entropy model to more accurately estimate the distribution of expression components with a generalized form with mu parameters and standard deviations.

The contexts utilized can be divided into two types. One of the two types may be a type of free context, and may include portions of hidden variables known to both the encoder 800 and the decoder 900. The other of the two types may be a context requiring allocation of additional bits to be shared. The former can be contexts commonly used in various codecs. The latter may have been proven to aid compression. In an embodiment, a framework of entropy models utilizing these contexts has been provided.

Additionally, various methods of improving the performance of an embodiment can be considered.

One method for improving performance may be to generalize a distribution model on which an entropy model is based. In an embodiment, performance can be improved by generalizing previous entropy models and significantly acceptable results can be detected. However, Gaussian-based entropy models can obviously have limited expression power.

For example, when more elaborate models, such as non-parametric models, are combined with the context-adaptivity of the embodiment, this combination is actual distributions and estimation models. Reducing liver mismatch can provide better results.

Another way to improve performance may be to improve the levels of contexts.

The embodiment may use low level expression components within limited adjacent regions. Given sufficient capacity of the networks and a higher level of contexts, a more accurate estimate can be made possible by the embodiment.

For example, with respect to the structures of the human faces, if the entropy model understands that the above structures generally have two eyes, and that there is symmetry between the two eyes, the entropy model will look at the remaining one eye In coding, it is possible to approximate the distributions more accurately (with reference to the shape and position of one eye).

를 학습할 수 있다. 또한, 인--페인팅(in-painting) 방법들은 보이는 영역들이

로 주어졌을 때 조건적인(conditional) 분포

를 학습할 수 있다. 이러한 고-레벨 이해들(understandings)이 실시예에 결합될 수 있다.For example, a generative entropy model is a distribution of images within a specific domain, such as human faces and bedrooms.

Can learn. Also, in-painting methods are used for

Conditional distribution given by

Can learn. These high-level understandings can be incorporated into the embodiment.

Furthermore, contexts provided through additional information can be extended to high-level information such as a segmentation map and other information that aids compression. For example, the segmentation map can help to estimate the distribution of the expression components discriminatively according to the segment class to which the expression component belongs.

11 shows the structure of a hybrid network for higher bit-rate environments according to an example.

The notation methods used in FIG. 11 may be the same as those of the previous embodiment.

In FIG. 11, the first EC / ED may indicate entropy encoding and entropy decoding using the context-adaptive entropy model in the above-described embodiment. The second EC / ED following the zero-average Gaussian model may represent entropy encoding and entropy decoding using a lightweight entropy model.

Hybrid network 1100 may be an ANN that follows the hybrid approach described above.

는 2 개의 파트들 제1 부분 은닉 표현성분

및 제2 부분 은닉 표현성분

로 분할될 수 있고,

는 제1 양자화된 부분 은닉 표현성분

로 양자화될 수 있고,

는 제2 양자화된 부분 은닉 표현성분

로 양자화될 수 있다.In the hybrid network 1100, the hidden expression component

Is two parts, the first part hidden expression component

And the second part hidden expression component

Can be divided into

Is the first quantized partial hidden expression component

Can be quantized to

Is the second quantized partial hidden expression component

Can be quantized to

는 실시예의 문맥-적응적 엔트로피 모델을 사용하여 부호화될 수 있다. 반면, 분할의 결과들 중 다른 하나인

는 표준 추정을 사용하는 더 단순한 경량 엔트로피 모델을 사용하여 부호화될 수 있다.One of the results of the split

Can be coded using the context-adaptive entropy model of the embodiment. On the other hand, one of the results of division

Can be coded using a simpler lightweight entropy model using standard estimation.

All concatenation and split operators can be performed in a channel-wise manner.

A lightweight entropy model can be combined with the context-adaptive entropy model to reduce implementation costs for high-bit-per-pixel (Bits Per Pixel) configurations.

The lightweight entropy model can utilize a scale (say, standard deviation) estimate, assuming that the PMF approximations of the quantized expression components follow a zero-means Gaussian distribution with a standard uniform distribution.

는 채널-단위로 2 개의 파트들

및

로 분할될 수 있다.

는

채널들을 가질 수 있다.

는

채널들을 가질 수 있다. 다음으로,

및

는 양자화될 수 있다.

가 양자화됨에 따라

가 생성될 수 있다.

가 양자화됨에 따라

가 생성될 수 있다.Expression

Is two parts in channel-unit

And

Can be divided into

The

You can have channels.

The

You can have channels. to the next,

And

Can be quantized.

As is quantized

Can be generated.

As is quantized

Can be generated.

는 문맥-적응적 엔트로피 모델로 엔트로피 인코딩될 수 있다. 반면,

는 경량 엔트로피 모델로 엔트로피 인코딩될 수 있다.

Can be entropy encoded with a context-adaptive entropy model. On the other hand,

Can be entropy encoded with a lightweight entropy model.

의 표준 편차들은

및

를 통해 추정될 수 있다.

The standard deviations of

And

Can be estimated through

의 결과들을 추정자

로의 입력 소스로서 사용할 수 있다. 이러한 문맥-적응적 엔트로피 모델과는 달리, 경량 엔트로피 모델은

로부터 직접적으로 추정된 표준 편차들을 검출할 수 있다. 여기에서,

는

및

의 연쇄를 입력으로서 취할 수 있다.

는

와 함께

를 동시에 생성할 수 있다.The context-adaptive entropy model

The results of the estimator

Can be used as an input source for furnaces. Unlike this context-adaptive entropy model, a lightweight entropy model

The standard deviations estimated can be directly detected from. From here,

The

And

You can take the chain of as input.

The

with

Can be created simultaneously.

Equation 6 below may represent a total loss function.

,

및

의 3 개로 나뉠 수 있다. 말하자면, 율 항은

,

및

를 포함할 수 있다. 왜곡 항은 전술된 실시예에서의 왜곡 항과 동일할 수 있다. 단,

는

및

의 채널-단위로 연쇄된 표현성분일 수 있다.The total loss function can also include rate terms and distortion terms. Rate terms

,

And

Can be divided into three. In other words, the rate term

,

And

It may include. The distortion term may be the same as the distortion term in the above-described embodiment. only,

The

And

It may be an expression component chained in a channel-unit of.

,

및

의 노이즈 낀 표현성분은 표준 균일 분포를 따를 수 있다.

의 평균값은

일 수 있다.

의 평균값은

일 수 있다.

의 평균값은

일 수 있다.

,

And

The noisy expression component of can follow the standard uniform distribution.

The average value of

Can be

The average value of

Can be

The average value of

Can be

및

는

로부터의 채널-단위로 분할된 표현성분들일 수 있다.

는 변환

의 결과일 수 있다.

And

The

It may be expression components partitioned by channel-unit from.

Convert

May be the result of

는

채널들을 가질 수 있다.

는

채널들을 가질 수 있다.

The

You can have channels.

The

You can have channels.

에 대한 율 항은 전술된 수학식 4에서의 모델과 동일한 모델일 수 있다.

The rate term for may be the same model as the model in Equation 4 described above.

는

에 대한 모델에는 기여하지 않을 수 있고,

에 대한 모델에는 기여할 수 있다.

The

May not contribute to the model for,

Can contribute to the model.

에 대한 율 항에서, 노이즈 낀 표현성분들은 단지 학습을 위해서만 엔트로피 모델들의 입력으로 사용될 수 있다. 말하자면,

에 대한 율 항에서, 노이즈 낀 표현성분들은 엔트로피 모델들의 조건들에 대해서는 사용되지 않을 수 있다.

In the rate term for, the noisy expression components can be used as input to entropy models only for learning. as it were,

In the rate term for, noisy expression components may not be used for the conditions of entropy models.

의 엔트로피 모델은 전술된 수학식 5에서의 엔트로피 모델과 동일할 수 있다.

The entropy model of may be the same as the entropy model in Equation 5 described above.

In an implementation, a hybrid structure can be used for the top four configurations in descending order of bit-rate.

은 400으로 세트될 수 있고,

는 192로 세트될 수 있고,

는 408로 세트될 수 있다.For example, for the top two configurations,

Can be set to 400,

Can be set to 192,

Can be set to 408.

은 320으로 세트될 수 있고,

는 192로 세트될 수 있고,

는 228로 세트될 수 있다.For example, for the following two configurations,

Can be set to 320,

Can be set to 192,

Can be set to 228.

12 is a structural diagram of an encoding device according to an embodiment.

The encoding apparatus 1200 includes a processor 1210, a memory 1230, a user interface (UI) input device 1250, a UI output device 1260, and a storage that communicates with each other through the bus 1290. It may include (1240). Also, the encoding device 1200 may further include a communication unit 1220 connected to the network 1299.

The processing unit 1210 may be a central processing unit (CPU), a semiconductor device that executes processing instructions stored in the memory 1230 or the storage 1240. The processor 1210 may be at least one hardware processor.

The processing unit 1210 may generate and process signals, data, or information input to the device 1200, output from the device 1200, or used inside the device 1200, and may generate signals, data, or information. Related inspections, comparisons and judgments. That is, in the embodiment, generation and processing of data or information, and inspection, comparison, and judgment related to data or information may be performed by the processing unit 1210.

At least some of the elements constituting the processing unit 1210 may be program modules, and may communicate with an external device or system. The program modules may be included in the encoding device 1200 in the form of an operating system, application program modules, and other program modules.

The program modules may be physically stored on various known storage devices. Also, at least some of the program modules may be stored in a remote storage device capable of communicating with the encoding device 1200.

Program modules perform a function or operation according to an embodiment, or a routine, subroutine, program, object, component and data implementing an abstract data type according to an embodiment It may include a data structure, but is not limited thereto.

The program modules may be composed of instructions or codes performed by at least one processor of the encoding device 1200.

The processor 1210 may correspond to the encoder 800 described above.

The storage unit may represent the memory 1230 and / or the storage 1240. The memory 1230 and the storage 1240 may be various types of volatile or nonvolatile storage media. For example, the memory 1230 may include at least one of a ROM (ROM) 1231 and a RAM (1232).

The storage unit may store data or information used for the operation of the encoding device 1200. In an embodiment, data or information possessed by the encoding apparatus 1200 may be stored in the storage unit.

The encoding apparatus 1200 may be implemented in a computer system including a recording medium that can be read by a computer.

The recording medium may store at least one module required for the encoding apparatus 1200 to operate. The memory 1230 may store at least one module, and at least one module may be configured to be executed by the processing unit 1210.

A function related to communication of data or information of the encoding device 1200 may be performed through the communication unit 1220.

The network 1299 may provide communication between the encoding device 1200 and the decoding device 1300.

13 is a structural diagram of a decoding apparatus according to an embodiment.

The decoding apparatus 1300 includes a processing unit 1310, a memory 1330, a user interface (UI) input device 1350, a UI output device 1360, and a storage unit that communicates with each other through the bus 1390. It may include (1340). In addition, the decoding device 1300 may further include a communication unit 1320 connected to the network 1399.

The processing unit 1310 may be a central processing unit (CPU), a semiconductor device that executes processing instructions stored in the memory 1330 or the storage 1340. The processor 1310 may be at least one hardware processor.

The processor 1310 may be input to the device 1300, output from the device 1300, or may generate and process signals, data, or information used inside the device 1300, and may perform signal, data, or information Related inspections, comparisons and judgments. That is, in the embodiment, the generation and processing of data or information, and the inspection, comparison, and judgment related to the data or information may be performed by the processing unit 1310.

At least some of the elements constituting the processing unit 1310 may be program modules, and may communicate with an external device or system. The program modules may be included in the decoding device 1300 in the form of an operating system, application program modules, and other program modules.

The program modules may be physically stored on various known storage devices. Also, at least some of the program modules may be stored in a remote storage device capable of communicating with the decoding device 1300.

Program modules perform a function or operation according to an embodiment, or a routine, subroutine, program, object, component and data implementing an abstract data type according to an embodiment It may include a data structure, but is not limited thereto.

The program modules may be composed of instructions or codes performed by at least one processor of the decoding device 1300.

The processor 1310 may correspond to the decoder 900 described above.

The storage unit may represent the memory 1330 and / or the storage 1340. The memory 1330 and the storage 1340 may be various types of volatile or nonvolatile storage media. For example, the memory 1330 may include at least one of a ROM (1331) and a RAM (RAM) 1332.

The storage unit may store data or information used for the operation of the decoding device 1300. In an embodiment, data or information possessed by the decoding apparatus 1300 may be stored in the storage unit.

The decoding apparatus 1300 may be implemented in a computer system including a recording medium that can be read by a computer.

The recording medium may store at least one module required for the decoding apparatus 1300 to operate. The memory 1330 may store at least one module, and at least one module may be configured to be executed by the processing unit 1310.

A function related to communication of data or information of the decoding device 1300 may be performed through the communication unit 1320.

The network 1399 may provide communication between the encoding device 1200 and the decoding device 1300.

14 is a flowchart of an encoding method according to an embodiment.

In step 1410, the processor 1210 of the encoding apparatus 1200 may generate a bitstream.

The processor 1210 may generate an bitstream by performing entropy encoding using an entropy model on the input image.

The entropy model can be the context-adaptive entropy model described above. The context-adaptive entropy model can utilize multiple different types of contexts. A plurality of different types of contexts may include a bit-consumption context and a bit-free context.

The standard deviation parameter and the mean value parameter of the entropy model can be estimated from a plurality of different types of contexts. In other words, the entropy model can be based on a Gaussian model with mean value parameters.

Alternatively, the entropy model may be a plurality of types of entropy models. For example, the entropy model can include a context-adaptive entropy model and a lightweight entropy model.

In operation 1420, the communication unit 1220 of the encoding device 1200 may transmit a bitstream. The communication unit 1220 may transmit the bitstream to the decoding device 1300. Alternatively, the bitstream may be stored in the storage unit of the encoding device 1200.

The contents related to the entropy encoding and the entropy engine of the image described in the above-described embodiment can also be applied to this embodiment. Duplicate description is omitted.

15 is a flowchart of a decoding method according to an embodiment.

In operation 1510, the communication unit 1320 of the decoding apparatus 1300 may acquire a bitstream.

In operation 1520, the processing unit 1310 of the decoding apparatus 1300 may generate a reconstructed image using a bitstream.

The processor 1310 of the decoding apparatus 1300 may perform decoding using an entropy model on the bitstream to generate a reconstructed image.

The entropy model can be the context-adaptive entropy model described above. The context-adaptive entropy model can utilize multiple different types of contexts. A plurality of different types of contexts may include a bit-consumption context and a bit-free context.

The standard deviation parameter and the mean value parameter of the entropy model can be estimated from a plurality of different types of contexts. In other words, the entropy model can be based on a Gaussian model with mean value parameters.

Alternatively, the entropy model may be a plurality of types of entropy models. For example, the entropy model can include a context-adaptive entropy model and a lightweight entropy model.

The contents related to the entropy decoding and the entropy engine of the image described in the above-described embodiment can also be applied to this embodiment. Duplicate description is omitted.

In the above-described embodiments, the methods are described based on a flow chart as a series of steps or units, but the present invention is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. You can. In addition, those skilled in the art may recognize that the steps shown in the flowchart are not exclusive, other steps are included, or one or more steps in the flowchart may be deleted without affecting the scope of the present invention. You will understand.

The above-described embodiments include examples of various aspects. Although not all possible combinations for indicating various aspects can be described, those skilled in the art will recognize that other combinations are possible in addition to those explicitly described. Accordingly, the present invention will be said to include all other replacements, modifications and changes that fall within the scope of the following claims.

The embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and can be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and available to those skilled in the computer software field.

The computer-readable recording medium may include information used in embodiments according to the present invention. For example, a computer-readable recording medium may include a bitstream, and the bitstream may include information described in embodiments according to the present invention.

The computer readable recording medium may include a non-transitory computer-readable medium.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

In the above, the present invention has been described by specific matters such as specific components, etc. and limited embodiments and drawings, which are provided to help the overall understanding of the present invention, but the present invention is not limited to the above embodiments , Those skilled in the art to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention is not limited to the above-described embodiment, and should not be determined, and all claims that are equally or equivalently modified with the claims as described below are within the scope of the spirit of the present invention. Would say

Claims (20)

Generating a bitstream by performing entropy encoding on the input image using an entropy model; And
Transmitting or storing the bitstream
Encoding method comprising a. According to claim 1,
The entropy model is a context-adaptive entropy model,
The context-adaptive entropy model utilizes a plurality of different types of contexts. According to claim 2,
A plurality of types of different types of contexts include a bit-consumption context and a bit-free context. According to claim 3,
A coding method in which standard deviation parameters and average value parameters of the entropy model are estimated from a plurality of different types of the contexts. According to claim 2,
The input method of the context-adaptive entropy model to the analysis transform is a uniformly quantized expression component. According to claim 1,
The entropy model is a coding method based on a Gaussian model having an average value parameter. According to claim 1,
The entropy model includes a context-adaptive entropy model and a lightweight entropy model. The method of claim 7,
The hidden expression component is divided into a first partial hidden expression component and a second partial hidden expression component,
The first partial hidden expression component is quantized as a first quantized partial hidden expression component,
The second partial hidden expression component is quantized as a second quantized partial hidden expression component,
The first quantized partial hidden expression component is encoded using the context-adaptive entropy model,
The second quantized partial hidden expression component is encoded using the lightweight entropy model. The method of claim 7,
The lightweight entropy model is a coding method using scale estimation. The method of claim 7,
The lightweight entropy model is a coding method for detecting standard deviations estimated directly from an analytical transform. A communication unit that acquires a bitstream; And
A processor for decoding the bitstream using an entropy model and generating a reconstructed image
Decoding device comprising a. Obtaining a bitstream; And
Generating a reconstructed image by performing decoding using an entropy model on the bitstream
Decoding method comprising a. The method of claim 12,
The entropy model is a context-adaptive entropy model,
The context-adaptive entropy model utilizes a plurality of different types of contexts. The method of claim 13,
A plurality of different types of contexts include a bit-consumption context and a bit-free context. The method of claim 14,
A decoding method in which a standard deviation parameter and an average value parameter of the entropy model are estimated from a plurality of different types of the contexts. The method of claim 13,
The input method of the context-adaptive entropy model to the analysis transform is a uniformly quantized expression component. The method of claim 12,
The entropy model is a decoding method based on a Gaussian model having an average value parameter. The method of claim 12,
The entropy model includes a context-adaptive entropy model and a lightweight entropy model. The method of claim 18,
The lightweight entropy model is a decoding method using scale estimation. The method of claim 18,
The lightweight entropy model is a decoding method for detecting standard deviations estimated directly from an analytical transformation.

Images