KR20220048875A - Method for generating quantization table - Google Patents

Abstract

According to the present invention, provided is a method for generating a quantization table for each target task, which includes the steps of: setting a target task; determining a distortion loss function corresponding to the target task; estimating a bitrate loss function expressed in a differentiable form; calculating a quantization variable by adding predetermined weights to the distortion loss function and the bitrate loss function, respectively, and adding the same to each other; and deriving an optimal value of the quantization variable and generating a quantization table based on the optimal value.

Description

Method for generating quantization table for each target task

The present invention relates to a method of generating a quantization table for each target task, and more particularly, to a method of generating an optimal quantization table according to each image and purpose in order to improve the performance of a JPEG compression algorithm.

The present invention relates to a method of generating a quantization table when compressing a JPEG image, and more particularly, to a method of generating a quantization table optimized for each task.

With the development of communication technology such as the Internet, technology for transmitting and processing images is also developing at a rapid pace. This technical background increases the need for an image compression method that can obtain better image quality even at a low speed.

There are two main types of compression: lossless and lossy. Lossless compression is compression that allows images to be restored almost to the original. Image loss compression seeks to increase compression performance through RD (rate-distortion, bitrate-quality degradation trade-off) optimization to minimize image quality degradation due to loss.

A representative technique among these compression methods is JPEG compression, and the JPEG method performs quantization and inverse quantization using a quantization table. The quantization table is a decisive factor in JPEG compression performance. Depending on which quantization table is used for compression, the quality of the image can be greatly affected. The parameters constituting it can be defined by the user at any time, but it is usually fixed as a standard. use the specified value.

However, the standardized quantization table has a disadvantage in that it is difficult to achieve optimal performance depending on the purpose, and accordingly, a method for generating a quantization table optimized for each task is required.

KR 0620606

The problem to be solved by the present invention is to optimize and generate a quantization table required for JPEG compression according to the purpose.

Another object to be solved by the present invention is to provide a method for estimating a JPEG bitrate based on a neural network in order to generate the optimized quantization table.

A method for generating a quantization table for each target task of the present invention for solving the above problem includes: setting a target task; determining a distortion loss function corresponding to the target task; estimating a bitrate loss function expressed in a differentiable form; calculating a quantization variable by adding predetermined weights to each of the distortion loss function and the bitrate loss function; and calculating an optimal value of the quantization variable and generating a quantization table based on the optimal value.

In one embodiment of the present invention, the method further comprises the step of learning a regression model for estimating the bitrate loss function, wherein the regression model can be learned through a neural network model.

In an embodiment of the present invention, the neural network model may be a Long Short-Term Memory (LSTM).

In an embodiment of the present invention, the regression model may be trained to estimate a Huffman model.

In an embodiment of the present invention, the regression model may be trained through a multiple perceptron (MLP).

In one embodiment of the present invention, the bitrate loss function is composed of the sum of a first part based on a DC component and a second part based on an AC component, wherein the first part and the second part are each multiple It can be derived by a regression model that is learned through a perceptron (MLP).

In an embodiment of the present invention, the second part may use a result derived by a regression model learned through a long short-term memory (LSTM) as an input variable.

In an embodiment of the present invention, the regression model may use training data that is used by synthesizing random data.

In an embodiment of the present invention, the optimal value may be to minimize the quantization variable.

In an embodiment of the present invention, the target task may be any one of preset ones.

In an embodiment of the present invention, the preset tasks may include image classification, image segmentation, image captioning, and image quality.

In an embodiment of the present invention, the distortion loss function corresponding to the preset tasks may include a classification error, MSE, and perceptual loss.

In an embodiment of the present invention, the method may further include determining a bit rate based on the generated quantization table.

The method for generating a quantization table for each target task according to an embodiment of the present invention generates a quantization table optimized for each task, thereby maximally maintaining required performance even in a compressed image.

In addition, since only the quantization table is optimized and provided and the JPEG baseline is used as it is, it can be used without changing the existing codec, which has a high usability effect.

1 is a flowchart of a method of generating a quantization table for each target task according to an embodiment of the present invention.
2 is a block diagram of bitrate loss function estimation according to an embodiment of the present invention.
3 is a block diagram of a learning framework according to an embodiment of the present invention.
4 is a block diagram of a JPEG compression method according to an embodiment of the present invention.
5A is a block diagram for generating a quantization table for each task according to an embodiment of the present invention.
5B is a configuration diagram for generating a quantization table for each image according to an embodiment of the present invention.
6 is a flowchart of a method of generating a quantization table for each target task according to another embodiment of the present invention.
7 is a graph of a compression result according to an embodiment of the present invention.
8 is an image comparing compression results according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the present invention, if it is determined that adding a detailed description of a technique or configuration already known in the relevant field may make the gist of the present invention unclear, some of it will be omitted from the detailed description. In addition, the terms used in this specification are terms used to properly express the embodiments of the present invention, which may vary according to a person or custom in the relevant field. Accordingly, definitions of these terms should be made based on the content throughout this specification.

The terminology used herein is for the purpose of referring to specific embodiments only, and is not intended to limit the invention. As used herein, the singular forms also include the plural forms unless the phrases clearly indicate the opposite. As used herein, the meaning of 'comprising' specifies a particular characteristic, region, integer, step, operation, element and/or component, and other specific characteristic, region, integer, step, operation, element, component, and/or group. It does not exclude the existence or addition of

In the embodiments of the present specification, terms such as first, second, etc. are used only for the purpose of distinguishing one component from other components, and unless otherwise specified, do not limit the order or importance between the components. does not Accordingly, within the scope of the embodiments herein, a first component in an embodiment may be referred to as a second component in another embodiment, and similarly, a second component in an embodiment is referred to as a first component in another embodiment. can also be called

In the present specification, the communication method of the network is not limited, and the connection between each component may not be connected in the same network method. The network may include not only a communication method using a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network, a satellite network, etc.) but also short-range wireless communication between devices. For example, the network may include all communication methods through which an object and an object can network, and is not limited to wired communication, wireless communication, 3G, 4G, 5G, or other methods. For example, a wired and/or network may be a Local Area Network (LAN), a Metropolitan Area Network (MAN), a Global System for Mobile Network (GSM), an Enhanced Data GSM Environment (EDGE), a High Speed Downlink Packet Access (HSDPA), W-CDMA (Wideband Code Division Multiple Access), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), Bluetooth, Zigbee, Wi-Fi, VoIP (Voice over) Internet Protocol), LTE Advanced, IEEE802.16m, WirelessMAN-Advanced, HSPA+, 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and MBWA (IEEE 802.20) systems, HIPERMAN, Beam-Division Multiple Access (BDMA), Wi-MAX (World Interoperability for Microwave Access), and ultrasonic-based communication can refer to a communication network by one or more communication methods selected from the group consisting of However, the present invention is not limited thereto.

Hereinafter, a method of generating a quantization table for each target task according to an embodiment of the present invention will be described with reference to FIGS. 1 to 5 .

The present invention relates to a method of generating an optimal quantization table for each target task in the process of compressing an image using the JPEG method.

1 is a flowchart of a method of generating a quantization table for each target task according to an embodiment of the present invention.

Referring to FIG. 1 , the method for generating a quantization table for each target task of the present invention includes the steps of setting a target task (S110), determining a distortion loss function corresponding to the target task (S120), and a bit expressed in a differentiable form. Estimating the rate loss function (S130), weighting the distortion loss function and the bitrate loss function (S140), calculating a quantization variable (S150), and deriving an optimal value of the quantization variable (S160) ) and generating a quantization table based on the optimal value (S170).

Each of the above-described steps may be performed irrespective of the listed order, except when performed in the listed order due to a special causal relationship. However, hereinafter, for convenience of description, it is assumed that each of the above-described steps is performed according to the listed order.

Hereinafter, the step of setting the target task ( S110 ) will be described. The step of setting the target task ( S110 ) is a step of setting one target task according to the user's selection among at least one preset task.

The task is about what the user wants to achieve through image compression. In general, in image compression, a task may be to maximize the visual quality of the image compared to the compression ratio. However, other various tasks may exist. For example, the tasks may include, but are not limited to, image classification, image segmentation, and image captioning, and may include any purpose that a user seeks to achieve through image compression.

The present invention relates to providing an optimal compression method according to the task. For example, if the task is to classify a human face during image classification, the compression ratio of the human face part in the compressed image is reduced so that the human face part matches the original image as much as possible, and the compression ratio of the rest part is increased. there is. Through this, the size of the image file can be compressed to the maximum possible while being optimized for the task. Therefore, if the task of classifying human faces is achieved through the cloud, the amount of data transmitted through the network in the process can be minimized.

The target task should be set according to a user's selection among at least one preset task. After the target task is determined in step S110, a distortion loss function corresponding to the target task must be determined in step S120, since the distortion loss function is provided in a limited number according to the required purpose of the task.

Hereinafter, the step of determining the distortion loss function corresponding to the target task (S120) will be described.

Distortion loss function is a loss function related to image quality. It occurs in image loss compression method and is an error for the task.

Various metrics may be applied to the distortion loss function according to the target task set in step S110. For example, when the set target task is image classification, a classification error may be applied to the distortion loss function.

As for the distortion loss function, MSE and perceptual loss may be provided in addition to the above-described classification error, and an additional distortion loss function corresponding to a preset task may be provided without being limited thereto.

Hereinafter, the step of estimating the bitrate loss function expressed in a differentiable form ( S130 ) will be described with reference to FIG. 2 together with FIG. 1 .

The bitrate loss function is a loss function related to the compression capacity, and in the present invention, it is estimated to determine the bitrate by generating a quantization table. The bitrate loss function must be expressed in a differentiable form for neural network training.

2 is a block diagram of bitrate loss function estimation according to an embodiment of the present invention.

Referring to FIG. 2 , the bitrate loss function may be estimated through a neural network. Specifically, the regression model 220 for predicting the bit rate may be learned through the neural network 210 and the bit rate loss function 230 may be estimated through this.

More specifically, in order to predict the bitrate of a JPEG image, a neural network that simulates each step such as run-length encoding and Huffman coding in the entropy coding process of JPEG may be constructed. The training data can be used by synthesizing random data so that any image can be predicted, and can be applied to each 8x8 pixel block, which is the basic unit of JPEG encoding.

The neural network model may be a Long Short-Term Memory (LSTM). However, the present invention is not limited thereto, and other algorithms corresponding to a recurrent neural network (RNN) may be included in the scope of the present invention.

A regression model can be trained to estimate a Huffman model. Specifically, the regression model may be trained to estimate the Huffman code length according to the Huffman model. The bitrate loss function consists of the sum of a first part based on the DC component and a second part based on the AC component. The first part and the second part may be derived by a regression model learned through a multi-layer perceptron (MLP), respectively.

Here, the first part may be expressed as a sum of the DC component of the Huffman model based on an 8x8 pixel block, which is a basic unit of JPEG encoding.

In addition, the second part based on the AC component may use a result derived by a regression model learned through a long short-term memory (LSTM) as an input variable.

Expressing this as an equation, it is expressed as follows.

는 각각 픽셀 블록의 JPEG의 DC 성분, AC 성분 벡터를 의미하며, Hdc, Hac는 허프만 모델에 따른 허프만 코드 길이를 나타낸다. 상기 회귀모델은 허프만 모델을 추정하도록 학습될 수 있으며 다중 퍼셉트론(MLP)을 통하여 학습될 수 있다. 다만, 이에 한정되지 않고 다양한 모델을 통하여 학습될 수 있다. Sac는 Run-length 인코딩의 결과를 예측하는 모델로 상술한 LSTM을 이용할 수 있다.

denotes the DC component and AC component vectors of the JPEG of the pixel block, respectively, and Hdc and Hac denote the Huffman code length according to the Huffman model. The regression model can be trained to estimate the Huffman model and can be trained through multiple perceptrons (MLP). However, the present invention is not limited thereto and may be learned through various models. Sac may use the above-described LSTM as a model for predicting the result of run-length encoding.

Hereinafter, the step of giving weight to the distortion loss function and the bitrate loss function (S140) and the step of calculating the quantization variable (S150) will be described.

After weighting the distortion loss function determined in step S120 and the bitrate loss function estimated in step S130, a quantization variable can be calculated by adding them together.

Expressing this as an equation, it is expressed as follows.

은 각각 원본 이미지, 양자화 테이블, 압축된 인코딩 벡터를 의미하며, Enc, Dec은 JPEG 인코더, 디코더이다.Here, D denotes a distortion loss function, and R denotes a bitrate loss function. Also, L _Q represents a quantization variable.

is the original image, quantization table, and compressed encoding vector, respectively, and Enc and Dec are JPEG encoders and decoders.

Hereinafter, the step of deriving the optimal value of the quantization variable ( S160 ) will be described.

As described above, the quantization variable is learned in the direction of deriving an optimal value by considering both the distortion loss function and the bitrate loss function. Such an optimal value may be derived in a direction in which the quantization variable is minimized, but is not limited thereto, and may be derived by various methods.

Hereinafter, the step ( S170 ) of generating a quantization table based on the optimal value of the quantization variable will be described.

In the JPEG compression method, quantization and inverse quantization are performed using a quantization table. The quantization table is a decisive factor in JPEG compression performance. Depending on which quantization table is used for compression, the quality of the image can be greatly affected. The parameters constituting it can be defined by the user at any time, but it is usually fixed as a standard. use the specified value.

Finally, the bit rate is determined based on the quantization table generated according to the present invention.

In this case, only the quantization table itself can be optimized and generated, and the JPEG standard value can be used in the rest of the process. Therefore, when a change in the JPEG codec itself is required, it may not operate if there is no corresponding decoder. However, in case of optimizing only the value of the quantization table, the existing codec can be used without changing, so there is an advantage of high applicability.

3 is a block diagram of a learning framework according to an embodiment of the present invention.

Referring to FIG. 3 , the JPEG compression process is performed in the order of a color space conversion step, a discrete cosine transform (DCT) step, and a quantization step.

Hereinafter, the color space conversion step will be described.

In general, JPEG image compression converts the RGB color space to the YCbCr color space (RGB-to-YCbCr). The reason for converting the color space is to compress the color information more than the brightness information because the human eye is more sensitive to the brightness (luminance) component than the color component. For this purpose, downsampling is performed, and sampling is performed in the direction of reducing Cb and Cr while maintaining the Y component. Downsampling is usually expressed as a ratio of J:a:b. Here, J means the width of the pixel block to be sampled. And a means the number of samples extracted from the first row of J pixels. b denotes the number of samples extracted from the second row of J pixels. In JPEG, the 4:2:0 method is mainly used. If the Cb component and the Cr component are downsampled in the 4:2:0 method, 2 values are extracted per 8 pixels and 6 values are discarded, so the capacity can be reduced by that much.

Hereinafter, the DCT step will be described.

After downsampling, each channel is divided into 8x8 blocks. 2D Discrete Cosine Transform (DCT) is applied to each 8x8 block. As a result, the image can be viewed in terms of frequency, enabling compression. In general, JPEG compresses images by removing high-frequency components.

Hereinafter, the quantization step will be described.

Specifically, quantization is a process for removing frequency components. After dividing each coefficient in the frequency domain by a quantization table, rounded values are taken. Accordingly, DCT coefficients are quantized to remove coefficients related to a high frequency signal.

In this process, the quantization table can be a major factor in JPEG compression performance. In the present invention, by deriving and applying a non-standard quantization table for each image and each task, JPEG compression optimized for the task can be achieved.

In addition, in the decoding process, decoding is possible without changing the existing codec by using the generated quantization table as it is for decoding.

4 is a block diagram of a JPEG compression method according to an embodiment of the present invention.

Referring to FIG. 4 , specifically, the method of applying the quantization table generated in the present invention is performed in the order of a quantization table generation step and a subsequent step.

Hereinafter, the step of generating the quantization table will be described.

Just before DCT, which is the start of the JPEG encoder, the image is put into the learned quantization network, the derived quantization table is parameterized, and this can be utilized in the quantization process (Quantizer).

Hereinafter, steps after generation of the quantization table will be described.

After the quantization table is created, all processes are performed in the same way as the existing standard JPEG encoding process. In the process of decoding, since quantization table information is already encoded in the JPEG bitstream, it can be automatically decoded based on this. Therefore, as described above, since only the quantization table is optimized and provided and the JPEG baseline is used as it is, it can be used without changing the existing codec, which has a high usefulness effect.

5A is a block diagram for generating a quantization table for each task according to an embodiment of the present invention.

Referring to FIG. 5A , when one image has various tasks, different quantization tables may be generated for each task. For example, when securing image quality in one image is a task, a quantization table can be generated through general R-D optimization, but other tasks such as image classification, image segmentation, and image captioning , it is difficult to see it as an optimized table.

Therefore, a quantization table optimized for a task can be created through learning and used in the compression process. That is, in the process of JPEG compression of one image, different quantization tables are derived for each preset task, so that compression is possible in an optimized manner for each task.

5B is a configuration diagram for generating a quantization table for each image according to an embodiment of the present invention.

Referring to FIG. 5B , when several images have one task, different quantization tables may be generated for each image. For example, when a plurality of images have one task of image classification, different quantization tables can be created through learning for each image and used in the compression process.

Combining FIGS. 5A and 5B , a quantization table optimized according to each image and each task can be generated, and this can be utilized in the compression process. That is, it is possible to generate a quantization table optimized for a specific task with respect to a specific image.

Hereinafter, a method of generating a quantization table for each target task according to another embodiment of the present invention will be described with reference to FIG. 6 attached thereto.

6 is a flowchart of a method of generating a quantization table for each target task according to another embodiment of the present invention.

Referring to FIG. 6 , the method for generating a quantization table for each target task according to the present embodiment includes all the steps of the embodiment described above with reference to FIGS. 1 to 5, but generates a quantization table based on the optimal value of the quantization variable ( After S170), the step of determining a bit rate based on the quantization table (S610) is further included. Hereinafter, the added step (S610) will be described.

Specifically, since the JPEG baseline is used as it is in the JPEG compression process according to the present invention, a desired image quality can be obtained by determining the bit rate. That is, in the learning process, it is possible to estimate whether a desired bit rate is obtained by a specific quantization table, and JPEG image compression can be performed with the bit rate determined by the estimation.

7 is a graph of a compression result according to an embodiment of the present invention.

Referring to FIG. 7 , the R-D curve result according to the compression method is shown. As image evaluation methods, image classification using Inception-v3, image caption evaluation (BLEU score) generated by an RNN-based model, and PSNR and MS-SSIM, which are image quality evaluation scales, were used.

Specifically, in the case of evaluation (710a, 710b) using Inception-v3 when the task is image classification, comparing the results of the compressed image and the compressed image according to the JPEG baseline after generating a quantization table according to the present invention It can be seen that better quality can be obtained in all bitrate areas. In particular, it can be seen that the higher the compression rate, the greater the difference in performance.

Similarly, even in the case of evaluation using BLEU (720a, 720b, 720c, 720d) when the task is image captioning, the result of the compressed image and the compressed image according to the JPEG baseline after generating the quantization table according to the present invention By comparison, it can be seen that better quality can be obtained in all bitrate regions. In particular, it can be seen that the higher the compression rate, the greater the difference in performance.

Finally, when the task is image quality, the same result can be confirmed in the case of the evaluation 730a and the MS-SSIM 730b using PSNR. In the case of evaluation using PSNR (730a), the distortion loss function shows almost the same results when using MSE and MS-SSIM. Better quality can be obtained, and it can be seen that the higher the compression rate, the greater the difference in performance. In the case of evaluation using MS-SSIM (730b), the distortion loss function when MS-SSIM was used showed better results than when MSE was used. Better quality can be obtained in all bitrate areas, and it can be seen that the higher the compression rate, the greater the difference in performance.

8 is an image comparing compression results according to an embodiment of the present invention.

Referring to FIG. 8 , the results according to the compression method are shown. Specifically, an original image 810a, an image 820a compressed after generating a quantization table according to the present invention, and an image 830a compressed according to a JPEG baseline are shown. As for whether the texture or color of the main part for recognition is better preserved, the result value expressed in color is the original image result 810b, the image result 820b compressed after creating a quantization table according to the present invention, and JPEG An image result 730b compressed according to the baseline is shown. From this result, the bit-ray of the compressed image 820a after generating the quantization table according to the present invention? The image 830a compressed according to the 0.298bpp and JPEG baseline is 0.301bpp, so the result according to the present invention increases the compression rate for relatively less important parts, so that higher recognition performance can be maintained despite compression with a lower bitrate. indicates the result.

The technical features disclosed in each embodiment of the present invention are not limited only to the corresponding embodiment, and unless they are mutually incompatible, the technical features disclosed in each embodiment may be combined and applied to different embodiments.

In the above, embodiments of the method for generating a quantization table for each target task of the present invention have been described. The present invention is not limited to the above-described embodiments and the accompanying drawings, and various modifications and variations will be possible from the point of view of those skilled in the art to which the present invention pertains. Accordingly, the scope of the present invention should be defined not only by the claims of the present specification, but also by those claims and their equivalents.

The present invention described above can be implemented as computer-readable code (or application or software) on a medium in which a program is recorded. The above-described generating method may be realized by a code stored in a memory or the like.

The computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is also a carrier wave (eg, transmission over the Internet) that is implemented in the form of. In addition, the computer may include a processor or a control unit. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (13)

setting a target task;
determining a distortion loss function corresponding to the target task;
estimating a bitrate loss function expressed in a differentiable form;
calculating a quantization variable by adding predetermined weights to the distortion loss function and the bitrate loss function, respectively; and
deriving an optimal value of the quantization variable and generating a quantization table based on the optimal value;
A method of generating a quantization table for each target task comprising:
According to claim 1,
Further comprising the step of learning a regression model for estimating the bitrate loss function,
The regression model is a method of generating a quantization table for each target task that is learned through a neural network model.
3. The method of claim 2,
The method for generating a quantization table for each target task, wherein the neural network model is a Long Short-Term Memory (LSTM).
3. The method of claim 2,
The regression model is a method of generating a quantization table for each target task that is trained to estimate a Huffman model.
5. The method of claim 4,
The regression model is a method of generating a quantization table for each target task that is learned through a multiple perceptron (MLP).
6. The method of claim 5,
The bitrate loss function is composed of the sum of a first part based on a DC component and a second part based on an AC component,
The first part and the second part are each derived by a regression model trained through a multiple perceptron (MLP).
7. The method of claim 6,
The second part is a method of generating a quantization table for each target task in which a result derived by a regression model learned through LSTM (Long Short-Term Memory) is used as an input variable.
3. The method of claim 2,
The regression model is a method of generating a quantization table for each target task using training data that is used by synthesizing random data.
According to claim 1,
wherein the optimal value minimizes the quantization variable.
According to claim 1,
The target task is any one of preset quantization table generation method for each target task.
11. The method of claim 10,
The preset tasks include image classification, image segmentation, image captioning, and image quality. A method of generating a quantization table for each target task.
12. The method of claim 11,
The method of generating a quantization table for each target task, wherein the distortion loss function corresponding to the preset tasks includes a classification error, an MSE, and a perceptual loss.
According to claim 1,
The method of generating a quantization table for each target task further comprising; determining a bit rate based on the generated quantization table.

Images