KR20230140755A - Method and Apparatus for Improving Video Compression Performance for Video Codecs - Google Patents

Abstract

The disclosed embodiment includes the steps of performing a neural network operation on an applied input image to obtain at least one compression parameter according to the characteristics of the input image, and adjusting the weight of an artificial neural network that performs preprocessing on the input image according to the at least one compression parameter. Step, preprocessing the input image with an artificial neural network with adjusted weights to obtain a compressed image, and encoding the compressed image to obtain an encoded image, adaptively adapting the input image according to the characteristics of the input image. By compressing and restoring temporal and spatial resolution, we provide a method and device for improving image compression performance that can not only significantly improve the compression rate, but also suppress deterioration in the quality of restored images as much as possible even at the improved compression rate.

Description

Method and apparatus for improving video compression performance {Method and Apparatus for Improving Video Compression Performance for Video Codecs}

The disclosed embodiments relate to a method and device for improving video compression performance, and to a method and device for improving video compression performance that can improve the efficiency of a video codec by adaptively adjusting spatiotemporal resolution according to the characteristics of the input video.

As video communication and video-based OTT (Over-The-Top media service) services rapidly increase, the importance of video compression or codec technology to effectively transmit or store video signals continues to increase. Existing legacy video codecs such as H.26L and HEVC (High Efficiency Video Codec) codec, which are implemented based on signal processing and probability/statistics theory, have been developed into various application fields through international standardization work such as ITU-T and MPEG. is being successfully commercialized.

Meanwhile, the size of video display screens is continuously increasing to address the needs of users who prefer high-definition video, and the resolution of video in time and space is also rapidly increasing. As a result, the amount of information required to express high-definition video has also increased, and the memory capacity required to store high-definition video with the increased information amount and the network capacity for streaming services are also increasing exponentially.

Therefore, image compression technology is being actively researched to efficiently process high-definition images. Video compression technology is a technology that expresses video data with a small number of bits while maintaining the quality of the video as close to the original as possible. It is a technology to improve the efficiency of transmission and storage by reducing the amount of data to express the video. Codecs can also be seen as part of video compression technology.

Meanwhile, due to the development of deep learning technology using artificial neural networks, super-resolution techniques and frame interpolation techniques that can restore compressed images close to the original images have been proposed even in low-level computer vision. Using deep learning technology, even if the scale or frame rate of the input image is reduced, it can be restored more closely to the original image. In other words, compared to the existing hand-crafted algorithm, the occurrence of distortion can be reduced as much as possible when restoring compressed images. Accordingly, in video compression, an artificial neural network is used to perform preprocessing by reducing the temporal and spatial resolution of the input video, and the reduced video is encoded using an existing codec to significantly increase the compression rate, and the compressed bitstream is transferred to the decoder of the existing codec. A method of restoring the resolution of the image using an artificial neural network after performing decoding was also proposed.

Compression techniques that use these codecs and artificial neural networks together with existing video codec modules have achieved excellent compression performance improvements. However, since the artificial neural network that performs the pre- and post-processing tasks of the video codec is trained using an end-to-end learning method, there is a limitation in that the weights are fixed during learning and still do not clearly reflect the various characteristics of the video.

If the size and frame rate of the input video are reduced during the encoding process using video compression technology, the total amount of information in the video is reduced, so the bitrate of the encoded bitstream can be compressed to be lower than that of the original video. However, when input images with various characteristics are compressed in the same way, information loss occurs in different sizes depending on the characteristics of the input images. Therefore, when bilinear downsampling and frame rate downsampling are simply performed, the distortion of the recovered image may be significantly greater than the bitrate gain. For example, in the past, the spatial resolution of the input video signal was reduced to N:1 format, input into a legacy codec, encoded/decoded, and then decoded using a super-resolution technique to restore the original video resolution. Alternatively, a method of restoring time axis resolution using a frame interpolation technique was used. However, since the temporal and spatial characteristics of the image change depending on the type of content of the input image or the shooting method, etc., if the super-resolution technique and frame interpolation technique are simply applied, the image quality of the restored image may deteriorate. In other words, there is a problem that compression performance cannot be considered to be generally improved because compression performance is not uniform depending on the characteristics of the input image.

Korean Patent Publication No. 10-2018-0119753 (published on November 5, 2018)

The disclosed embodiments provide a method and device for improving image compression performance that can improve the compression rate by compressing and restoring images differently depending on the characteristics of the input image.

The disclosed embodiments provide a method and device for improving image compression performance that can minimize degradation of the quality of restored images while improving compression rates.

A method for improving image compression performance according to an embodiment includes performing a neural network operation on an applied input image to obtain at least one compression parameter according to characteristics of the input image; adjusting weights of an artificial neural network that performs preprocessing on the input image according to the at least one compression parameter; Obtaining a compressed image by performing preprocessing on the input image using an artificial neural network with adjusted weights; and encoding the compressed video to obtain an encoded video.

The step of adjusting the weights includes mixing the weights constituting the main kernel and at least one sub-kernel included in each of the plurality of computational layers of the artificial neural network performing the preprocessing at a ratio according to the at least one compression parameter, The weight of the calculation layer can be adjusted.

In the step of adjusting the weights, the weights can be adjusted by adding a ratio (1-α) according to the compression parameter to the weights constituting the main kernel and adding the compression parameter (α) to the weights constituting the subkernel. .

In the step of adjusting the weights, if there are multiple compression parameters, the weights of the main kernel are adjusted by mixing the weights of the subkernels in a ratio according to the weight compression parameters of the main kernel, and then, the weights of the main kernel adjusted by the previous compression parameters are adjusted. The weights of subkernels can be adjusted by mixing them in proportions according to different compression parameters sequentially with the kernel weights.

The step of obtaining at least one compression parameter includes two or more artificial neural networks, each of the two or more artificial neural networks receives the input image and performs neural network operation, and a scale parameter indicating a ratio for reducing the size of the input image. and a complexity parameter representing the intra- and inter-frame complexity of the input image can be obtained as the compression parameter.

The step of acquiring the compressed image includes outputting a feature map by performing a neural network operation on the input image using a plurality of calculation layers of an artificial neural network with adjusted weights, and if the scale parameter is included in the at least one compression parameter, According to the scale parameter, the scale down layer can downscale the feature map to obtain the compressed image.

In the step of acquiring the encoded video, the encoded video may be obtained by including the at least one compression parameter in the encoded video.

The method for improving video compression performance includes receiving and decoding the encoded video to obtain a decoded video, and extracting the at least one compression parameter included in the encoded video; adjusting post-processing weights of an artificial neural network that performs post-processing on the decoded image according to the at least one compression parameter; And it may further include performing post-processing on the decoded image using an artificial neural network with adjusted weights to obtain a restored image.

The step of adjusting the weights includes mixing the weights constituting the main kernel and at least one sub-kernel included in each of the plurality of computational layers of the artificial neural network performing the post-processing at a ratio according to the at least one compression parameter. The weight of each computational layer can be adjusted.

The step of acquiring the reconstructed image includes outputting a feature map by performing a neural network operation on the input image using a plurality of calculation layers of an artificial neural network with adjusted weights, and if the at least one compression parameter includes a scale parameter, the scale Depending on the parameter, the scale-up layer may upscale the feature map to obtain the restored image.

An apparatus for improving image compression performance according to an embodiment includes one or more processors; and a memory that stores one or more programs executed by the one or more processors, wherein the processor performs a neural network operation on the input image to obtain at least one compression parameter according to the characteristics of the input image, Adjusting the weight of an artificial neural network that performs preprocessing on the input image according to at least one compression parameter, performing preprocessing on the input image with the artificial neural network with the adjusted weight to obtain a compressed image, and obtaining the compressed image. Encode to obtain the encoded video.

Therefore, the image compression performance improvement method and device according to the embodiment analyzes the characteristics of the input image and adaptively compresses and restores the temporal and spatial resolution of the input image according to the characteristics of the input image, thereby significantly improving the compression rate. Not only that, but even with the improved compression rate, the quality deterioration of the restored video can be suppressed as much as possible.

Figure 1 shows a configuration divided according to operations performed in an apparatus for improving video compression performance according to an embodiment.
Figure 2 shows a schematic configuration of the pre-processing network and post-processing network of Figure 1.
FIG. 3 shows an example of a detailed configuration of multiple computational layers of the pre-processing network and post-processing network of FIG. 2.
Figure 4 shows an example of the detailed configuration of the parameter filter of Figure 4.
Figure 5 shows a method for improving image compression performance according to an embodiment.
FIG. 6 is a diagram for explaining a computing environment including a computing device according to an embodiment.

Hereinafter, specific embodiments of one embodiment will be described with reference to the drawings. The detailed description below is provided to provide a comprehensive understanding of the methods, devices and/or systems described herein. However, this is only an example and the present invention is not limited thereto.

In describing one embodiment, if it is determined that a detailed description of the known technology related to the present invention may unnecessarily obscure the gist of an embodiment, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification. The terminology used in the detailed description is intended to describe only one embodiment and should in no way be limiting. Unless explicitly stated otherwise, singular forms include plural meanings. In this description, expressions such as "comprising" or "having" are intended to indicate any features, numbers, steps, operations, elements, part or combination thereof, and one or more other than those described. It should not be construed to exclude the existence or possibility of other characteristics, numbers, steps, operations, elements, or parts or combinations thereof. In addition, "... part", "... period", " described in the specification. Terms such as “module” and “block” refer to a unit that processes at least one function or operation, which may be implemented as hardware, software, or a combination of hardware and software.

Figure 1 shows a configuration divided according to operations performed in an apparatus for improving video compression performance according to an embodiment.

In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components other than those described below. Additionally, in one embodiment, each component may be implemented using one or more physically separate devices, one or more processors, or a combination of one or more processors and software, and, unlike the example shown, may be implemented in specific operations. It may not be clearly distinguished.

Additionally, the image compression performance improvement device shown in FIG. 1 may be implemented in a logic circuit using hardware, firmware, software, or a combination thereof, and may also be implemented using a general-purpose or special-purpose computer. The device may be implemented using hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. Additionally, the device may be implemented as a System on Chip (SoC) including one or more processors and a controller.

In addition, the image compression performance improvement device may be mounted on a computing device or server equipped with hardware elements in the form of software, hardware, or a combination thereof. A computing device or server includes all or part of a communication device such as a communication modem for communicating with various devices or a wired or wireless communication network, a memory for storing data to execute a program, and a microprocessor for executing a program to perform calculations and commands. It can refer to a variety of devices, including:

Referring to FIG. 1, an apparatus for improving image compression performance according to an embodiment may include an image compression module 100 and an image restoration module 200. Here, the image compression module 100 and the image restoration module 200 may be implemented as an image compression device and an image restoration device, respectively. In this case, each of the image compression device and the image restoration device may further include a communication module and operate as an image transmission device and an image reception device.

The image compression module 100 acquires an image and compresses the acquired image for storage or transmission, and the image restoration module 200 receives and restores the compressed image that has been stored or transmitted.

In particular, in the embodiment, the image compression module 100 determines the characteristics of the image acquired using an artificial neural network, performs compression by adaptively adjusting the scale or compression rate of the image according to the determined characteristics, and the image restoration module ( 200) Additionally, restoration is performed by reflecting the scale or compression rate of the image applied according to the image characteristics determined by the image compression module 100. Therefore, by performing compression and restoration at different scales and compression rates according to the characteristics of each image, not only can the compression rate be improved, but also the deterioration of the quality of the restored image can be suppressed as much as possible. In addition, the video compression performance improvement device makes full use of existing legacy codecs, making it as compatible as possible with standard video compression techniques.

Specifically, the image compression module 100 may include an image acquisition module 110, an image characteristic estimation module 120, a preprocessing network 130, and an encoder 140.

The image acquisition module 110 acquires an input image to be compressed. The image acquisition module 110 acquires an input image whose data amount must be reduced for storage or transmission. At this time, the acquired input image may be a still image of a single frame or a moving image composed of multiple frames. The image acquisition module 110 may be implemented as a camera module that directly acquires the input image, or may be implemented as a computer-readable storage medium such as a memory in which the acquired input image is stored.

The image characteristic estimation module 120 performs a neural network operation with an artificial neural network learned on the input image acquired in the image acquisition module 110, estimates the characteristics of the input image, and obtains compression parameters according to the estimated image characteristics. do. Compression parameters are parameters determined to enable adaptive compression of input images at different compression rates, and are determined taking into account the characteristics of the input images.

When the input image is an image with a simple structure, such as an image with a very monotonous structure or a monochromatic flat image, and an image with a complex structure containing many objects and various backgrounds and colors, the image characteristics are very different. In this case, when images with very different image characteristics are to be compressed at the same compression rate, performing compression based on a simple image can greatly improve the compression rate, but the quality of the restored complex image is greatly reduced. On the other hand, if compression is performed based on a complex image, image quality can be maintained, but the compression rate becomes very low, so the meaning of image compression fades.

Accordingly, in the embodiment of FIG. 1, the image characteristic module 120 determines the characteristics of the image in advance to obtain compression parameters, and the preprocessing network 130 compresses the input image differently according to the obtained compression parameters. At this time, the compression parameters that the image characteristics module 120 acquires according to the characteristics of the image may be set in various ways, but here, as an example, the image characteristics module 120 uses the scale parameter (β) and complexity parameter (α) as compression parameters. Assume it is obtained. Here, each compression parameter (α, β) can be obtained as a real value in the range between [0,1].

In Figure 1, it is assumed that the image characteristic estimation module 120 obtains the scale parameter (β) and the complexity parameter (α) as compression parameters, so the image characteristic estimation module 120 is implemented as an artificial neural network. ) and a complexity determination network 122. However, the image characteristic estimation module 120 may be equipped with a separate artificial neural network for extracting compression parameters classified according to the type of characteristic of the image to be acquired.

The scale determination network 121 receives the input image, performs neural network calculations, and determines a scale parameter (β) for adjusting the resolution of the input image. The input image may be downscaled to a level that can be restored depending on the structural form of the background or object included in the image. In other words, even if the resolution of the input image is scaled down to a lower resolution image, the quality of the reconstructed image restored to the resolution of the input image may not be significantly different from the quality of the input image. However, as mentioned above, in the case of images containing a single color or objects with a simple structure, it is possible to obtain a restored image that is not significantly different from the original input image even if the scale down ratio is greatly increased, whereas objects with a complex structure can be obtained. In the case of restored images, increasing the scale down ratio may significantly deteriorate the quality of the restored images. For example, if an input image with UHD (Ultra High Definition) resolution of 3,080 It indicates quality.

Accordingly, the scale determination network 121 may perform a neural network operation on the input image to estimate scale characteristics according to the resolution of the input image, and estimate scale parameters suitable for the input image according to the estimated scale characteristics. At this time, the scale determination network 121 may be configured to obtain a uniform ratio of scale parameters (β) in the horizontal and vertical directions, but in some cases, scale parameters (β1, β2) are obtained for the horizontal and vertical directions, respectively. It may be configured to do so.

This is because different characteristics may appear in the horizontal and vertical directions depending on the configuration of the input image. For example, in the case of an image of a street tree or a streetlight, even if the scale is scaled down at a greater rate in the horizontal direction than in the vertical direction, a high-quality image can be easily restored later. On the other hand, if the scale is scaled down to a larger ratio in the vertical direction, it is difficult to restore high-quality video. Accordingly, the scale determination network 121 may separately obtain scale parameters (β1, β2) for each of the horizontal and vertical directions so as to improve compression performance and easily restore the image later.

Meanwhile, the complexity determination network 122 performs a neural network operation on the input image to estimate the complexity within each frame of the input image and between adjacent frames, and obtains a complexity parameter (α) according to the estimated complexity. The complexity of an image is related to the scale of the image resolution, but apart from the scale, it is also greatly affected by the characteristics of the image itself. For example, the complexity of an image of a monochromatic wall and an image of a forest or a plaza with many people can be seen as having different frame complexity regardless of scale. In other words, the complexity within a frame may appear different depending on the similarity between pixels within the frame.

Additionally, in the case of an image composed of multiple frames, the frame complexity is different between an image of a stationary object and an image of a fast-moving object. In other words, complexity between frames may appear different depending on pixel changes between different frames.

Image compression and restoration according to the complexity of the image needs to be separated from image compression and restoration according to scale down and scale up. Accordingly, in this embodiment, the complexity determination network 122 is configured separately from the scale determination network 121 to obtain the complexity parameter (α).

The preprocessing network 130 is implemented as a pre-trained artificial neural network and compresses the input image by performing a neural network operation on the input image. At this time, the preprocessing network 130 compresses the input image differently according to the compression parameters obtained from the image characteristic estimation module 120. The preprocessing network 130 performs compression on the input image by varying the weight of the artificial neural network according to the compression parameters. Here, since the image characteristic estimation module 120 is assumed to obtain the scale parameter (β) and complexity parameter (α) as compression parameters, the preprocessing network 130 calculates the input image differently depending on the complexity parameter (α). While performing compression, the input image can be scaled down according to the scale parameter (β) to obtain a compressed image.

The encoder 140 receives compressed video, encodes it in a designated manner, and outputs the encoded video. Here, the encoder 140 may be implemented as an encoder of an existing legacy codec. The encoder 140 is also used to compress images, but performs image compression in a different way from the preprocessing network 130 implemented with an artificial neural network. However, the preprocessing network 130 implemented as an artificial neural network may be viewed as performing preprocessing on the input image to maximize the image compression efficiency of the encoder 140 depending on the characteristics of the input image. In other words, the preprocessing network 130 does not simply compress the input video to the maximum by self-performing video compression, but the encoder 140 additionally encodes the video to maximize the compression rate of the output encoded video. Can be learned to perform.

And the encoder 140 can add compression parameters to the encoded video and output it. When encoding a video with a legacy codec, various encoding information is included in the encoded video, and among the data space for this encoding information, there is a blank space where no data is recorded for information that can be added later. Accordingly, in this embodiment, the encoder 140 may output the bit string of the compression parameter as side information in the blank space of the encoded video.

Meanwhile, the image restoration module 200 includes a decoder 210 and a post-processing network 220. The decoder 210 receives the encoded video output from the encoder 140, decodes it, and outputs a decoded video. Like the encoder 140, the decoder 210 is implemented as a decoder of a legacy codec and can decode encoded video. Then, the decoder 210 decodes the encoded image to obtain a decoded image, extracts compression parameters included in the encoded image, and transmits them to the post-processing network 220.

The post-processing network 220 is implemented as an artificial neural network, receives the decoded image and compression parameters from the decoder 210, and obtains a restored image by performing different neural network operations on the decoded image according to the applied compression parameters. At this time, the post-processing network 220, like the pre-processing network 130, adjusts the weights according to the applied compression parameters and performs a neural network operation on the decoded image according to the adjusted weights to obtain a restored image. And the post-processing network 220 is also trained to maximize image quality based on the decoded image decoded by the decoder 21.

FIG. 2 shows a schematic configuration of the pre-processing network and the post-processing network of FIG. 1, FIG. 3 shows an example of the detailed configuration of a plurality of computational layers of the pre-processing network and the post-processing network of FIG. 2, and FIG. 4 shows the schematic configuration of the pre-processing network and the post-processing network of FIG. Shows an example of the detailed configuration of a parameter filter.

In FIG. 2, (a) shows the schematic structure of the pre-processing network 130, and (b) shows the schematic structure of the post-processing network 220. As shown in (a) and (b) of Figure 2, the pre-processing network 130 and the post-processing network 220 are artificial neural networks that each include a plurality of computational layers (L1 to Ln) and perform neural network operations. It can be implemented. And when the pre-processing network 130 and the post-processing network 220 perform scale-down or scale-up on the applied image, the pre-processing network 130 and the post-processing network 220 have a plurality of operation layers (L1 to Ln). ), a scale-down layer (Lsd) or a scale-up layer (Lsu) may be further included after the last layer (Ln).

Each of the multiple calculation layers (L1 to Ln) performs calculations with weights specified through learning on images input to the network or feature maps output from the previous layer and outputs them. Here, it is assumed that the pre-processing network 130 and the post-processing network 220 are implemented based on a convolutional network (CNN), which is mainly used in image processing, and each of the plurality of operation layers (L1 to Ln) is used to process the input image. Alternatively, a convolution operation can be performed with weights on the feature map and output. However, in this embodiment, each of the multiple calculation layers (L1 to Ln) can perform calculations while adjusting weights according to compression parameters. In other words, unlike the calculation layer of a general convolutional network, the calculation can be performed and output by varying the weights according to the compression parameters. Therefore, even if the same image or feature map is applied, the calculations are performed with different weights depending on the compression parameters, resulting in different results. A feature map can be output.

FIG. 3 shows the detailed configuration of the ith computational layer (Li) among the plurality of computational layers (L1 to Ln) as an example, and other computational layers may also have the same configuration. Referring to FIG. 3, the calculation layer (Li) receives the feature map (FMi-1) output from the previous calculation layer (Li-1). However, when the computation layer (Li) is the first computation layer (L1) of the preprocessing network 130, the computation layer (L1) receives the input image. And when the computation layer (Li) is the first computation layer (L1) of the post-processing network 220, the computation layer (L1) receives the decoded image.

And the operation layer (Li) may include a main kernel (MK) and a number of parameter filters (PFL1 to PFLm). Here, the main kernel (MK) and multiple parameter filters (PFL1 to PFLm) each have weights specified by learning. At this time, the main kernel (MK) has weights that are unrelated to the compression parameters. However, multiple parameter filters (PFL1 to PFLm) each have a weight for adjusting the weight of the main kernel (MK) or the weight adjusted in the previous parameter filter at a ratio according to the specified compression parameter.

Each of the multiple parameter filters (PFL1 to PFLm) adds its own weight to the weight of the main kernel (MK) or the weight adjusted in the previous parameter filter in a ratio according to the compression parameters, so that the operation layer (Li) It allows calculations to be performed with variable weights. The number (m) of parameter filters (PFL1 to PFLm) included in the calculation layer (Li) may be determined according to the number of compression parameters acquired by the image characteristic estimation module 120. As an example, as described above, when the image characteristic estimation module 120 obtains two compression parameters, the complexity parameter (α) and the scale parameter (β), the plurality of operation layers (L1 to Ln) each obtain two parameters. May include filters (PFL1, PFL2). At this time, among the two parameter filters (PFL1, PFL2), the first parameter filter (PFL1) adjusts the weight of the main kernel (MK) according to the complexity parameter (α), and the second parameter filter (PFL1) adjusts the weight of the main kernel (MK) according to the scale parameter (β). Accordingly, the weight of the main kernel (MK) adjusted in the first parameter filter (PFL1) can be additionally adjusted.

However, when the image characteristic estimation module 120 obtains two scale parameters (β1, β2) by dividing the scale parameter (β) in the horizontal and vertical directions, each of the plurality of operation layers (L1 to Ln) has two The scale parameters can also be divided into three parameter filters (PFL1 to PFL3).

Each of the plurality of parameter filters (PFL1 to PFLm) adjusts the sub-weights stored for the output of the main kernel (MK) or the previous parameter filter by weighting them with a weighting ratio according to the compression parameter. That is, each of the multiple parameter filters (PFL1 to PFLm) plays the role of adjusting the weights constituting the main kernel (MK) according to the corresponding compression parameters.

Figure 4 shows the detailed configuration of the first parameter filter (PFL1) among the plurality of parameter filters (PFL1 to PFLm) as an example. It is assumed that the first parameter filter (PFL1) receives the complexity parameter (α) among the compression parameters. did.

Referring to FIG. 4, the parameter filter (PFL1) may include a filter transfer network 410, a main alleviation module 420, a subkernel 430, a subweighting module 440, and a kernel mixing module 450. .

The filter transition network 410 receives the output of the main kernel (MK) or the previous parameter filter and connects it to the subkernel 430. The filter transition network 410 is a configuration for delivering the output of the main kernel (MK) or the previous parameter filter to the subkernel 430. For example, two 1 1 It can be implemented with PReLU (Parametric Rectified Linear Unit), which is an activation function located between convolutional filters.

The main reduction module 420 reduces the output of the main kernel (MK) or the previous parameter filter at a ratio (1-α) according to the compression parameter (α). The sub-kernel 430 is composed of sub-weights designated according to the type of compression parameter (α) by learning in advance. The sub-weighting module 440 weights the compression parameter (α) to the sub-weights constituting the sub-kernel 430. And the kernel mixing module 450 adds the relief weight reduced at a ratio (1-α) according to the compression parameter (α) and the sub-weight weighted by the compression parameter (α) to adjust the weight according to the compression parameter (α). Obtain the kernel.

Then, an operation is performed on the image or feature map applied as the weight of the weight-adjusted kernel and output.

Meanwhile, the scale conversion layer (Lsd) or scale up layer (Lsu) of the pre-processing network 130 and the post-processing network 220 receives the scale parameter (β) among the compression parameters (α, β). Then, the feature map output from the last calculation layer (Ln) is scaled down or scaled up to have a resolution according to the applied scale parameter (β). At this time, the scale conversion layer (Lsd) or scale up layer (Lsu) scales the feature map to a real value scale in the range [1, 2] in response to the scale parameter (β) with a value in the range [0, 1]. You can scale down or scale up. And when the scale parameter (β) is divided into horizontal and vertical directions, a scale conversion layer (Lsd) and a scale up layer (Lsu) are also divided into two horizontal and vertical directions. It may be composed of layers (Lsu).

At this time, the scale conversion layer (Lsd) or scale up layer (Lsu) also includes a number of parameter filters (PFL1 to PFLm) to adjust the weights according to the compression parameters (α, β) and then perform scale down or scale up. You can.

Since the image characteristic estimation module 120, the pre-processing network 130, and the post-processing network 220 are implemented as an artificial neural network, they must be learned in advance and the weights must be set. Accordingly, the image compression performance improvement device uses the learning module 310 during training. It can be further provided. The learning module 310 inputs the learning image as an input image to the image compression performance improvement device, receives the restored image output from the post-processing network 220, analyzes it, and backpropagates the loss, thereby being included in the image compression performance improvement device. An artificial neural network can be trained. However, the codec consisting of the encoder 140 and the decoder 210 has a non-differentiable quantization operation, making it impossible to transfer the gradient for lossy backpropagation. To solve this problem, learning can be performed using a codec simulation network that can replace the encoder 140 and decoder 210 by modeling the codec. The codec simulation network for the legacy codec is a known technology, and a separately trained artificial neural network can be used. Therefore, it will not be explained in detail here.

The learning module 310 may perform learning by back-propagating the loss resulting from the difference between the training image and the restored image to a post-processing network. At this time, the learning module 310 classifies the effective compression scale of the multiple learning images according to the maximum signal-to-noise ratio (PSNR) and bitrate, and the intra-frame complexity and inter-frame complexity. The complexity of the training images can be classified according to complexity, and the compression parameters of the classified training images can be labeled.

When learning using a training image, the learning module 310 calculates the error between the compression parameters obtained in the image characteristic estimation module 120 and the compression parameters labeled in the learning image as a loss and backpropagates it, thereby calculating the error between the compression parameters obtained in the image characteristic estimation module 120 and the compression parameters labeled in the learning image. 120) can be learned.

However, the video compression performance improvement device of this embodiment not only includes a sub-kernel along with the main kernel, but the weights of the main kernel and sub-kernel are mixed at different ratios according to compression parameters. Additionally, compression parameters also vary depending on the video. Therefore, when learning is performed using all training data with various compressible scales and complexities, the kernel weights of the artificial neural network may take a very long time to converge or may not converge. In order to solve this problem and reduce the learning time, the learning module 310 first performs learning based on the learning image having the compression scale and complexity with the minimum value. In this case, the compression parameters (α, β) of the learning data are all labeled with a value of 0, which not only excludes the influence of the compression parameters (α, β), but also excludes the computational layers (L1 to Lm) of each artificial neural network. Several parameter filters (PFL1 to PFLm) are also disabled. Therefore, the learning module 310 can learn the weights of the main kernel (MK) of each artificial neural network. Afterwards, learning is performed by sequentially inputting training images in which only one of the multiple compression parameters (α, β) has the maximum value or only one has the minimum value, thereby learning the weight of the subkernel according to each of the multiple compression parameters. You can do it. Additionally, the learning module 310 can ultimately perform detailed adjustments to the weights of the kernel by inputting learning images regardless of the labeled parameters.

Figure 5 shows a method for improving image compression performance according to an embodiment.

When explaining the image compression performance improvement method of FIG. 5 with reference to FIGS. 1 to 4, first, an input image to be stored or transmitted is obtained (51). Then, the learned artificial neural network receives the input image, performs neural network operations, extracts features of the input image, and obtains compression parameters (51). At this time, a number of compression parameters can be obtained by extracting different features of the input image using a number of different artificial neural networks. For example, scale parameters and image frames that indicate resolution characteristics that can be compressed while minimizing image loss are used. Complexity parameters representing intra- and inter-frame complexity can be obtained.

When a number of compression parameters are obtained, the weights of the artificial neural network that performs preprocessing on the input image are adjusted using the obtained compression parameters (53). At this time, the artificial neural network that performs preprocessing includes multiple operation layers (L1 to Ln) and at least one scale down layer (Lsd), and each of the multiple operation layers (L1 to Ln) has a weight that is unrelated to the compression parameter. It may include one main kernel and a plurality of sub-kernels according to each of a plurality of compression parameters obtained separately for each image characteristic. And each of the multiple operation layers (L1 to Ln) has a weight that is a mixture of the main kernel and multiple subkernels according to the ratio specified as the compression parameter.

When the weights of multiple calculation layers (L1 to Ln) of the artificial neural network are adjusted according to the compression parameters, the artificial neural network performing preprocessing receives the input image, performs neural network calculation with the adjusted weights, and outputs a feature map, at least One scale down layer (Lsd) obtains a compressed image by receiving a feature map calculated by a neural network according to the scale parameter among the compression parameters and performing preprocessing to scale down (54).

The compressed video is encoded in the encoder of the legacy codec and obtained as an encoded video (55). At this time, the encoder can include compression parameters by adding them to the encoded video.

The encoded video is applied to the decoder of the legacy codec, and the decoder decodes the applied encoded video to obtain a decoded video (56). Then, the compression parameters included in the encoded video are extracted (57).

Once the compression parameters are extracted, the weights of the artificial neural network that performs post-processing on the decoded image are adjusted using the extracted compression parameters (58). At this time, similar to the artificial neural network that performs pre-processing, the artificial neural network that performs post-processing includes a plurality of operation layers (L1 to Ln) and at least one scale-up layer (Lsu). Accordingly, each of the multiple operation layers (L1 to Ln) has a weight that is a mixture of the main kernel and multiple subkernels according to the ratio specified as the compression parameter.

When the weights of multiple calculation layers (L1 to Ln) of the artificial neural network are adjusted according to the compression parameters, the artificial neural network that performs post-processing receives the decoded image, performs neural network calculation with the adjusted weights, and outputs a feature map. At least one scale-up layer (Lsu) obtains a restored image by receiving a feature map calculated by a neural network according to a scale parameter among compression parameters and performing post-processing to scale up the feature map (59).

In FIG. 5, each process is described as being sequentially executed, but this is only an illustrative explanation, and those skilled in the art can change the order shown in FIG. 5 and execute it without departing from the essential characteristics of the embodiments of the present invention. Alternatively, it can be applied through various modifications and modifications by executing one or more processes in parallel or adding other processes.

FIG. 6 is a diagram for explaining a computing environment including a computing device according to an embodiment.

In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those not described below. The illustrated computing environment 60 includes a computing device 61 . In one embodiment, the computing device 61 may be one or more components included in the image compression performance improvement device shown in FIG. 1.

Computing device 61 includes at least one processor 62, computer-readable storage medium 63, and communication bus 65. Processor 62 may cause computing device 61 to operate according to the above-mentioned example embodiments. For example, the processor 62 may execute one or more programs 64 stored in the computer-readable storage medium 63. The one or more programs 64 may include one or more computer-executable instructions, which, when executed by the processor 62, cause the computing device 61 to operate according to an example embodiment. It can be configured to perform these.

Communication bus 65 interconnects various other components of computing device 61, including processor 62 and computer-readable storage medium 63.

Computing device 61 may also include one or more input/output interfaces 66 and one or more communication interfaces 67 that provide an interface for one or more input/output devices 68 . The input/output interface 66 and communication interface 67 are connected to the communication bus 65. Input/output device 68 may be connected to other components of computing device 61 through input/output interface 66. Exemplary input/output devices 68 include, but are not limited to, a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touch screen), a voice or sound input device, various types of sensor devices, and/or imaging devices. It may include input devices and/or output devices such as display devices, printers, speakers, and/or network cards. The exemplary input/output device 68 is a component constituting the computing device 61 and may be included within the computing device 61, or may be a separate device distinct from the computing device 61 and may be included in the computing device 61. It may be connected.

Although the present invention has been described in detail through representative embodiments above, those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached claims.

100: video compression module 110: video acquisition module
120: Image feature estimation module 121: Scale decision network
122: Complexity determination network 130: Preprocessing network
140: Encoder 200: Video restoration module
210: decoder 220: post-processing network
410: Filter transition network 420: Main mitigation module
430: subkernel 440: subweighting module
450: Kernel mixing module Ln1 ~ Ln: Operation layer
Lsd: scale down layer Lsu: scale up layer
MK: Main kernel PFL1 ~ PFLm: Parameter filter

Claims (20)

1. A method performed by a computing device having one or more processors and a memory that stores one or more programs to be executed by the one or more processors, comprising:
Obtaining at least one compression parameter according to the characteristics of the input image by performing a neural network operation on the applied input image;
adjusting weights of an artificial neural network that performs preprocessing on the input image according to the at least one compression parameter;
Obtaining a compressed image by performing preprocessing on the input image using an artificial neural network with adjusted weights; and
A method of improving video compression performance, comprising the step of encoding the compressed video to obtain an encoded video. The method of claim 1, wherein adjusting the weight
Adjusting the weight of each calculation layer by mixing the weights constituting the main kernel and at least one sub-kernel included in each of the plurality of calculation layers of the artificial neural network that performs the preprocessing in a ratio according to at least the at least one compression parameter. How to improve video compression performance. The method of claim 2, wherein the step of adjusting the weight is
A method for improving video compression performance in which the weights are adjusted by adding a ratio (1-α) according to the compression parameter to the weights constituting the main kernel and adding the compression parameters (α) to the weights constituting the sub-kernel. The method of claim 3, wherein the step of adjusting the weight is
If there are multiple compression parameters, the weight of the main kernel is adjusted by mixing the weights of the subkernels in a ratio according to the weight compression parameter of the main kernel, and then sequentially different compression is performed on the main kernel weight adjusted by the previous compression parameter. A method of improving video compression performance by mixing and adjusting the weights of subkernels at a ratio according to parameters. The method of claim 1, wherein obtaining the at least one compression parameter
It includes two or more artificial neural networks, each of the two or more artificial neural networks receives the input image and performs a neural network operation, and includes a scale parameter indicating a ratio for reducing the size of the input image and intra-frame and inter-frame complexity of the input image. A method for improving image compression performance by obtaining a complexity parameter representing as the compression parameter. The method of claim 5, wherein the step of acquiring the compressed image is
A feature map is output by performing a neural network operation on the input image using a plurality of calculation layers of an artificial neural network with adjusted weights, and if the scale parameter is included in the at least one compression parameter, a scale down layer is performed according to the scale parameter. A method for improving image compression performance by downscaling a feature map to obtain the compressed image. The method of claim 1, wherein the step of acquiring the encoded image is
A method for improving video compression performance, wherein the encoded video is obtained by including the at least one compression parameter in the encoded video. The method of claim 7, wherein the image compression performance improvement method is
Obtaining a decoded video by receiving and decoding the encoded video, and extracting the at least one compression parameter included in the encoded video;
adjusting post-processing weights of an artificial neural network that performs post-processing on the decoded image according to the at least one compression parameter; and
A method for improving image compression performance further comprising performing post-processing on the decoded image using an artificial neural network with adjusted weights to obtain a restored image. The method of claim 8, wherein the step of adjusting the weight is
The weights of each calculation layer are adjusted by mixing the weights constituting the main kernel and at least one sub-kernel included in each of the plurality of calculation layers of the artificial neural network that performs the post-processing in a ratio according to at least the at least one compression parameter. How to improve video compression performance. The method of claim 9, wherein the step of acquiring the restored image is
A feature map is output by performing a neural network operation on the input image using a plurality of calculation layers of an artificial neural network with adjusted weights, and if the at least one compression parameter includes a scale parameter, a scale-up layer features the scale parameter according to the scale parameter. A method for improving image compression performance by upscaling a map to obtain the restored image. One or more processors; and a memory storing one or more programs executed by the one or more processors,
The processor is
Performing a neural network operation on the applied input image to obtain at least one compression parameter according to the characteristics of the input image,
Adjusting the weights of an artificial neural network that performs preprocessing on the input image according to the at least one compression parameter,
Obtain a compressed image by performing preprocessing on the input image using an artificial neural network with adjusted weights,
A video compression performance improvement device that obtains an encoded video by encoding the compressed video. The method of claim 11, wherein the processor
Adjusting the weight of each calculation layer by mixing the weights constituting the main kernel and at least one sub-kernel included in each of the plurality of calculation layers of the artificial neural network that performs the preprocessing in a ratio according to at least the at least one compression parameter. Video compression performance improvement device. The method of claim 12, wherein the processor
An image compression performance improvement device that adjusts the weight by adding a ratio (1-α) according to the compression parameter to the weight constituting the main kernel and adding the compression parameter (α) to the weight constituting the sub-kernel. The method of claim 13, wherein the processor
If there are multiple compression parameters, the weight of the main kernel is adjusted by mixing the weights of the subkernels in a ratio according to the weight compression parameter of the main kernel, and then sequentially different compression is performed on the main kernel weight adjusted by the previous compression parameter. A video compression performance improvement device that mixes and adjusts the weights of subkernels at a ratio according to parameters. The method of claim 12, wherein the processor
Using two or more artificial neural networks that each receive the input image and perform neural network calculations, a scale parameter indicating a ratio for reducing the size of the input image and a complexity parameter indicating the intra- and inter-frame complexity of the input image are set. A device for improving video compression performance obtained as compression parameters. The method of claim 15, wherein the processor
A feature map is output by performing a neural network operation on the input image using the plurality of calculation layers with adjusted weights, and if the scale parameter is included in the at least one compression parameter, the feature map is downscaled according to the scale parameter. An image compression performance improvement device that obtains the compressed image. The method of claim 11, wherein the processor
An image compression performance improvement device for obtaining an encoded image by including the at least one compression parameter in the encoded image. The method of claim 17, wherein the processor
Obtaining a decoded image by receiving and decoding the encoded image, extracting the at least one compression parameter included in the encoded image,
Adjusting post-processing weights of an artificial neural network that performs post-processing on the decoded image according to the at least one compression parameter,
An image compression performance improvement device that obtains a restored image by performing post-processing on the decoded image using an artificial neural network with adjusted weights. The method of claim 18, wherein the processor
The weights of each calculation layer are adjusted by mixing the weights constituting the main kernel and at least one sub-kernel included in each of the plurality of calculation layers of the artificial neural network that performs the post-processing in a ratio according to at least the at least one compression parameter. A device that improves video compression performance. The method of claim 19, wherein the processor
A feature map is output by performing a neural network operation on the input image using a plurality of computational layers of an artificial neural network whose weights are adjusted and post-processing is performed. If the at least one compression parameter includes a scale parameter, the feature map is output according to the scale parameter. An image compression performance improvement device for obtaining the restored image by upscaling the feature map.