JP2022129683A - Image encoding device, image encoding method, image decoding device, and image decoding method - Google Patents

Abstract

To provide an image encoding device capable of compressing an input image without degrading its image quality.SOLUTION: An image encoding device 10 includes: a saliency map generation unit 12 that generates a saliency map of an input image from the input image; a gradient intensity map generation unit 13 that generates a gradient intensity map of the input image from the input image; an importance map generation unit 14 that generates an importance map of the input image from the saliency map and the gradient intensity map; and an image compression unit 15 that compresses the input image using the importance map.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image encoding device, an image encoding method, an image decoding device, and an image decoding method.

MPEG standard (H.265/HEVC) is known as a standard by ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) for compression encoding technology of moving images. HEVC defines encoding methods for 4K (3840×2160 pixels) images and 8K (7680×4320 pixels) images.

In the future, it is expected that the need for high-definition video playback such as 8K images will increase, and the development of technology that compresses large-capacity images without degrading quality and transmits them with low delay is desired. .

Accordingly, an object of the present invention is to propose an image coding apparatus and an image coding method capable of compressing an input image without degrading its image quality.

In order to solve the above-described problems, the image coding apparatus according to the present invention includes a saliency map generation unit that generates a saliency map of an input image from an input image, and a gradient map that generates a gradient intensity map of the input image from the input image. An intensity map generator, an importance map generator for generating an importance map of the input image from the saliency map and the gradient intensity map, and an image compressor for compressing the input image using the importance map. Image compression using an importance map can achieve efficient image compression without degrading image quality.

The image compression unit divides the input image into a plurality of blocks, and normalizes the value obtained by dividing the sum of the values of the importance map corresponding to each of the plurality of pixels in the block by the number of pixels in the block. A calculation unit that executes processing for each block to calculate a value as the sampling rate of the block, a sampling unit that samples each block at the sampling rate calculated for each block, and an encoding unit that encodes each sampled block. and may be provided. As a result, the sampling rate can be adaptively changed for each block according to the importance of each block, and efficient image compression can be realized without degrading the image quality.

An image decoding apparatus according to the present invention is an image decoding apparatus that decodes each block of an input image encoded by the image encoding apparatus according to the present invention, and decodes each encoded block. an image restoration unit that generates a restored image of the input image from each decoded block by an image interpolation method; and an image quality evaluator for calculating. By calculating the image quality evaluation value of the restored image with respect to the input image using the value of the importance map as a weighting factor, it is possible to perform appropriate image quality evaluation according to the importance of the image.

An image coding method according to the present invention comprises the steps of generating a saliency map of the input image from the input image; generating a gradient intensity map of the input image from the input image; and compressing the input image using the importance map. Image compression using an importance map can achieve efficient image compression without degrading image quality.

The step of compressing the input image includes dividing the input image into a plurality of blocks, and normalizing the sum of the values of the importance map corresponding to each of the plurality of pixels in the block divided by the number of pixels in the block. a step of calculating the unitized value as the sampling rate of the block for each block, a step of sampling each block at the sampling rate calculated for each block, and a step of encoding each sampled block. may contain. As a result, the sampling rate can be adaptively changed for each block according to the importance of each block, and efficient image compression can be realized without degrading the image quality.

An image decoding method according to the present invention is an image decoding method for decoding each block of an input image encoded by the image encoding method according to the present invention, and decoding each encoded block. generating a restored image of the input image from each decoded block by an image interpolation method; and calculating an image quality evaluation value of the restored image with respect to the input image using the value of the importance map as a weighting factor. including. By calculating the image quality evaluation value of the restored image with respect to the input image using the value of the importance map as a weighting factor, it is possible to perform appropriate image quality evaluation according to the importance of the image.

According to the present invention, an input image can be efficiently compressed without degrading its image quality.

1 is an explanatory diagram showing the hardware configuration of an image encoding device according to an embodiment of the present invention; FIG. 1 is a functional block diagram of an image encoding device according to an embodiment of the present invention; FIG. 1 is an explanatory diagram showing the hardware configuration of an image decoding device according to an embodiment of the present invention; FIG. 1 is a functional block diagram of an image decoding device according to an embodiment of the present invention; FIG. It is a figure which shows an example of the input image input into the image input part in connection with embodiment of this invention. It is a figure which shows an example of the saliency map produced|generated by the saliency map production|generation part in connection with embodiment of this invention. It is a figure which shows an example of the gradient intensity|strength map produced|generated by the gradient intensity|strength map production|generation part in connection with embodiment of this invention. It is a figure which shows an example of the input image sampled by the sampling part in connection with embodiment of this invention. It is a figure which shows an example of the input image decoded by the decoding part in connection with embodiment of this invention. It is a figure which shows an example of the input image restored by the image restoration part in connection with embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Here, the same reference numerals denote the same components, and duplicate descriptions are omitted.

FIG. 1 is an explanatory diagram showing the hardware configuration of an image encoding device 10 according to an embodiment of the invention. The image encoding device 10 includes a processor 101 , memory 102 , storage device 103 , camera 104 and communication device 105 . The storage device 103 stores software resources such as an operating system 106 and an image processing program 107 . These software resources are loaded into memory 102 and executed by processor 101 .

FIG. 2 is a functional block diagram of the image encoding device 10 according to the embodiment of the invention. Hardware resources such as the processor 101, the memory 102, the storage device 103, the camera 104, and the communication device 105 cooperate with software resources such as the operating system 106 and the image processing program 107 to generate the image input unit 11 and the saliency map. The functions of the generation unit 12, the gradient intensity map generation unit 13, the importance map generation unit 14, the image compression unit 15, and the communication unit 16 are realized.

The image input unit 11 takes in input images that form a moving image captured by the camera 104 .

The saliency map generator 12 generates a saliency map of the input image from the input image. A saliency map indicates the saliency of each region in the input image, that is, the spatial distribution of salience in human vision. A saliency map can be generated by calculating a saliency value that quantifies the degree to which a person instantly pays attention to a point or area in an input image, that is, the level of saliency. For example, retinal ganglion cells in the retina of the eye have a region called a receptive field, and when this receptive field is stimulated by light, the information is transmitted to the brain. The receptive field consists of two parts, a central circular part and a peripheral area. Using such a mechanism in the receptive field, a model that quantifies the point where the signal becomes strong due to the stimulation of the circular area in the center and the surrounding area (place that attracts attention) is used as a saliency map. can be done. Specifically, a method of generating a saliency map by creating a pyramid image from an input image and extracting features using a Gaussian filter is known. Also known is a method of generating a saliency map by analyzing how the output of a convolutional neural network changes with respect to input perturbations.

The gradient intensity map generation unit 13 generates a gradient intensity map of the input image from the input image. The gradient strength map shows the spatial distribution of the gradient strength values (ie, pixel intensity differences) for each pixel in the input image. For example, the x direction and the y direction are two directions orthogonal to each other, and the gradient strength value in the x direction of a certain pixel is Ex, and the gradient strength value in the y direction is Ey. is calculated as the square root of the sum of squares of A gradient intensity map shows the spatial distribution of the edges of the input image.

The importance map generator 14 generates an importance map of the input image from the saliency map and the gradient intensity map. The importance map shows the spatial distribution of importance values for each pixel of the input image. The importance map generating unit 14 generates an importance map by combining a saliency map indicating the spatial distribution of salience of the input image and a gradient intensity map indicating the spatial distribution of edges of the input image. The importance map generated in this way shows the high salience portion of the input image and the edge portion of the input image relative to other portions (low salience portions or non-edge portions). The spatial distribution of the importance value of each pixel is shown as a portion of high visual importance.

For example, the input image is represented as a matrix I of l rows and w columns of pixel values, and the saliency map of the input image is represented as a matrix S of l rows and w columns of saliency values. Let the gradient magnitude map be represented as a matrix G of l rows and w columns of gradient magnitude values, and let the importance map of the input image be represented as a matrix IM of l rows and w columns of importance values. An element I(i,j) at the i-th row and j-th column of the matrix I indicates the value of the i-th row and j-th column pixel. An element S(i,j) at the i-th row and j-th column of the matrix S indicates the saliency value of the i-th row and j-th column pixel. An element G(i,j) at the i-th row and j-th column of the matrix G indicates the gradient intensity value of the i-th row and j-th column pixel. An element IM(i,j) at the i-th row and j-th column of the matrix IM indicates the importance value of the i-th row and j-th column pixel. However, 1≤i≤l and 1≤j≤w.

The importance map generation unit 14 may calculate IM(i, j) by, for example, one of the following formulas (1) to (3).

IM(i, j)={S(i, j)+G(i, j)} ⁿ /2 ⁿ (1)

IM(i,j)={S(i,j)} ⁿ (when S(i,j)>G(i,j))
IM(i,j)={G(i,j)} ⁿ (when G(i,j)>S(i,j)) (2)

IM(i,j)=α{S(i,j)} ⁿ¹ +(1−α){G(i,j)} ⁿ² (3)

However, the relationships 0<n<1, 0<n1<1, 0<n2<1, and 0<α<1 are satisfied.

The image compression unit 15 compresses the input image using the importance map. The image compression unit 15 includes a division unit 151 , a calculation unit 152 , a sampling unit 153 and an encoding unit 154 .

A dividing unit 151 divides an input image into a plurality of blocks. Here, a block is a coding unit containing a plurality of pixels, and is called a coding tree unit (CTU).

Calculation unit 152 calculates a value obtained by dividing the sum of importance map values (importance values) corresponding to a plurality of pixels in a block among the plurality of blocks by the number of pixels in the block, for example, 0. to 1 (normalized value) is calculated as the sampling rate of the block. That is, the sampling rate of a certain block is calculated as a value obtained by normalizing the average value of the respective importance values of the plurality of pixels within that block. The calculation unit 152 executes processing for calculating the sampling rate for each block.

The sampling unit 153 samples each block at the sampling rate calculated for each block. Since the sampling rate can vary according to the normalized average value of the importance values calculated for each block, the sampling rate can be adaptively changed for each block according to the importance of each block. . As a result, efficient image compression can be achieved without degrading image quality.

The encoder 154 encodes each sampled block. As an encoding method for each block, for example, the Huffman encoding method can be used.

The communication unit 16 transmits the information encoded by the encoding unit 154 to the outside through the communication network. The communication network may be a wireless network, a wired network, or a mixed network of wireless and wired networks.

FIG. 3 is an explanatory diagram showing the hardware configuration of the image decoding device 20 according to the embodiment of the present invention. The image decoding device 20 includes a processor 201 , a memory 202 , a storage device 203 , an image display device 204 and a communication device 205 . The storage device 203 stores software resources such as an operating system 206 and an image processing program 207 . These software resources are loaded into memory 202 and executed by processor 201 .

FIG. 4 is a functional block diagram of the image decoding device 20 according to the embodiment of the invention. Hardware resources such as the processor 201, the memory 202, the storage device 203, the image display device 204, and the communication device 205 cooperate with software resources such as the operating system 206 and the image processing program 207 to enable the communication unit 21, the decoding Functions of the unit 22, the image restoration unit 23, the image output unit 24, and the image quality evaluation unit 25 are realized.

The communication unit 21 receives encoded information transmitted from the image encoding device 10 through a communication network.

The decoding unit 22 decodes each encoded block. As a decoding method for each block, for example, a Huffman decoding method can be used.

The image restoration unit 23 generates a restored image of the input image from each decoded block by an image interpolation method. Examples of image interpolation methods include (1) a method based on tensor reconstruction, (2) a method based on a low-dimensional multi-body model, (3) a method based on rank minimization of a structured matrix, and (4) singular value decomposition. (5) a method based on deep learning, and (6) a method based on discrete Fourier transform.

References to methods based on tensor reconstruction include, for example, (A) Zhao Q, Zhang L, Cichocki A. Bayesian CP factorization of incomplete tensors with automatic rank determination. IEEE transactions on pattern analysis and machine intelligence, 2015, 37(9 ): 1751-1763, and (B) Chen Y L, Hsu C T, Liao H Y M. Simultaneous tensor decomposition and completion using factor priors. IEEE transactions on pattern analysis and machine intelligence, 2013, 36(3): 577-591, etc. be.

For example, Yokota T, Hontani H, Zhao Q, et al. Manifold Modeling in Embedded Space: An Interpretable Alternative to Deep Image Prior. IEEE Transactions on Neural Networks and Learning Systems, 2020, etc.

Takahashi T, Konishi K, Furukawa T. Structured matrix rank minimization approach to image inpainting, IEEE 55th International Midwest Symposium on Circuits and Systems (MWSCAS). , 2012: 860-863.

For example, Song L, Du B, Zhang L, et al. Nonlocal patch based T-SVD for image inpainting: Algorithm and error analysis, Proceedings of the AAAI Conference on Artificial Intelligence. 2018, 32(1), etc.

References to deep learning-based methods include, for example, Ulyanov D, Vedaldi A, Lempitsky V. Deep image prior, Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 9446-9454.

References to methods based on discrete Fourier transform include, for example, Sridevi G, Kumar S S. Image inpainting based on fractional-order nonlinear diffusion for image reconstruction. Circuits, Systems, and Signal Processing, 2019, 38(8): 3802 -3817 and so on.

The image output unit 24 video-displays the restored image. The restored image to be displayed may be a moving image or a still image.

The image quality evaluation unit 25 calculates the image quality evaluation value of the restored image with respect to the input image using the value of the importance map as a weighting factor.

When MSE (Mean Square Error) and PSNR (Peak Signal-to-Noise Ratio) are used as image quality evaluation values, the image quality evaluation unit 25 calculates MSE and PSNR using equations (4) and (5). do.

In equation (4), IMWMSE(i,j) indicates the MSE of the pixel at the i-th row and the j-th column of the input image. O(i, j) indicates the pixel value of the i-th row and j-th column of the restored image. W(i, j) indicates a weighting factor used for image quality evaluation of the i-th row and j-th column pixel of the input image. The image quality evaluation unit 25 sets W(i, j)=IM(i, j) and calculates IMWMSE(i, j) from equation (1).

Note that the image evaluation of the restored image does not necessarily have to be performed during restoration of all input images, and may be performed when restoring a specific input image that requires image evaluation. When image evaluation of the restored image is performed by the image quality evaluation unit 25 , the communication unit 16 of the image encoding device 10 transmits information of the importance map of the input image to the communication unit 21 of the image decoding device 20 . The communication unit 21 transfers the received importance map information to the image quality evaluation unit 25 . The image quality evaluation unit 25 that has received the importance map information from the communication unit 21 can calculate IMWMSE(i, j) from equation (1) with W(i, j)=IM(i, j). can.

In equation (5), IMWPSNR(i, j) indicates the PSNR of the pixel at the i-th row and the j-th column of the input image. MAX(I·W) ² indicates the maximum possible pixel value. For example, MAX(I·W) ² is 255 when the pixel value is represented by 8 bits. Image quality evaluation unit 25 calculates IMWPSNR(i, j) from equation (2).

When using an SSIM (Structural similarity index) as an image quality evaluation value, the image quality evaluation unit 25 calculates the SSIM using equations (6) to (8).

In equation (8), μIMI indicates the average pixel value of _IMI (i, j) calculated by equation (6). μ _IMO indicates the average pixel value of IMO (i, j) calculated by equation (7). δIMI indicates the standard deviation of _IMI (i,j) calculated by equation (6). δ _IMO indicates the standard deviation of IMO (i, j) calculated by equation (7). _δIMIIMO denotes the covariance between IMI(i,j) and IMO(i,j). c1 and c2 each represent a constant. Image quality evaluation unit 25 calculates IMWPSNR(i, j) from equations (6) to (8).

The communication unit 21 transmits the image quality evaluation value calculated by the image quality evaluation unit 25 to the image encoding device 10 as feedback information. Upon receiving the image quality evaluation value through the communication unit 16 , the image encoding device 10 transfers the received image quality evaluation value to the importance map generation unit 14 . The importance map generator 14 can regenerate the importance map so as to improve the image quality evaluation value. For example, the importance map generation unit 14 uses parameters used for generating the importance map (for example, n, n1, n2, and α, etc.) may be updated. In this way, by calculating the image quality evaluation value of the restored image with respect to the input image using the value of the importance map as a weighting factor, it is possible to perform appropriate image quality evaluation according to the importance of the image.

5 shows an example of an input image input to the image input unit 11. FIG. FIG. 6 shows an example of a saliency map generated by the saliency map generator 12. As shown in FIG. FIG. 7 shows an example of a gradient intensity map generated by the gradient intensity map generator 13. FIG. FIG. 8 shows an example of an input image sampled by the sampling unit 153. As shown in FIG. FIG. 9 shows an example of an input image decoded by the decoding unit 22. As shown in FIG. FIG. 10 shows an example of an input image (restored image) restored by the image restoration unit 23. As shown in FIG.

According to the embodiment of the present invention, image compression using an importance map can achieve efficient image compression without degrading image quality. In particular, the sampling rate can be adaptively changed for each block according to the importance of each block of the input image, and efficient image compression can be realized without degrading the image quality. Further, by calculating the image quality evaluation value of the restored image with respect to the input image using the value of the importance map as a weighting factor, it is possible to perform appropriate image quality evaluation according to the importance of the image.

According to the embodiments of the present invention, since image compression can be performed efficiently, moving images can be transmitted with low delay. For example, it is suitable for moving image transmission for telemedicine systems, remote control systems for robots, teleconferencing systems, or virtual reality systems.

In addition, the embodiment described above is intended to facilitate understanding of the present invention, and is not intended to limit and interpret the present invention. The present invention may be modified/improved without departing from its spirit, and the present invention also includes equivalents thereof. In other words, any design modifications made by those skilled in the art to the embodiments are also included in the scope of the present invention as long as they have the features of the present invention. In addition, each element provided in the embodiment can be combined as long as it is technically possible, and the combination thereof is also included in the scope of the present invention as long as it includes the features of the present invention.

REFERENCE SIGNS LIST 10 image encoding device 11 image input unit 12 saliency map generation unit 13 gradient intensity map generation unit 14 importance map generation unit 15 image compression unit 16 communication unit 20 image decoding device 21 communication Section 22 Decoding section 23 Image restoration section 24 Image output section 25 Image quality evaluation section 151 Division section 152 Calculation section 153 Sampling section 154 Encoding section

Claims (6)

a saliency map generator that generates a saliency map of the input image from the input image;
a gradient intensity map generator that generates a gradient intensity map of the input image from the input image;
an importance map generator that generates an importance map of the input image from the saliency map and the gradient intensity map;
an image compression unit that compresses the input image using the importance map;
An image encoding device comprising: The image encoding device according to claim 1,
The image compression unit
a dividing unit that divides the input image into a plurality of blocks;
For each block, a process of calculating the sampling rate of the block by normalizing the value obtained by dividing the sum of the values of the importance map corresponding to each of the plurality of pixels in the block by the number of pixels in the block. a calculation unit that
a sampling unit that samples each block at a sampling rate calculated for each block;
an encoder that encodes each sampled block;
An image encoding device comprising: An image decoding device for decoding each block of the input image encoded by the image encoding device according to claim 2,
a decoding unit that decodes each encoded block;
an image restoration unit that generates a restored image of the input image from each decoded block by an image interpolation method;
an image quality evaluation unit that calculates an image quality evaluation value of the restored image with respect to the input image using the value of the importance map as a weighting factor;
An image decoding device. generating from an input image a saliency map of said input image;
generating a gradient intensity map of the input image from the input image;
generating an importance map of the input image from the saliency map and the gradient strength map;
compressing the input image using the importance map;
An image encoding method, comprising: The image encoding method according to claim 4,
Compressing the input image comprises:
dividing the input image into a plurality of blocks;
For each block, a process of calculating the sampling rate of the block by normalizing the value obtained by dividing the sum of the values of the importance map corresponding to each of the plurality of pixels in the block by the number of pixels in the block. and
sampling each block at a sampling rate calculated for each block;
encoding each sampled block;
An image encoding method, comprising: An image decoding method for decoding each block of the input image encoded by the image encoding method according to claim 5,
decoding each encoded block;
generating a restored image of the input image from each decoded block by an image interpolation method;
calculating an image quality evaluation value of the restored image with respect to the input image using the value of the importance map as a weighting factor;
An image decoding method comprising:

Images