JP2022129735A - Image encoding device, image encoding method, image decoding device, and image decoding method - Google Patents

Abstract

To provide an image encoding device capable of compressing an input image without degrading its image quality.SOLUTION: An image encoding device 10 includes: a saliency map generation unit 12 that generates a saliency map of an input image from the input image; a gradient intensity map generation unit 13 that generates a gradient intensity map of the input image from the input image; an importance map generation unit 14 that generates an importance map of the input image from the saliency map and the gradient intensity map; a binarization unit 15 that binarizes the importance map into a first gradation value and a second gradation value lighter than the first gradation value by dithering; a color information extraction unit 16 that extracts, from the input image, image information including position information and color information on each pixel having a first grayscale value in the importance map from among a plurality of pixels of the input image; and an encoding unit 17 that encodes the pixel information.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image encoding device, an image encoding method, an image decoding device, and an image decoding method.

MPEG standard (H.265/HEVC) is known as a standard by ISO/IEC (International Organization for Standardization/International Electrotechnical Commission) for compression encoding technology of moving images. HEVC defines encoding methods for 4K (3840×2160 pixels) images and 8K (7680×4320 pixels) images.

In the future, it is expected that the need for high-definition video playback such as 8K images will increase, and the development of technology that compresses large-capacity images without degrading quality and transmits them with low delay is desired. .

Accordingly, an object of the present invention is to propose an image coding apparatus and an image coding method capable of compressing an input image without degrading its image quality.

In order to solve the above-described problems, the image coding apparatus according to the present invention includes a saliency map generation unit that generates a saliency map of an input image from an input image, and a gradient map that generates a gradient intensity map of the input image from the input image. an intensity map generator for generating an importance map of the input image from the saliency map and the gradient intensity map; a binarization unit that binarizes the image into a second grayscale value that is lighter than the value; position information and color of each pixel having the first grayscale value in the importance map among the plurality of pixels of the input image A color information extraction unit extracting pixel information including information from an input image, and an encoding unit encoding the pixel information. Instead of encoding the input image itself, the important information of the input image ( For example, the image can be compressed efficiently while maintaining the information of the highly conspicuous part of the human visual sense.

An image decoding apparatus according to the present invention is an image decoding apparatus that restores an input image from pixel information encoded by the image encoding apparatus according to the present invention. and an image restoration unit for restoring an input image by an image interpolation method from the decoded pixel information. Since the pixel information holds the information necessary to restore the input image with sufficient accuracy for practical use, the restored image can be generated from the pixel information by image interpolation without degrading the image quality. .

As an image interpolation method, for example, an image interpolation method based on a low-dimensional manifold model can be used. According to this image interpolation method, an input image can be accurately restored from a small amount of information (pixel information that does not include position information or color information of pixels whose value in the importance map has the second grayscale value). .

An image coding method according to the present invention comprises the steps of generating a saliency map of the input image from the input image; generating a gradient intensity map of the input image from the input image; binarizing the importance map into a first gray value and a second gray value lighter than the first gray value by dithering; extracting from the input image pixel information including position information and color information for each pixel having a first gray value in the importance map among the plurality of pixels in the input image; and encoding the pixel information. . Instead of encoding the input image itself, the important information of the input image ( For example, the image can be compressed efficiently while maintaining the information of the highly conspicuous part of the human visual sense.

An image decoding method according to the present invention is an image decoding method for restoring an input image from pixel information encoded by the image encoding method according to the present invention, the step of decoding the encoded pixel information. and reconstructing the input image from the decoded pixel information by an image interpolation method. Since the pixel information holds the information necessary to restore the input image with sufficient accuracy for practical use, the restored image can be generated from the pixel information by image interpolation without degrading the image quality. .

As an image interpolation method, for example, an image interpolation method based on a low-dimensional manifold model can be used. According to this image interpolation method, an input image can be accurately restored from a small amount of information (pixel information that does not include position information or color information of pixels whose value in the importance map has the second grayscale value). .

According to the present invention, an input image can be efficiently compressed without degrading its image quality.

1 is an explanatory diagram showing the hardware configuration of an image encoding device according to an embodiment of the present invention; FIG. 1 is a functional block diagram of an image encoding device according to an embodiment of the present invention; FIG. 1 is an explanatory diagram showing the hardware configuration of an image decoding device according to an embodiment of the present invention; FIG. 1 is a functional block diagram of an image decoding device according to an embodiment of the present invention; FIG. It is a figure which shows an example of the input image input into the image input part in connection with embodiment of this invention. It is a figure which shows an example of the saliency map produced|generated by the saliency map production|generation part in connection with embodiment of this invention. It is a figure which shows an example of the gradient intensity|strength map produced|generated by the gradient intensity|strength map production|generation part in connection with embodiment of this invention. It is a figure which shows an example of the importance map produced|generated by the importance map production|generation part in connection with embodiment of this invention. It is a figure which shows an example of the importance map binarized by the binarization part in connection with embodiment of this invention. It is a figure which shows an example of the pixel information decoded by the decoding part in connection with embodiment of this invention. It is a figure which shows an example of the input image restored by the image restoration part in connection with embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. Here, the same reference numerals denote the same components, and duplicate descriptions are omitted.

FIG. 1 is an explanatory diagram showing the hardware configuration of an image encoding device 10 according to an embodiment of the invention. The image encoding device 10 includes a processor 101 , memory 102 , storage device 103 , camera 104 and communication device 105 . The storage device 103 stores software resources such as an operating system 106 and an image processing program 107 . These software resources are loaded into memory 102 and executed by processor 101 .

FIG. 2 is a functional block diagram of the image encoding device 10 according to the embodiment of the invention. Hardware resources such as the processor 101, the memory 102, the storage device 103, the camera 104, and the communication device 105 cooperate with software resources such as the operating system 106 and the image processing program 107 to generate the image input unit 11 and the saliency map. The functions of the generation unit 12, the gradient intensity map generation unit 13, the importance map generation unit 14, the binarization unit 15, the color information extraction unit 16, the encoding unit 17, and the communication unit 18 are realized.

The image input unit 11 takes in input images that form a moving image captured by the camera 104 .

The saliency map generator 12 generates a saliency map of the input image from the input image. A saliency map indicates the saliency of each region in the input image, that is, the spatial distribution of salience in human vision. A saliency map can be generated by calculating a saliency value that quantifies the degree to which a person instantly pays attention to a point or area in an input image, that is, the level of saliency. For example, retinal ganglion cells in the retina of the eye have a region called a receptive field, and when this receptive field is stimulated by light, the information is transmitted to the brain. The receptive field consists of two parts, a central circular part and a peripheral area. Using such a mechanism in the receptive field, a model is used as a saliency map that quantifies the points where the signal becomes stronger due to the stimulation of the circular area in the center and the surrounding areas (places that attract attention). can be done. Specifically, a method of generating a saliency map by creating a pyramid image from an input image and extracting features using a Gaussian filter is known. Also known is a method of generating a saliency map by analyzing how the output of a convolutional neural network changes with respect to input perturbations.

The gradient intensity map generation unit 13 generates a gradient intensity map of the input image from the input image. The gradient strength map shows the spatial distribution of the gradient strength values (ie, pixel intensity differences) for each pixel in the input image. For example, the x direction and the y direction are two directions orthogonal to each other, and the gradient strength value in the x direction of a certain pixel is Ex, and the gradient strength value in the y direction is Ey. is calculated as the square root of the sum of squares of A gradient intensity map shows the spatial distribution of the edges of the input image.

The importance map generator 14 generates an importance map of the input image from the saliency map and the gradient intensity map. The importance map shows the spatial distribution of importance values for each pixel of the input image. The importance map generating unit 14 generates an importance map by combining a saliency map indicating the spatial distribution of salience of the input image and a gradient intensity map indicating the spatial distribution of edges of the input image. The importance map generated in this way shows the high salience portion of the input image and the edge portion of the input image relative to other portions (low salience portions or non-edge portions). The spatial distribution of the importance value of each pixel is shown as a portion of high visual importance.

For example, the input image is represented as a matrix I of l rows and w columns of pixel values, and the saliency map of the input image is represented as a matrix S of l rows and w columns of saliency values. Let the gradient magnitude map be represented as a matrix G of l rows and w columns of gradient magnitude values, and let the importance map of the input image be represented as a matrix IM of l rows and w columns of importance values. An element I(i,j) at the i-th row and j-th column of the matrix I indicates the value of the i-th row and j-th column pixel. An element S(i,j) at the i-th row and j-th column of the matrix S indicates the saliency value of the i-th row and j-th column pixel. An element G(i,j) at the i-th row and j-th column of the matrix G indicates the gradient intensity value of the i-th row and j-th column pixel. An element IM(i,j) at the i-th row and j-th column of the matrix IM indicates the importance value of the i-th row and j-th column pixel. However, 1≤i≤l and 1≤j≤w.

The importance map generation unit 14 may calculate IM(i, j) by, for example, one of the following formulas (1) to (3).

IM(i, j)={S(i, j)+G(i, j)} ⁿ /2 ⁿ (1)

IM(i,j)={S(i,j)} ⁿ (when S(i,j)>G(i,j))
IM(i,j)={G(i,j)} ⁿ (when G(i,j)>S(i,j)) (2)

IM(i,j)=α{S(i,j)} ⁿ¹ +(1−α){G(i,j)} ⁿ² (3)

However, the relationships 0<n<1, 0<n1<1, 0<n2<1, and 0<α<1 are satisfied.

Note that the importance map generator 14 may normalize the importance value of each pixel.

The binarization unit 15 binarizes the importance map into a first grayscale value and a second grayscale value lighter than the first grayscale value by dithering. The first gray value is, for example, the gray value "1" indicating "black", and the second gray value is, for example, the gray value "0" indicating "white". In this binarization process, for example, an error diffusion method can be applied in which the error generated by the binarization of each pixel is weighted according to the distance between the pixels and added to the surrounding pixels.

The color information extraction unit 16 extracts pixel information including position information and color information of each pixel having the first grayscale value in the importance map from among the plurality of pixels of the input image. For example, if the position information of a pixel having the first grayscale value in the importance map is (X, Y) and its color information is (R, G, B), then the pixel information is (X, Y, R, G, B). Here, R, G, and B represent color information of red, green, and blue, respectively. For example, if the importance map value of the pixel in the first row and second column of the input image has the first grayscale value and the color information of that pixel is (20, 0, 10), the pixel information is , (001,002,020,000,010). It should be noted that the pixel information does not include the position information of the pixel whose value in the importance map has the second gray value, nor the color information thereof.

The encoder 17 encodes the pixel information output from the color information extractor 16 . For example, the Huffman coding method can be used as the pixel information coding method. Instead of encoding the input image itself, the important information of the input image ( For example, the image can be compressed efficiently while maintaining the information of the highly conspicuous part of the human visual sense.

The communication unit 18 transmits the pixel information encoded by the encoding unit 17 to the outside through a communication network. The communication network may be a wireless network, a wired network, or a mixed network of wireless and wired networks.

FIG. 3 is an explanatory diagram showing the hardware configuration of the image decoding device 20 according to the embodiment of the present invention. The image decoding device 20 includes a processor 201 , a memory 202 , a storage device 203 , an image display device 204 and a communication device 205 . The storage device 203 stores software resources such as an operating system 206 and an image processing program 207 . These software resources are loaded into memory 202 and executed by processor 201 .

FIG. 4 is a functional block diagram of the image decoding device 20 according to the embodiment of the invention. Hardware resources such as the processor 201, the memory 202, the storage device 203, the image display device 204, and the communication device 205 cooperate with software resources such as the operating system 206 and the image processing program 207 to enable the communication unit 21, the decoding Functions of the unit 22, the image restoration unit 23, and the image output unit 24 are realized.

The communication unit 21 receives encoded pixel information transmitted from the image encoding device 10 through a communication network.

The decoding unit 22 decodes the encoded pixel information. A Huffman decoding method, for example, can be used as a method for decoding the encoded pixel information.

The image restoration unit 23 restores the input image from the pixel information decoded by the decoding unit 22 by an image interpolation method. The pixel information includes location information and color information for each pixel whose importance map value has a first gray value, but location information for a pixel whose importance map value has a second gray value also includes its color information. does not include The image restoration unit 23 performs interpolation processing on the assumption that the color information of pixels having the second grayscale value in the importance map is zero. Since the pixel information holds the information necessary to restore the input image with sufficient accuracy for practical use, the restored image can be generated from the pixel information by image interpolation without degrading the image quality. .

As the image interpolation method, for example, (1) a method based on tensor reconstruction, (2) a method based on a low-dimensional multi-body model, (3) a method based on rank minimization of a structured matrix, (4) a singular value (5) deep learning-based methods; and (6) discrete Fourier transform-based methods.

References to methods based on tensor reconstruction include, for example, (A) Zhao Q, Zhang L, Cichocki A. Bayesian CP factorization of incomplete tensors with automatic rank determination. IEEE transactions on pattern analysis and machine intelligence, 2015, 37(9 ): 1751-1763, and (B) Chen Y L, Hsu C T, Liao H Y M. Simultaneous tensor decomposition and completion using factor priors. IEEE transactions on pattern analysis and machine intelligence, 2013, 36(3): 577-591, etc. be.

For example, Yokota T, Hontani H, Zhao Q, et al. Manifold Modeling in Embedded Space: An Interpretable Alternative to Deep Image Prior. IEEE Transactions on Neural Networks and Learning Systems, 2020, etc.

Takahashi T, Konishi K, Furukawa T. Structured matrix rank minimization approach to image inpainting, IEEE 55th International Midwest Symposium on Circuits and Systems (MWSCAS). , 2012: 860-863.

For example, Song L, Du B, Zhang L, et al. Nonlocal patch based T-SVD for image inpainting: Algorithm and error analysis, Proceedings of the AAAI Conference on Artificial Intelligence. 2018, 32(1), etc.

References to deep learning-based methods include, for example, Ulyanov D, Vedaldi A, Lempitsky V. Deep image prior, Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 9446-9454.

References to methods based on discrete Fourier transform include, for example, Sridevi G, Kumar S S. Image inpainting based on fractional-order nonlinear diffusion for image reconstruction. Circuits, Systems, and Signal Processing, 2019, 38(8): 3802 -3817 and so on.

The image output unit 24 video-displays the restored image. The restored image to be displayed may be a moving image or a still image.

5 shows an example of an input image input to the image input unit 11. FIG. FIG. 6 shows an example of a saliency map generated by the saliency map generator 12. As shown in FIG. FIG. 7 shows an example of a gradient intensity map generated by the gradient intensity map generator 13. FIG. FIG. 8 shows an example of an importance map generated by the importance map generation unit 14. As shown in FIG. FIG. 9 shows an example of the importance map binarized by the binarization unit 15 . FIG. 10 shows an example of pixel information decoded by the decoding unit 22. As shown in FIG. FIG. 11 shows an example of an input image (restored image) restored by the image restoration unit 23. As shown in FIG.

According to embodiments of the present invention, rather than encoding the input image itself, the value of the importance map encodes pixel information including position and color information for each pixel having a first gray value. Therefore, image compression can be performed efficiently while maintaining important information of the input image (for example, information of highly noticeable parts visually by humans). In addition, since the pixel information holds the information necessary to restore the input image with practically sufficient accuracy, the restored image can be generated from the pixel information by image interpolation without degrading the image quality. can be done. In addition, by using an image interpolation method based on a low-dimensional manifold model as an image interpolation method, less information (the value of the importance map does not include the position information of the pixel having the second grayscale value and the color information thereof) The input image can be restored with high accuracy from the pixel information).

According to the embodiments of the present invention, since image compression can be performed efficiently, moving images can be transmitted with low delay. For example, it is suitable for moving image transmission for telemedicine systems, remote control systems for robots, teleconferencing systems, or virtual reality systems.

In addition, the embodiment described above is intended to facilitate understanding of the present invention, and is not intended to limit and interpret the present invention. The present invention may be modified/improved without departing from its spirit, and the present invention also includes equivalents thereof. In other words, any design modifications made by those skilled in the art to the embodiments are also included in the scope of the present invention as long as they have the features of the present invention. In addition, each element provided in the embodiment can be combined as long as it is technically possible, and the combination thereof is also included in the scope of the present invention as long as it includes the features of the present invention.

DESCRIPTION OF SYMBOLS 10... Image encoding apparatus 11... Image input part 12... Saliency map generation part 13... Gradient intensity map generation part 14... Importance map generation part 15... Binarization part 16... Color information extraction part 17... Encoding part DESCRIPTION OF SYMBOLS 18... Communication part 20... Image decoding apparatus 21... Communication part 22... Decoding part 23... Image restoration part 24... Image output part

Claims (6)

a saliency map generator that generates a saliency map of the input image from the input image;
a gradient intensity map generator that generates a gradient intensity map of the input image from the input image;
an importance map generator that generates an importance map of the input image from the saliency map and the gradient intensity map;
a binarization unit that binarizes the importance map into a first gradation value and a second gradation value lighter than the first gradation value by dithering;
a color information extraction unit for extracting, from the input image, pixel information including position information and color information of each pixel having the first grayscale value in the importance map among the plurality of pixels of the input image;
an encoding unit that encodes the pixel information;
An image encoding device comprising: An image decoding device for restoring the input image from the pixel information encoded by the image encoding device according to claim 1,
a decoding unit that decodes the encoded pixel information;
an image restoration unit that restores the input image from the decoded pixel information by an image interpolation method;
An image decoding device. The image decoding device according to claim 2,
The image decoding device, wherein the image interpolation method is an image interpolation method based on a low-dimensional manifold model. generating from an input image a saliency map of said input image;
generating a gradient intensity map of the input image from the input image;
generating an importance map of the input image from the saliency map and the gradient strength map;
binarizing the importance map into a first grayscale value and a second grayscale value lighter than the first grayscale value by dithering;
extracting from the input image pixel information including position information and color information of each pixel having the first grayscale value in the importance map among the plurality of pixels of the input image;
encoding the pixel information;
An image encoding method, comprising: An image decoding method for restoring the input image from the pixel information encoded by the image encoding method according to claim 4,
decoding the encoded pixel information;
reconstructing the input image from the decoded pixel information by an image interpolation method;
An image decoding method comprising: The image decoding method according to claim 5,
The image decoding method, wherein the image interpolation method is an image interpolation method based on a low-dimensional manifold model.

Images