JP2020017229A - Image processing apparatus, image processing method and image processing program - Google Patents

Abstract

To provide a combination technique, in which when a plurality of image processing techniques having different properties are combined to use, advantage of each of these techniques can be brought out.SOLUTION: In an image processing apparatus, a first image generation unit generates a first image from an input image using a first image processing technique. A second image generation unit generates a second image from the input image using a second image processing technique. A synthesis ratio control unit synthesizes the first image and the second image to control an image synthesis unit that generates an output image, and a synthesis ratio for each pixel of the first image and the second image. The synthesis ratio control unit determines the synthesis ratio for each pixel of the first image and the second image on the basis of learning.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, an image processing method, and an image processing program.

  Techniques used in the single image super-resolution technique include a filtering-based technique, a learning-based technique, and a case-based technique.

"Enhanced deep residual networks for single image super-resolution" In CVPR Workshops. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Vol. 1, p. 3, 2017. https://arxiv.org/pdf/1707.02921.pdf

  The single-image super-resolution technique is a technique for generating a higher-resolution high-quality image from one low-resolution low-quality image.

  Techniques used in the single image super-resolution technique include filtering-based techniques such as Bicubic Interpolation, learning-based techniques such as Convolutional Neural Networks (CNN), and low learning techniques such as convolutional neural networks (CNN). There is a case-based method for increasing the resolution by retrieving a low-resolution image similar to the input from a database prepared with a number of pairs of a resolution image and a high-resolution image. As described below, the filtering-based method and the learning-based method have significantly different properties.

  The filtering-based method creates an enlarged image having a large number of pixels by performing interpolation or the like based on a predetermined rule for each pixel of the low-quality image. For example, bicubic interpolation, which is one of the filtering-based methods, creates an enlarged image by interpolating surrounding 4 × 4 pixels (16 pixels) with a cubic function for each pixel of a low-quality image. In general, various types of filtering-based methods have been proposed since they can perform high-speed calculations and can estimate a low-frequency region (such as a uniform portion of an image) with high accuracy. On the other hand, the filtering-based method has a drawback that estimation accuracy of a high-frequency region (such as an edge portion of an image) is poor.

  On the other hand, in a learning-based method such as CNN, a large number of pairs of an original image and a low-quality image created by degrading the original image are prepared, and the correspondence between the two is learned by a neural network. The learning-based method generates a high-quality image by applying the correspondence thus learned to a low-quality image. The learning-based method has an advantage that the high-frequency region can be estimated with extremely high accuracy. On the other hand, however, the learning-based method has lower accuracy in the low-frequency region than the filtering base, or even in the high-frequency region, tries to emphasize the high-frequency component excessively, so that ringing (a pattern that did not originally exist around the edge) Phenomena that occur as a secondary phenomenon).

As described above, the filtering-based method and the learning-based method have a complementary relationship with respect to the estimation accuracy in the low frequency region and the high frequency region.
Therefore, a technique of correcting the result obtained by the filtering-based method (specifically, bicubic interpolation) using a learning-based method (specifically, CNN) for the purpose of realizing more accurate estimation. Has been proposed (for example, Non-Patent Document 1).

  The technique (EnhanceNet) described in Non-Patent Document 1 first creates an enlarged image by applying bicubic interpolation to a low-quality image. After that, the difference between the enlarged image and the correct data (ground truth) is learned by the CNN. Then, the difference obtained by this learning is added to the enlarged image created earlier. As a result, it is possible to reduce errors in a high frequency region generated in the process of bicubic interpolation.

  However, EnhanceNet has a problem in that the quality of the enlarged image obtained by the bicubic interpolation in the low frequency region is adversely deteriorated. It is considered that this is because EnhanceNet does not apply the single image super-resolution technique to the combination of the filtering technique and the learning-based technique so as to exhibit the advantages of each technique.

  Thus, the present inventors have found that when using different types of image processing techniques in combination, it is an important issue in this field to combine them in such a way that the advantages of each technique can be exhibited. Recognized.

  The present invention has been made in view of such circumstances, and a purpose thereof is to combine a plurality of image processing methods having different properties so that the advantages of each of these methods are exhibited. Is to provide.

  In order to solve the above problem, an image processing apparatus according to an aspect of the present invention includes a first image generation unit configured to generate a first image from an input image using a first image processing method, A second image generation unit that generates a second image from the input image using an image processing method, an image synthesis unit that synthesizes the first image and the second image to generate an output image, A combination ratio control unit that controls a combination ratio for each pixel of the first image and the second image. The composition ratio control unit dynamically determines a composition ratio for each pixel of the first image and the second image according to the input image based on the learning.

  Another embodiment of the present invention relates to an image processing method. The method includes a first image generating step of generating a first image from an input image using a first image processing technique, and a second image from an input image using a second image processing technique. A second image generating step of generating, an image synthesizing step of synthesizing the first image and the second image to generate an output image, and a synthesizing ratio for each pixel of the first image and the second image And controlling the combination ratio. The combining ratio control step determines a combining ratio for each pixel of the first image and the second image based on the learning.

  Still another preferred embodiment according to the present invention relates to an image processing program. The program includes a first image generation step of generating a first image from an input image using a first image processing method, and a second image from an input image using a second image processing method. A second image generating step of generating, an image synthesizing step of synthesizing the first image and the second image to generate an output image, and a synthesizing ratio for each pixel of the first image and the second image And a combining ratio controlling step of controlling the computer. The combining ratio control step determines a combining ratio for each pixel of the first image and the second image based on the learning.

  It is to be noted that any combination of the above-described components and any conversion of the expression of the present invention between an apparatus, a method, a system, a recording medium, a computer program, and the like are also effective as embodiments of the present invention.

  According to the present invention, when a plurality of image processing methods having different properties are used in combination, it is possible to realize, by learning, combinations in which the advantages of each of these methods are exhibited.

FIG. 2 is a diagram illustrating a configuration of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram illustrating respective images related to the image processing apparatus in FIG. 1. FIG. 1 is a diagram illustrating a configuration of an image processing apparatus according to an embodiment.

(Embodiment 1)
FIG. 1 shows a configuration of an image processing apparatus 10 according to the first embodiment.
The image processing apparatus 10 includes a first image generation unit 100, a second image generation unit 200, an image synthesis unit 300, and a synthesis ratio control unit 400.

  The first image generation unit 100 receives an input image 1000 and generates a first image 1010 from the input image 1000 using a first image processing method. The first image generation unit 100 outputs the generated first image 1010 to the image synthesis unit 300.

The second image generation unit 200 receives the input image 1000 and generates a second image 1020 from the input image 1000 using a second image processing method. The second image generator 200 outputs the generated second image 1020 to the image synthesizer 300.
Typically, the first image processing method and the second image processing method have different properties.

  The image combining unit 300 combines the first image 1010 and the second image 1020 to generate an output image 1030. At this time, the image synthesis unit 300 synthesizes the first image 1010 and the second image 1020 based on the synthesis ratio 4000 for each pixel determined by the synthesis ratio control unit.

  The composition ratio control unit 400 determines a composition ratio 4000 for each pixel of the first image 1010 and the second image 1020 based on learning.

Hereinafter, an example of learning (hereinafter, simply referred to as “learning”) of the synthesis ratio 4000 for each pixel of the first image 1010 and the second image 1020 by the synthesis ratio control unit 400 will be described.
In the learning, a training image (hereinafter, referred to as a “learning original image”) used in the learning is provided. This learning original image corresponds to Ground Truth in machine learning. The learning original image is typically a set of various (typically hundreds of thousands) sample images having a predetermined resolution and size.

  As the input image 1000 for learning, an image created by applying a predetermined conversion to the learning original image (hereinafter, referred to as “learning input image”) is used. That is, in learning, the first image 1010, the second image 1020, and the output image 1030 are generated from the input image 1000 for learning. Typically, the input image for learning is obtained by reducing the size of the original image for learning (for example, reducing the vertical and horizontal sizes to そ れ ぞ れ each).

  At the start of learning, the synthesis ratio 4000 for each pixel of the first image 1010 and the second image 1020 is set to a predetermined initial value.

  First, the first learning input image 1000 is input to the image processing apparatus 10. The image combining unit 300 combines the first image 1010 generated from the learning input image 1000 and the second image 1020 generated from the learning input image 1000 to generate an output image 1030. I do. At this time, the image combining unit 300 combines the first image 1010 and the second image 1020 based on the initial value of the combining ratio 4000 for each pixel preset in the combining ratio control unit 400. The image composition unit 300 outputs the generated output image 1030 to the composition ratio control unit 400.

  The composition ratio control unit 400 compares the output image 1030 with the learning original image corresponding to the first learning input image 1000. This comparison is typically performed by calculating the mean square error between each pixel value of the output image 1030 and the original training image.

  When the comparison on the first learning input image is completed, the second learning input image 1000 is input to the image processing apparatus 10. The combining ratio control unit 400 compares the output image 1030 and the learning original image with respect to the second learning input image 1000 by the same processing as described above.

  Similarly, the output image 1030 and the original learning image are compared for all the input images 1000 for learning.

  When the comparison between the output image 1030 and the learning original image is completed for all the learning input images 1000, the combining ratio control unit 400 optimizes the comparison between the output image 1030 and the learning original image 4000. To determine.

As an example, the composite ratio 4000 is calculated by minimizing the loss function: f _loss defined in Equation 1 for each pixel. Here, I _SR and I _HR are the pixel values of the output image and the learning original image, respectively, (i, j) are the coordinates of the pixel on the two-dimensional plane, and n is the difference between the output image 1030 and the learning original image. Shows the number of comparisons.

The learning is completed by executing all the above processes.
As described above, by performing learning using various learning original images (typically several hundred thousand types), a calculation method of generating a pixel-by-pixel combination ratio 4000 including optimal information on these various images is performed. Is determined. In other words, for various images, it becomes clear at what ratio the first image processing method and the second image processing method should be applied for each pixel.

  By the above-described learning, when the unknown input image 1000 is input to the image processing apparatus 10, the combining ratio control unit 400 determines what kind of combining ratio 4000 according to the type and characteristics of the unknown input image 1000. , It is determined whether to combine the first image 1010 and the second image 1020 for each pixel. Then, when a new input image 1000 is input to the image processing apparatus 10, an output image 1030 is generated using the synthesis ratio 4000 for each pixel determined by this learning.

For example, the (i, j) pixel of the first image 1010 is P ¹ _{(i, j)} , the (i, j) pixel of the second image 1020 is P ² _{(i, j)} , P ¹ _{(i ,} the mixing ratio _M of _{^{j) (i, j),}} P 2 (i, 1-M (i , the mixing ratio of _{j), j)} to the _{(where, 0 ≦ M (i, j} ) ≦ 1), The composition ratio control unit 400 generates an output image 1030 in which the (i, j) -th pixel P ³ _{(i, j)} is represented by Expression (2).
_{^{P 3 (i, j) =}} M (i, j) · P 1 (i, j) + (1-M (i, j)) · P 2 (i, j) ··· (2)

The same can be explained using a matrix expression as follows.
Assuming that the first image 1010 is I ₁ , the second image 1020 is I ₂ , the output image 1030 is I _SR , the composition ratio of I ₁ is M, and the composition ratio of I ₂ is JM, I _SR is represented by the following equation. It can be seen that it can be expressed as (3). Here, I ₁ , I ₂ , I _SR , and M are matrix expressions each having a value for each pixel as a component. J represents a matrix in which all components are 1, and ○ represents a product of each matrix component.
I _SR = M ○ I ₁ + (J−M) ○ I ₂ (3)

FIG. 2 shows images of the image processing apparatus of FIG.
These images include an input image 1000, a first image 1010, a second image 1020, and an output image 1030. In the image 1040, the composition ratio 4000 is set to white for each pixel, a ratio 0 is set to white, a ratio 1 is set to black, and a ratio between 0 and 1 is set to gray of the corresponding density. , Respectively. When the composition ratio 4000 is represented by the image 1040 in this manner, the output image 1030 can be represented as a superimposition of the first image 1010 and the second image 1020 with the image 1040 interposed therebetween. In this sense, the image 1040 expressing the combination ratio 4000 can be interpreted as a “mask” used when the first image 1010 and the second image 1020 are superimposed. This mask realizes a combination that makes use of the advantages of the respective methods in an apparatus that generates an output image using the first image processing method and the second image processing method.

  According to the present embodiment, an apparatus that generates an image using two types of image processing methods having different properties and combines the generated images to generate a new image can exhibit the advantages of each method. A simple synthesis can be realized.

(Embodiment 2)
In one embodiment, the output image may have a higher resolution than the input image.
In the present embodiment, typically, the image processing apparatus 10 is applied to a single image super-resolution technique. In this case, the first and second image processing techniques correspond to the first and second single image super-resolution techniques, respectively.

  The input image 1000, which is a low quality image, is converted into a first image 1010 and a second image 1020 using two single image super-resolution techniques having different properties. The image synthesizing unit 300 generates an output image 1030 by synthesizing the first image 1010 and the second image 1020 at a ratio that best reflects the advantages of the two single image super-resolution techniques.

  According to the present embodiment, in an apparatus that generates a high-quality image by combining two types of single image super-resolution techniques, it is possible to generate a high-quality image so that the advantages of these techniques are best reflected. it can.

(Embodiment 3)
In one embodiment, the first image processing technique may be a filtering based technique and the second image processing technique may be a learning based technique.
The image processing method based on the filtering-based method may be a nearest neighbor interpolation, a bilinear interpolation, a Lanczos interpolation, or the like, in addition to the above-mentioned bicubic interpolation.

  FIG. 2 shows a first image 1010 generated by bicubic interpolation and a second image 1020 generated by CNN. From FIG. 2, it can be seen that the quality of the first image 1010 is improved in the low frequency region and the quality of the second image 1020 is improved in the high frequency region, reflecting the characteristics of each method. By superimposing the first image 1010 and the second image 1020 with an image 1040 expressing a composition ratio in the form of a mask interposed therebetween, an output image 1030 reflecting the advantages of each technique is obtained. Can be

  In the present embodiment, the second image generation unit 200 and the composition ratio control unit 400 are each configured by a neural network. At this time, an error from the Ground Truth of the generated image recognized by comparing the output image 1030 with the learning original image is fed back to the combining ratio control unit 400 and is fed back to the second image generating unit 200. , May be used for improving the performance of the second image generation unit 200. The composition ratio control unit 400 increases the contribution of the second image processing method (learning-based method) to image quality improvement (that is, the “image that is better at the learning-based method”). The synthesis ratio is increased, and the higher the contribution of the first image processing method (the filtering-based method) is, the lower the synthesis ratio of the second image 1020 is (i.e., “the image that is not good at the learning-based method”). Therefore, if the learning result related to the combination ratio is fed back to the second image generation unit 200, the second image generation unit 200 executes the process more aggressively for the image that “is good at itself”, The processing learning can be improved, such as performing the processing more passively on an image that is “my weak point”. As described above, by feeding back the learning result during learning in the combination ratio control unit 400 to the second image generation unit 200, the neural network of the second image generation unit 200 can be improved.

  According to the present embodiment, in a single image super-resolution technique that combines a filtering-based technique and a learning-based technique, the two techniques are designed so that the mutual advantages of the two techniques are exhibited most without being impaired. , And can produce high quality images.

(Embodiment 4)
In one embodiment, when the number of image comparisons for determining the composition ratio 4000 for each pixel of the first image 1010 and the second image 1020 is n, and k is an integer satisfying 1 ≦ k <n In the first to k-th image comparisons, the combining ratio control unit 400 compares the second image 1020 generated from the learning input image 1000 created by converting the learning original image with the learning In the k + 1-th to n-th image comparisons, the output image 1030 generated from the learning input image 1000 created by converting the learning original image, and the learning original image May be compared.

  For example, if the first image processing method is bicubic interpolation and the second image processing method is CNN, the accuracy of CNN is lower than that of bicubic interpolation in the initial stage of learning. , M = J at an early stage, and learning may not proceed any further. Therefore, in the first k image comparisons, instead of improving the image obtained by combining the image obtained by the bicubic interpolation and the image obtained by the CNN interpolation, only the image generated by the CNN is improved so that the learning can proceed correctly. it can.

  According to the present embodiment, the quality of the obtained image can be improved with a higher probability in the single image super-resolution technique combining the filtering-based technique and the learning-based technique.

(Modification)
The present invention has been described based on the embodiments. These embodiments are exemplifications, and it is understood by those skilled in the art that various modifications can be made to the combination of each component and each processing process, and that such modifications are also within the scope of the present invention. is there.

(Modification 1)
When the image generation unit includes a plurality of processing blocks including an intermediate stage processing block, the composition ratio control unit may use information generated in the intermediate stage block for learning.

FIG. 3 shows a configuration of the image processing apparatus 11 according to one embodiment. The difference from the image processing apparatus 10 of FIG. 1 is that the second image generation unit 200 includes a first intermediate processing block 210, a second intermediate processing block 220, and a final processing block 230.

  The second image generation unit 200 processes the input image 1000 that has been input in the first intermediate processing block 210 and the second intermediate processing block 220, and then generates the second image 1020 in the final processing block 230. That is, the second image generation unit 200 sequentially processes the input image 1000 in the order of the first intermediate processing block 210, the second intermediate processing block 220, and the final processing block 230.

The first intermediate processing block 210 generates first intermediate data 5010, and the second intermediate processing block 220 generates second intermediate data 5020. In the learning, the combining ratio control unit 400 uses the first intermediate data 5010 and the second intermediate data 5020 in addition to comparing the output image 1030 with the original learning image.
The other configuration of the image processing apparatus 11 is common to the image processing apparatus 10 and thus the description is omitted.
According to this modification, the image quality of the output image can be further improved.

(Modification 2)
The number of image processing methods may be three or more instead of two.
According to the present modification, the range of selection of the image processing method can be expanded.

(Modification 3)
Instead of inputting only one input image, a plurality of images in different directions may be input, and an output image may be generated for each of the input images. For example, an input image may be a certain image, an image obtained by rotating the image by 90 degrees, 180 degrees, or 270 degrees, and eight images including the left-right inversion. Such a technique is one of known image quality improving techniques called, for example, geometric self-ensembling. By applying this method to the embodiment of the present invention, the image quality can be improved as compared with the case where the input image is only one time.
Thus, it can be seen that the present invention has good compatibility with known image quality improvement methods such as geometric self-ensembling.

(Modification 4)
In the image processing method according to one embodiment of the present invention, the number of steps for generating an image is not limited to two, and may be three or more. That is, a first image generation step of generating a first image from an input image using the first image processing method in the above-described embodiment, and a second image generation step of generating a second image from the input image using the second image processing method. In addition to the second image generation step of generating the image of the above, an additional image generation step may be included. At this time, in the further image generation step, an image may be generated using the first image processing method or the second image processing method, or an image may be generated using another image processing method. Good.
According to this modification, the image quality of the output image can be further improved by increasing the number of image generation steps.

  The present invention can be applied to various image processing such as color tone correction, image noise removal, image blur removal, and image defect repair, in addition to the above-described improvement of image resolution. Further, the method of the present invention can be applied to various data processing such as voice and text in addition to image processing, and has high industrial applicability.

  DESCRIPTION OF SYMBOLS 10 Image processing apparatus, 100 1st image generation part, 200 2nd image generation part, 300 image synthesis part, 400 synthesis ratio control part, 1000 input images, 1010 1st image, 1020 2nd image, 1030 output Image, 4000 composition ratio

Claims (7)

A first image generation unit configured to generate a first image from an input image using a first image processing method;
A second image generation unit that generates a second image from the input image using a second image processing method;
An image synthesizing unit that synthesizes the first image and the second image to generate an output image;
A combination ratio control unit that controls a combination ratio for each pixel of the first image and the second image,
The image processing device, wherein the combination ratio control unit determines a combination ratio for each pixel of the first image and the second image based on learning. The composition ratio control unit is configured to convert the first image for learning based on learning based on a comparison between an output image generated from an input image for learning generated by converting the original image for learning and the original image for learning. The image processing apparatus according to claim 1, wherein a composition ratio of each pixel of the image and the second image is determined. The image processing apparatus according to claim 1, wherein the output image has a higher resolution than the input image. The first image processing method is a filtering-based method,
The image processing apparatus according to claim 3, wherein the second image processing method is a learning-based method. When the number of image comparisons for determining the composition ratio of each pixel of the first image and the second image is n, and k is an integer satisfying 1 ≦ k <n,
The combination ratio control unit includes:
In the first to k-th image comparisons, image comparison is performed based on a comparison between a second image generated from a learning input image created by converting the learning original image and the learning original image. Run
In the k + 1-th to n-th image comparisons, image comparison is performed based on a comparison between an output image generated from a learning input image created by converting the learning original image and the learning original image. The image processing apparatus according to claim 4, wherein the image processing is performed. A first image generation step of generating a first image from an input image using a first image processing technique;
A second image generation step of generating a second image from the input image using a second image processing technique;
An image combining step of combining the first image and the second image to generate an output image;
A combination ratio control step of controlling a combination ratio for each pixel of the first image and the second image,
The image processing method, wherein the combining ratio control step determines a combining ratio for each pixel of the first image and the second image based on learning. A first image generation step of generating a first image from an input image using a first image processing technique;
A second image generation step of generating a second image from the input image using a second image processing technique;
An image combining step of combining the first image and the second image to generate an output image;
Causing a computer to execute a combination ratio control step of controlling a combination ratio of each pixel of the first image and the second image.
The image processing program, wherein the combining ratio control step determines a combining ratio for each pixel of the first image and the second image based on learning.