JP7002213B2 - Spatial / gradation super-resolution device and program - Google Patents

Description

The present invention relates to a spatial / gradation super-resolution device and a program that generate a spatial / gradation super-resolution image by super-resolution processing an original image in the spatial direction and the gradation direction.

Conventionally, there is known a spatial super-resolution technique in which an image is subjected to spatial super-resolution processing in the spatial direction to generate an image having a higher resolution than the original image. Learning type and reconstruction type are known as spatial super-resolution technology. The learning type is a method in which a set of a low frequency component and a high frequency component is held as a database, and the high frequency component is set as a super-resolution component by matching the original image and the low frequency component. The reconstruction type is a method of generating a super-resolution component by linear, non-linear filtering, or registration between a plurality of frames (see, for example, Non-Patent Document 1).

Further, there is known a gradation super-resolution device that performs gradation super-resolution processing on an image in the gradation direction to generate an image having a larger number of gradations than the original image. For example, under the assumption that adjacent pixel values are continuous, an intermediate gradation value can be generated by a Gaussian filter or the like (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2014-003370

Okutomi, Tanaka, Takeshima, Matsumoto, "Latest Trends in Image Super-Resolution Processing Technology", No.93 (8), p.693-698, Aug. 2010

In the conventional learning type spatial super-resolution, a huge database is required to obtain a high-quality spatial super-resolution image. Further, in the conventional reconstruction type spatial super-resolution, linear and non-linear filter processing does not always obtain a high-quality super-resolution image, and resisting between multiple frames has high alignment accuracy. Although high-quality super-resolution images can be obtained, it is generally difficult to obtain high alignment accuracy due to the influence of occlusion and noise.

Further, in the conventional gradation super-resolution technique, there is a problem that edges and textures are blurred and that they are vulnerable to noise and the like.

An object of the present invention made in view of such circumstances is to provide a space / gradation super-resolution device and a program capable of generating a high-quality space / gradation super-resolution image.

In order to solve the above problems, the spatial gradation super-resolution device according to the present invention super-resolutions the original image in the spatial direction and the gradation direction to generate a spatial / gradation super-resolution image. It is a toning super-resolution device, and a gradation super-resolution image is generated by multiplying the gradation value of each pixel of the original image by an integral number according to the number of gradations of the space / gradation super-resolution image. A gradation super-resolution image generation unit, a frequency decomposition unit that performs frequency decomposition processing on the gradation super-resolution image to generate a frequency-resolved image, the gradation super-resolution image, and the frequency-resolved image. An alignment unit that generates alignment information indicating the positional relationship of similar blocks with the low-frequency component image of the above, and the frequency as the high-frequency component of the spatial / gradation super-resolution image according to the alignment information. The super-resolution high-resolution segmentation section that allocates the high-resolution component image of the decomposed image to generate the super-resolution high-frequency component image, and the super-resolution high-frequency component to which the gradation super-resolution image is used as the low-frequency component and the allocation is made. It is characterized by including a frequency reconstruction unit that generates the space / gradation super-resolution image by frequency reconstruction processing using the component image as a high frequency component.
Further, in the spatial gradation super-resolution apparatus according to the present invention, the super-resolution high-frequency super-resolution image having the same size as the gradation super-resolution image and having an initial value of 0, the super-resolution horizontal component image. A super-resolution high-frequency component image consisting of a super-resolution vertical component image and a super-resolution diagonal component image is generated, and according to the alignment information, the super-resolution high-frequency component image is combined with the high-resolution component image of the frequency-resolved image. It is characterized by allocating.

Further, in the spatial gradation super-resolution device according to the present invention, the alignment unit establishes a similar block positional relationship between the gradation super-resolution image and the low-frequency component image of the frequency-resolved image. The super-resolution high-frequency segmentation unit is characterized by performing interpolation using a point spread function and setting the half-value width of the point spread function for each layer of the low-frequency component image.

Further, in the spatial gradation super-resolution device according to the present invention, the frequency decomposition unit has a linear phase property, and a wavelet decomposition process using a wavelet having a tap length of a threshold value or more is performed, and the frequency reconstruction unit is , The wavelet is reconstructed using the wavelet.

Further, in order to solve the above-mentioned problems, the program according to the present invention is characterized in that the computer functions as the above-mentioned spatial gradation super-resolution device.

According to the present invention, it is possible to generate a high-quality spatial / gradation super-resolution image.

It is a block diagram which shows the structural example of the spatial gradation super-resolution apparatus which concerns on one Embodiment of this invention. It is a figure which shows an example of the gradation value of the gradation super-resolution image in the spatial gradation super-resolution apparatus which concerns on one Embodiment of this invention. It is a figure which shows the outline of the processing of the alignment part in the spatial gradation super-resolution apparatus which concerns on one Embodiment of this invention. It is a figure which shows the outline of the processing of the super-resolution high-frequency generation division attachment part in the spatial gradation super-resolution apparatus which concerns on one Embodiment of this invention. It is a figure which shows the outline of the processing of the frequency reconstruction part in the spatial gradation super-resolution apparatus which concerns on one Embodiment of this invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 shows a configuration example of a spatial / gradation super-resolution device according to an embodiment of the present invention. The space / gradation super-resolution device 1 shown in FIG. 1 includes a gradation super-resolution image generation unit 11, a frequency decomposition unit 12, an alignment unit 13, a super-resolution high-frequency super-resolution unit 14, and frequency regeneration. A component 15 is provided.

The space / gradation super-resolution device 1 inputs an original image and performs space / gradation super-resolution processing to generate a space / gradation super-resolution image. In the present specification, "spatial / gradation super-resolution processing" means super-resolution processing of the original image in the spatial direction and the gradation direction, and "spatial / gradation super-resolution image" is used. , Spatial / gradation super-resolution processed image.

The gradation super-resolution image generation unit 11 inputs the original image, multiplies the gradation value of each pixel of the original image by L according to the number of gradations of the space / gradation super-resolution image, and super-gradations. A resolution image is generated and output to the frequency decomposition unit 12, the alignment unit 13, and the frequency reconstruction unit 15. Assuming that the number of gradations of the original image is a and the number of gradations of the space / gradation super-resolution image is b, L = 2 ^(ba) . For example, in the case of super-resolution of 8-bit gradation to 12-bit gradation, the gradation value is multiplied by 16.

FIG. 2 is a diagram showing an example of gradation values of a gradation super-resolution image. The left side of FIG. 2 shows the gradation value of the original image, and the right side shows the gradation value of the gradation super-resolution image. Since the gradation value of the gradation super-resolution image is multiplied by an integer with respect to the original image, the gradation value becomes a discrete value.

The frequency decomposition unit 12 generates a frequency decomposition image by performing frequency decomposition (multi-resolution decomposition) processing of a plurality of layers on the gradation super-resolution image generated by the gradation super-resolution image generation unit 11. The frequency-resolved images include a low-frequency component image LL ⁿ of a gradation super-resolution image, a horizontal high-frequency component image LH ⁿ which is a high-frequency component image of a gradation super-resolution image, a vertical high-frequency component image HL ⁿ , and a diagonal high frequency. It consists of a component image HH ⁿ . The subscript n means the decomposition order. For example, when the original image is frequency-resolved to the third order, a frequency-resolved image of n = 1, 2, and 3 is generated. The frequency decomposition unit 12 outputs the low frequency component image LL ⁿ to the alignment unit 13, and outputs the high frequency component images LH ⁿ , HL ⁿ , and HH ⁿ to the super-resolution high frequency division attachment unit 14.

The frequency decomposition unit 12 uses a wavelet (for example, CDF9 / 7) having a linear phase property and having a tap length equal to or larger than the threshold value in consideration of the discontinuity of the gradation value of the gradation super-resolution image. It is preferable to perform wavelet decomposition treatment.

The alignment unit 13 is between the gradation super-resolution image generated by the gradation super-resolution image generation unit 11 and the low-frequency component image LL ⁿ of the frequency-resolved images generated by the frequency decomposition unit 12. Alignment information (registration information) indicating the positional relationship of similar blocks is generated in, and is output to the super-resolution high-frequency segmentation section 14.

FIG. 3 is a diagram showing an outline of the alignment process in the alignment unit 13. The alignment unit 13 performs block matching between both frames using, for example, a gradation super-resolution image as a reference frame and a low-frequency component image LL ⁿ (n = 1 in FIG. 3) as a reference frame, and within the search range. Generates alignment information indicating the positional relationship of blocks with the highest degree of similarity (correlation). Block matching is performed by a known method using an evaluation function such as an absolute sum of errors (SAD) and a sum of squared errors (SSD). Further, the block matching is performed with a decimal pixel accuracy by, for example, interpolation processing using the parabolic fitting function shown in the equation (1). If the evaluation function value of SAD or SSD exceeds the threshold value, it may not be adopted as the alignment information.

Here, when the pixel position at the search position is x, the SSD (x) represents the SSD value at the pixel position, and more specifically, the SSD (0) is the center position (the position where the SSD value is minimized). SSD (-1) is the SSD value at the position of the adjacent pixel in the −x direction or −y direction from the center position, and SSD (1) is the SSD value at the position of the adjacent pixel in the + x direction or + y direction from the center position. Represents. From the equation (1), the pixel positions (decimal pixel positions) with the decimal pixel accuracy in the horizontal or vertical direction can be calculated respectively.

The super-resolution high-frequency segmentation section 14 first sets initial values in order to estimate a high-frequency component image (super-resolution high-frequency component image) of a spatial / gradation super-resolution image that exceeds the Nyquist frequency of the original image. .. The super-resolution high-resolution component image is a super-resolution horizontal component image which is a horizontal high-frequency component of a space / gradation super-resolution image and a super-resolution vertical component image which is a vertical high-frequency component of a space / gradation super-resolution image. It consists of a super-resolution diagonal component image, which is a diagonal high-frequency component of the space / gradation super-resolution image. For example, the super-resolution horizontal component image, super-resolution vertical component image, and super-resolution diagonal component image are each set to the same size as the gradation super-resolution image (that is, the same size as the original image), and all pixels are set as initial values. The value of is 0.

Next, the super-resolution high-frequency segmentation section 14 uses the high-frequency component image LH ⁿ generated by the frequency-resolving section 12 as the high-frequency component of the super-resolution high-frequency component image according to the alignment information generated by the alignment section 13. , HL ⁿ , HH ⁿ are assigned to generate a super-resolution high-frequency component image, and output to the frequency reconstruction unit 15.

FIG. 4 is a diagram showing an outline of the processing of the super-resolution high-frequency segmentation section 14. The super-resolution high-frequency segmentation section 14 converts the high-frequency component images LH ⁿ , HL ⁿ , and HH ⁿ into super-resolution horizontal component images, super-resolution vertical component images, and super-resolution diagonal component images according to the alignment information. Allocate to. Here, when allocating the high frequency component images LH ⁿ , HL ⁿ , and HH ⁿ , it is assumed that the alignment information of the same phase position in the low frequency component image LL ⁿ is followed. This means that if the block P in the gradation super-resolution image is similar to the block Q in the low-frequency component image LL ⁿ , the unknown super-resolution horizontal component image, super-resolution vertical component image, and super-resolution This is because the block having the same phase position as the block P in the diagonal component image is likely to be similar to the block having the same phase position as the block Q in the high frequency components LH ⁿ , HL ⁿ , and HH ⁿ .

Further, when the positional relationship of similar blocks is obtained with the fractional pixel accuracy in the alignment unit 13, the super-resolution high-frequency division with division 14 after allocation in order to align the fractional pixel position with the normal pixel position. Super-resolution horizontal component image, super-resolution vertical component image, and super-resolution diagonal component image are interpolated using a point spread function (PSF) that simulates the resolution deterioration process of the optical system. I do. Equation (2) shows the point spread function. Here, w is the half width (variance value) of the Gaussian function. Then, the variance of the point spread function is controlled for each nth order. Specifically, it is assumed that the larger the rank n, the larger the resolution deterioration (blurring), and the half width w is set to a large value. For example, w = 1 when n = 1, w = 2 when n = 2, and w = 3 when n = 3.

When there are multiple candidates for horizontal, vertical, and diagonal super-resolution components, the super-resolution high-frequency segmentation section 14 averages those values or reconstructs the maximum a posteriori (MAP). To estimate an unknown value. For more information on MAP reconstruction, see, for example, E. Levitan and G. Herman: “A maximum a posteriori probability expectation maximization algorithm for image reconstruction in emission tomography”, IEEE Transactions on Medical Imaging, vol. 6, no. 3, pp. . 185-192, Sep. 1987. Further, as another method, the super-resolution high-frequency component image may be estimated by the ML method or the weighting according to the distance of the allocated pixels.

The frequency reconstruction unit 15 uses the gradation super-resolution image generated by the gradation super-resolution image generation unit 11 as a low-frequency component, and the super-resolution high-frequency component image allocated by the super-resolution high-frequency generation division attachment unit 14. Is used as a high-frequency component, and frequency reconstruction processing is performed to generate a spatial / gradation super-resolution image and output to the outside. When the frequency decomposition unit 11 performs the wavelet decomposition processing as the frequency decomposition processing, the frequency reconstruction unit 15 performs the wavelet reconstruction processing using the same wavelet.

FIG. 5 is a diagram showing an outline of processing of the frequency reconstruction unit 15. In the example of FIG. 5, by performing the first-order wavelet reconstruction processing, a spatial / gradation super-resolution image in which the number of pixels in the horizontal direction is doubled and the number of pixels in the vertical direction is doubled with respect to the original image is generated. It shows how it is done.

A computer can be suitably used to function as the above-mentioned space / gradation super-resolution device 1, and such a computer has processing contents for realizing each function of the space / gradation super-resolution device 1. It can be realized by storing the program describing the above in the storage unit of the computer and reading and executing this program by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

The program may also be recorded on a computer-readable medium. It can be installed on a computer using a computer-readable medium. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

As described above, the present invention first generates a gradation super-resolution image by multiplying the gradation value of each pixel of the original image by an integral number according to the number of gradations of the space / gradation super-resolution image. A frequency-resolved image is generated by performing frequency-resolved processing on the gradation super-resolution image. Next, alignment information indicating the positional relationship of similar blocks between the gradation super-resolution image and the low-frequency component image of the frequency-resolved image is generated, and the spatial / gradation super is used using the alignment information. A super-resolution high-frequency component image is generated by allocating a high-frequency component image of a frequency-resolved image as a high-frequency component of the resolution image. Finally, the gradation super-resolution image is used as a low-frequency component, and the super-resolution high-frequency component image is used as a high-frequency component, and frequency reconstruction processing is performed to generate a spatial / gradation super-resolution image. This makes it possible to generate a highly accurate and highly accurate spatial / gradation super-resolution image in which the spatial resolution and the number of gradations of the original image are super-resolution processed.

In addition, in order to obtain the alignment with the fractional pixel accuracy and align the alignment result with the fractional pixel accuracy to the pixel position, a point spread function simulating the resolution deterioration process of the optical system is applied to the horizontal, vertical, and diagonal high frequency components. , This point spread function may be set for each layer of the low frequency component image. In particular, in the conventional method for super-resolution of the number of gradations, the number of gradations is interpolated using a Gaussian filter or the like on the basis that adjacent pixel values are easily continuous. On the other hand, in the present invention, since the decimal pixel accuracy alignment of similar objects in the same frame and the allocation using the point spread function according to the frequency decomposition order are performed, the spatial resolution and the spatial resolution and the accuracy are higher than before. The number of gradations can be interpolated at the same time.

Although the above embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and modifications can be made without departing from the scope of claims. For example, it is possible to combine a plurality of the constituent blocks described in the configuration diagram of the embodiment into one, or to divide one constituent block into one.

1 Spatial / gradation super-resolution device 11 Gradation super-resolution image generation unit 12 Frequency decomposition unit 13 Alignment unit 14 Super-resolution high-frequency super-resolution attachment unit 15 Frequency reconstruction unit

Claims (5)

It is a spatial / gradation super-resolution device that generates a spatial / gradation super-resolution image by super-resolution processing the original image in the spatial direction and the gradation direction.
A gradation super-resolution image generation unit that generates a gradation super-resolution image by multiplying the gradation value of each pixel of the original image by an integer according to the number of gradations of the space / gradation super-resolution image. ,
A frequency decomposition unit that performs frequency decomposition processing on the gradation super-resolution image to generate a frequency decomposition image,
An alignment unit that generates alignment information indicating the positional relationship of similar blocks between the gradation super-resolution image and the low-frequency component image of the frequency-resolved image.
According to the alignment information, the super-resolution high-frequency component image is assigned as the high-frequency component of the spatial / gradation super-resolution image, and the super-resolution high-frequency component image is generated.
The gradation super-resolution image is used as a low-frequency component, and the assigned super-resolution high-frequency component image is used as a high-frequency component for frequency reconstruction processing to generate the spatial / gradation super-resolution image. Department and
A spatial / gradation super-resolution device characterized by being equipped with. The super-resolution high-frequency segmentation section has a super-resolution horizontal component image, a super-resolution vertical component image, and a super-resolution diagonal component image having the same size as the gradation super-resolution image and an initial value of 0. The space according to claim 1, wherein a super-resolution high-frequency component image comprising the above is generated, and a high-frequency component image of the frequency-resolved image is assigned to the super-resolution high-frequency component image according to the alignment information. -Gradation super-resolution device. The alignment unit obtains the positional relationship of similar blocks between the gradation super-resolution image and the low-frequency component image of the frequency-resolved image with decimal pixel accuracy.
2 . The described space / gradation super-resolution device. The frequency decomposition unit has linear phase, and performs wavelet decomposition processing using a wavelet having a tap length equal to or larger than a threshold value.
The spatial / gradation super-resolution device according to any one of claims 1 to 3, wherein the frequency reconstruction unit performs wavelet reconstruction using the wavelet. A program for causing a computer to function as the spatial / gradation super-resolution device according to any one of claims 1 to 4 .

Images