JP6363878B2 - Super-resolution image optimization apparatus and program - Google Patents

Description

  The present invention relates to a super-resolution image optimization apparatus and program for determining an optimum super-resolution image from a plurality of super-resolution images.

  When a super-resolution image is estimated from the original image, the correct resolution of the super-resolution image is unknown, so there is no strict method for optimizing the parameters of the super-resolution image. A method is known in which a super-resolution image is generated by super-resolution processing, and parameters are selected so as to minimize the difference between the image reduced by using a point spread function and the original image (Patent Document 1). reference).

JP 2007-193508 A

  Patent Document 1 describes a technique for optimizing a super-resolution parameter (super-resolution parameter) using a back projection method. However, the conventional technique for optimizing the super-resolution parameter is effective only when super-resolution processing is performed using a filter with a clear inverse function and phase characteristics, such as super-resolution using a point spread function. . Therefore, there is a problem that the conventional super-resolution parameter optimization method cannot be used as it is for the registration super-resolution in which the super-resolution processing is performed by registration (registration).

  An object of the present invention made in view of such circumstances is to determine an optimum super-resolution image having the smallest difference value from the original image from a plurality of super-resolution images regardless of the super-resolution processing method. It is an object of the present invention to provide a possible super-resolution image optimization apparatus and program.

  In order to solve the above-described problem, a super-resolution image optimization device according to the present invention is a super-resolution image optimization device that determines an optimal super-resolution image from a plurality of super-resolution images, and includes N pieces of super-resolution images. Each of the super-resolution images is subjected to low-pass filter processing, and a first LPF processing unit 11 that generates N first low-frequency super-resolution images, and reduction processing is performed for each of the first low-frequency super-resolution images. A first reduction processing unit 12 for generating M reduced images having different phases and outputting N sets of first reduced images in pairs, and super-resolution processing for each of the first reduced images. Then, a super-resolution processing unit 13 that generates N sets of first super-resolution images paired with M sheets, and low-pass filter processing is performed for each of the first super-resolution images, and M sheets are paired. A second LPF processing unit 14 that generates N sets of second low-frequency super-resolution images and an original image for each second low-frequency super-resolution image A second reduction processing unit 15 that generates a reduced image by performing reduction processing so as to have the same phase and generates N sets of second reduced images that are M pairs, and a second that is a pair of the M sheets. A difference value from the original image is calculated for each set of reduced images, and a super-resolution image corresponding to the second set of reduced images that minimizes the difference value is output as an optimal super-resolution image. And an optimization unit 16.

  Furthermore, in the super-resolution image optimizing device according to the present invention, the first LPF processing unit and the second LPF processing unit are configured to perform a point check when parameters used for generating the plurality of super-resolution images are unknown. A low-pass filter process is performed using a spread function or a Gaussian function.

  Furthermore, in the super-resolution image optimizing device according to the present invention, the super-resolution processing unit is configured to perform a point spread function or a Gaussian function when parameters used for generating the plurality of super-resolution images are unknown. A super-resolution process is performed using

  Further, in the super-resolution image optimization device according to the present invention, the optimization unit calculates a difference value between the original image and a set of the second reduced image paired with the M sheets, An optimum super-resolution image is output for each region.

  In order to solve the above problems, a program according to the present invention causes a computer to function as the super-resolution image optimization device.

  According to the present invention, it is possible to determine an optimum super-resolution image having the smallest difference value from the original image from a plurality of super-resolution images regardless of the super-resolution processing method.

It is a figure explaining the process outline | summary of the super-resolution image optimization apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structural example of the super-resolution image optimization apparatus which concerns on one Embodiment of this invention. It is a figure explaining the process of the 1st reduction process part in the super-resolution image optimization apparatus which concerns on one Embodiment of this invention. It is a figure explaining the process of the 2nd reduction process part in the super-resolution image optimization apparatus which concerns on one Embodiment of this invention. It is a figure explaining the 1st example of the process of the optimization part in the super-resolution image optimization apparatus which concerns on one Embodiment of this invention. It is a figure explaining the 2nd example of the process of the optimization part in the super-resolution image optimization apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structural example of the super-resolution apparatus which inputs a super-resolution image into the super-resolution image optimization apparatus which concerns on one Embodiment of this invention. It is a figure explaining the process of the frequency decomposition part in the super-resolution apparatus shown in FIG. It is a figure explaining the process of the registration part in the super-resolution apparatus shown in FIG. It is a figure explaining the process of the super-resolution high frequency component production | generation part in the super-resolution apparatus shown in FIG. It is a figure explaining the process of the frequency reconstruction part in the super-resolution apparatus shown in FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The super-resolution image optimization device according to the present invention inputs an original image and a plurality of super-resolution images obtained by super-resolution processing of the original image, and an error from the original image among the plurality of super-resolution images. Determine and output the super-resolution image with the least amount of. If the parameters used for the super-resolution processing (hereinafter referred to as “super-resolution parameters”) are known, the super-resolution parameters are also input.

  FIG. 1 is a diagram for explaining the processing outline of a super-resolution image optimization apparatus according to an embodiment of the present invention. FIG. 1 illustrates an example in which the super-resolution image is an image obtained by super-resolution processing of the original image twice in the horizontal direction and the vertical direction.

  First, the super-resolution image optimization device performs low-pass filter processing on the super-resolution image group to prevent aliasing during reduction (step S11), and then performs reduction processing (step S12).

  That is, in step S11, a plurality of (N) super-resolution images are low-pass filtered, respectively, to generate N first low-frequency super-resolution images. The original image can be thought of as having been generated by reducing and degrading an image with a higher resolution with a camera optical system or the like. Therefore, even if the super-resolution image is low-pass filtered using a filter that simulates the degradation process. Good.

  In step S12, for each first low-frequency super-resolution image, reduction processing is performed to generate reduced images having M different phases, and N sets of M pairs, that is, M × N first images. Output a reduced image of. In FIG. 1, M = 4, and four first reduced images a, b, c, and d are generated from one first low-frequency super-resolution image O ′.

  In order to compare the original image and the reduced image, it is desirable that the phases are the same, but the first reduced image includes an image having a phase different from that of the original image. Therefore, the first reduced image is further subjected to super-resolution processing (step S13), subjected to low-pass filter processing (step S14), and then subjected to reduction processing (step S15) again, so that the first phase of the same phase as the original image is obtained. 2 reduced images are generated.

  That is, in step S13, super-resolution processing is performed for each first reduced image, and N sets of first super-resolution images with M pairs are generated. FIG. 1 shows how the first super-resolution images A, B, C, and D are generated from the first reduced images a, b, c, and d, respectively.

  In step S14, low-pass filter processing is performed for each first super-resolution image to generate N sets of second low-frequency super-resolution images with M pairs. FIG. 1 shows how the second low-frequency super-resolution images A ′, B ′, C ′, and D ′ are generated from the first super-resolution images A, B, C, and D, respectively.

  In step S15, each second low-frequency super-resolution image is reduced so as to have the same phase as the original image, and N sets of second reduced images having M pairs are generated. FIG. 1 shows how the second reduced images a ′, b ′, c ′, d ′ are generated from the second low-frequency super-resolution images A ′, B ′, C ′, D ′, respectively. Yes.

  Finally, in step S16, a difference value with respect to the original image is obtained for each pair of the second reduced images paired with M, and the super-resolution corresponding to the second reduced image group having the smallest difference value. The super-resolution image used to generate the image, that is, the second reduced image set having the smallest difference value is output as the optimum super-resolution image (optimum super-resolution image).

  FIG. 2 is a block diagram illustrating a configuration example of a super-resolution image optimization apparatus according to an embodiment of the present invention. The super-resolution device 2 generates a plurality of super-resolution images from the original image.

  The super-resolution image optimization apparatus 1 inputs a plurality of super-resolution images and determines an optimal super-resolution image. Note that the super-resolution parameter is input to the super-resolution image optimization apparatus 1 only when it is known. In the example illustrated in FIG. 2, the super-resolution image optimization apparatus 1 includes a first LPF processing unit 11, a first reduction processing unit 12, a super-resolution processing unit 13, a second LPF processing unit 14, and a second reduction. A processing unit 15 and an optimization unit 16 are provided.

  The first LPF processing unit 11 performs low-pass filter processing on each of the N super-resolution images input from the super-resolution device 2, and performs first reduction processing on the N first low-frequency super-resolution images. To the unit 12. When the super-resolution device 2 generates a super-resolution image by super-resolution processing of the original image using wavelet transform, the first LPF processing unit 11 selects the wavelet used for the super-resolution processing. It is preferable to perform low pass filter processing. For example, wavelet decomposition is performed on the super-resolution image without decimation using the wavelet, and the low-frequency component is set as the first low-frequency super-resolution image.

  Further, the first LPF processing unit 11 reduces the super-resolution image to the original image when the parameters used to generate the super-resolution image group input to the super-resolution image optimization apparatus 1 are unknown. Low-pass filter processing is performed using a degradation function that simulates the degradation process as degradation of the camera optical system. For example, a point spread function (PSF) or a Gaussian function is used as the deterioration function, and the dispersion value is variable as a parameter. Note that the low-pass filter processing may be performed using the degradation function even when the parameters used for generating the super-resolution image group input to the super-resolution image optimization apparatus 1 are known.

  The first reduction processing unit 12 performs reduction processing on each of the N first low-frequency super-resolution images generated by the first LPF processing unit 11 by 1 / M times, and performs reduction with M different phases. An image is generated, and N sets of first reduced images having M pairs are output to the super-resolution processing unit 13.

  FIG. 3 is a diagram for explaining the processing of the first reduction processing unit 12. Here, an example is shown in which the first low-frequency super-resolution image O ′ is reduced by a factor of 1/4 to the original image size. In this case, the first reduction processing unit 12 performs pixel thinning by shifting the phase by one pixel in the horizontal direction and the vertical direction, and performs four first reductions for one low-frequency super-resolution image O ′. Images a, b, c, and d are obtained. When the pixel positions of the original image o in the first low-frequency super-resolution image O ′ are a1, a2, a3, a4,..., The first reduced image a is the first low-frequency super-resolution image O. Although the phase is the same as', the first reduced images b, c, and d are different in phase from the original image o.

  The super-resolution processing unit 13 performs super-resolution processing on each of the first reduced images generated by the first reduction processing unit 12 to generate N sets of first super-resolution images in pairs of M sheets. And output to the second LPF processing unit 14. The super-resolution processing unit 13 performs super-resolution processing using the same parameters when the parameters used to generate the super-resolution image group input to the super-resolution image optimization apparatus 1 are known. . If the parameters used to generate the super-resolution image group are unknown, super-resolution processing is performed using a degradation function such as a point spread function or a Gaussian function that simulates the reduction degradation process. Note that the super-resolution processing may be performed using the degradation function even when the parameters used for generating the super-resolution image group are known.

  The second LPF processing unit 14 performs low-pass filter processing on the first super-resolution image generated by the super-resolution processing unit 13 in the same manner as the first LPF processing unit 11, and N sets of M pairs. A second low-frequency super-resolution image is generated and output to the second reduction processing unit 15.

  The second reduction processing unit 15 reduces the second low-frequency super-resolution image generated by the second LPF processing unit 14 to 1 / M times so as to have the same phase as the original image, and sets M images. N sets of second reduced images are generated and output to the optimization unit 16.

  FIG. 4 is a diagram for explaining the processing of the second reduction processing unit 15. Here, an example is shown in which the second low-frequency super-resolution image is reduced by a factor of 1/4 to the original image size. In this case, the second reduction processing unit 15 performs pixel thinning so that the phase is the same as that of the original image, and the four second low-frequency super-resolution images A ′ to D ′. Reduced images a ′ to d ′ are obtained. When the pixel positions of the original image o in the second low-frequency super-resolution images A ′ to D ′ are h1, h2, h3, h4,..., The second reduced images a ′ to d ′ are the original image o. And the same phase.

  The optimization unit 16 calculates a difference value between the original image o and the second reduced image for each pair of the second reduced images having a pair of M sheets, and the second reduced image that minimizes the difference value. A super-resolution image corresponding to the set is output as an optimal super-resolution image.

FIG. 5 is a diagram illustrating a first example of processing by the optimization unit 16. For example, a second reduced image a _n from N sheets of the super-resolution image _{O n ', b n',} c n ', d n' and is generated. The subscript n is a number for identifying the super-resolution image, and n = 1, 2,. In this case, the optimization unit 16, a difference value between the n-th set of original images o and the second reduced image, the original image o the second reduced image _{a n ', b n',} c n ', The minimum value of each difference value from d _n ′ or the average value of the difference values. Here, the difference value can be evaluated using, for example, PSNR (Peak Signal to Noise Ratio) or SSIM (Structural Similarity). Then, the optimization unit 16, for each set of the second reduced image by calculating a difference value between the original image o, the second super-resolution image O _n corresponding to the set of reduced image difference value is minimized Is output to the outside as an optimal super-resolution image. In FIG. 5, when n = k, the difference value between the set {a _k ′, b _k ′, c _k ′, d _k ′} of the original image o and the second reduced image is minimized, and super-resolution is performed. An example in which the image _Ok is an optimal super-resolution image is shown.

  FIG. 6 is a diagram illustrating a second example of the process of the optimization unit 16. The optimization unit 16 divides the original image into a plurality of regions, calculates a difference value between each set of the original image o and the second reduced image for each region, and a second difference value that minimizes the difference value for each region. The optimum super-resolution image corresponding to the set of reduced images may be output. In FIG. 6, the original image o is divided into nine regions, and in the upper right region, the difference value between the original image o and the second reduced image set is minimum when n = 1, In the lower area, the difference value between the original image o and the second reduced image set is minimized when n = 2, and in the lower left area, the set of the original image o and the second reduced image is set. In this example, the difference value between and is minimum when n = N. In this way, when determining the optimum super-resolution image for each region, the optimization unit 16 uses the identification information of the optimum super-resolution image determined for each region as the optimum super-resolution parameter, so that the optimum super-resolution image is determined. At the same time, it may be output to the outside.

(Super-resolution device)
Next, the super-resolution device 2 will be described. In the present invention, the super-resolution processing method is not limited, but an example is shown below.

  FIG. 7 is a block diagram illustrating a configuration example of the super-resolution device 2. In the example illustrated in FIG. 7, the super resolving device 2 includes a frequency resolving unit 21, a registration unit 22, a super resolving high frequency component generating unit 23, and a frequency reconfiguring unit 24.

  The frequency decomposition unit 21 generates a first frequency decomposition image by performing frequency decomposition on the original image using the first frequency decomposition parameter, for example, first-order wavelet decomposition without decimation, The high frequency component is output to the super-resolution high frequency component generation unit 23.

  Further, the frequency resolving unit 21 performs frequency decomposition using the second frequency decomposition parameter, for example, first-order wavelet decomposition with decimation to generate a second frequency decomposed image, and generates the second frequency decomposed image. The low frequency component is output to the registration unit 22, and the high frequency component of the second frequency resolved image is output to the super resolution high frequency component generation unit 23. Here, the first frequency decomposition parameter and the second frequency decomposition parameter are parameters for defining a filter used for frequency decomposition, such as a wavelet filter. By changing the value of the frequency resolution parameter, the frequency characteristics during frequency resolution can be made variable.

FIG. 8 is a diagram for explaining the processing of the frequency resolving unit 21. FIG. 8A shows a state in which the first frequency division image is generated by performing first-order frequency decomposition without decimation as an example of frequency decomposition of the original image using the first frequency decomposition parameter. Yes. Since there is no decimation, the image size of each frequency component (low frequency component LL ⁰ , horizontal high frequency component LH ⁰ , vertical high frequency component HL ⁰ , oblique high frequency component HH ⁰ ) is equal to the original image size. The frequency resolving unit 21 outputs LH ⁰ , HL ⁰ , and HH ⁰ to the super-resolution high-frequency component generation unit 23 as high-frequency component initial values.

FIG. 8B shows a state in which the original image is frequency-decomposed using the second frequency decomposition parameter and the original image is subjected to first-order frequency decomposition with decimation to generate a second frequency division image. Yes. Since it is a first-order decomposition with decimation, the image size of each frequency component (low frequency component LL ¹ , horizontal high frequency component LH ¹ , vertical high frequency component HL ¹ , oblique high frequency component HH ¹ ) is vertical and horizontal from the original image size. Reduced to 1/2. The frequency resolving unit 21 outputs LL ¹ as a low frequency component to the registration unit 22 and outputs LH ¹ , HL ¹ , and HH ¹ as high frequency components to the super-resolution high frequency component generation unit 23.

  The registration unit 22 performs block matching between the original image and the low frequency component of the second frequency decomposition image generated by the frequency decomposition unit 21 using the registration parameter, and each of the low frequency components Registration information indicating the positional relationship corresponding to the block is output to the super-resolution high-frequency component generation unit 23.

FIG. 9 is a diagram for explaining the processing of the registration unit 22. The registration unit 22 divides the low-frequency component LL ¹ of the second frequency-resolved image into blocks of a predetermined size, for example, 4 pixels × 4 pixels, and performs block matching with the original image O to perform registration. Output information. Block matching is performed by a known method using an evaluation function such as an absolute value error sum (SAD) or a square error sum (SSD). In addition, block matching is performed with decimal pixel accuracy by, for example, interpolation processing using a parabolic fitting function.

  As shown in FIG. 9, the registration unit 22 preferably performs registration (positioning) between a low-frequency component of an image obtained by frequency-decomposing the original image with decimation and the original image. The reason is that, when images having similar shapes and different sizes exist in the same frame, the similar images can be aligned and their high frequency components can be assigned. Note that there is a high possibility that similar images exist in the same frame when viewed locally, so that the accuracy of alignment increases as the block size when performing block matching to the size of the original image is reduced. it can.

  The registration parameter includes a block size and a search range when performing block matching. The search range may be the entire original image in order to improve accuracy, but the search range may be limited to a range near the target block in order to improve processing speed. Further, when a threshold value is included in the registration parameter and the evaluation function value of SAD or SSD exceeds the threshold value, it may not be adopted as the registration result.

  The super-resolution high-frequency component generation unit 23 includes the high-frequency component of the first frequency-resolved image and the high-frequency component of the second frequency-resolved image generated by the frequency decomposition unit 21, and the registration information generated by the registration unit 22. To generate a super-resolution high-frequency component of the original image and output it to the frequency reconstruction unit 24.

FIG. 10 is a diagram for explaining the processing of the super-resolution high-frequency component generation unit 23. The super-resolution high-frequency component generation unit 23 converts the high-frequency components (LH ¹ , HL ¹ , HH ¹ ) of the second frequency-resolved image and the high-frequency components (LH ⁰ , HL) of the first frequency-resolved image according to the registration information. ⁰ , HH ⁰ ). Here, when allocating the high frequency components (LH ¹ , HL ¹ , HH ¹ ) of the second frequency resolved image, follow the registration information at the same phase position as the low frequency component LL ¹ of the second frequency resolved image. And If the block P having the low frequency component (LL ¹ ) of the second frequency resolved image corresponds to (similar to) the block Q having the original image, each high frequency component (LH ¹ , HL ¹ , HH ^1). This is because the block in the same phase position as the block P) is likely to correspond to the block Q in the same manner. In other words, if the low-frequency components of the original image and the frequency-decomposed image with the decimation are the same, the high-frequency component exceeding the sampling frequency of the original image is estimated to be almost the same as the high-frequency component of the frequency-decomposed image with the decimation Because it can.

Then, after the allocation, the super-resolution high-frequency component generation unit 23 performs reconstruction processing, for example, reconstruction processing based on MAP (Maximum A Posterior) estimation, and performs super-resolution high-frequency components (LH ⁰ ′, HL ⁰ ′, HH ⁰ ′) is generated. The reconstruction process based on MAP estimation is a process of estimating and reconstructing a high-resolution image that maximizes the posterior probability when a low-resolution image is used as a condition. The second frequency resolved image to be allocated is estimated from the high resolution image as the initial value of the image, and the high resolution image is updated so as to minimize the error between the estimated value and the actual value. As another method, a super-resolution high-frequency component may be generated by performing reconstruction processing by ML (Maximum Likelihood) estimation or weighting according to the allocated pixel distance.

  The frequency reconstruction unit 24 reconstructs the frequency of the super-resolution high-frequency component generated by the original image and the super-resolution high-frequency component generation unit 23 using the frequency reconstruction parameter, and generates a super-resolution image. Output to. Here, the frequency reconstruction parameter is a parameter for defining a filter used for frequency reconstruction.

FIG. 11 is a diagram for explaining the processing of the frequency reconfiguration unit 24. The frequency reconstruction unit 24 uses the original image O as a spatial low frequency component, and generates a super-resolution high-frequency component (horizontal high-frequency component LH ⁰ ′, vertical high-frequency component HL ⁰ ′, diagonal) generated by the super-resolution high-frequency component generation unit 23. The super-resolution image _OSR is generated by performing frequency reconstruction, for example, wavelet reconstruction, using the high-frequency component HH ⁰ ′) as the spatial high-frequency component.

  As described above, the super-resolution image optimization device 1 according to the present invention performs low-pass filter processing by the first LPF processing unit 11 and reduction processing by the first reduction processing unit 12 on the input super-resolution image group. To generate a first reduced image group. Next, after super-resolution processing is performed on the first reduced image group by the super-resolution processing unit 13, low-pass filter processing by the second LPF processing unit 14 and reduction processing by the second reduction processing unit 15 are performed. Then, a second reduced image group having the same phase as the original image is generated. Finally, the optimization unit 16 outputs the super-resolution image that minimizes the difference value between the original image and the second reduced image group as an optimized super-resolution image. With such a configuration, the super-resolution image optimization apparatus 1 has the smallest difference value from the super-resolution image group to the original image regardless of the super-resolution processing method of the input super-resolution image group. An optimum super-resolution image can be determined.

  Note that a computer can be suitably used to function as the super-resolution image optimization device 1 described above, and such a computer describes the processing contents for realizing each function of the super-resolution image optimization device 1. This program can be realized by storing the program in a storage unit of the computer, and reading and executing the program by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

  Although the above embodiment has been described as a representative example, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments can be combined into one, or one constituent block can be divided.

DESCRIPTION OF SYMBOLS 1 Super-resolution image optimization apparatus 2 Super-resolution apparatus 11 1st LPF process part 12 1st reduction process part 13 Super-resolution process part 14 2nd LPF process part 15 2nd reduction process part 16 Optimization part

Claims (5)

A super-resolution image optimization device for determining an optimal super-resolution image from a plurality of super-resolution images,
A first LPF processing unit for low-pass filtering each of the N super-resolution images to generate N first low-frequency super-resolution images;
For each of the first low-frequency super-resolution images, a reduction process is performed to generate reduced images having M different phases, and N sets of first reduced images having M pairs are output. And
A super-resolution processing unit that performs super-resolution processing for each of the first reduced images and generates N sets of first super-resolution images in pairs of M sheets;
A second LPF processing unit that performs low-pass filtering for each of the first super-resolution images to generate N sets of second low-frequency super-resolution images paired with M images;
For each of the second low-frequency super-resolution images, a reduced image is generated by performing a reduction process so as to have the same phase as the original image, and N sets of second reduced images having M pairs are generated. 2 reduction processing unit;
A difference value with respect to the original image is calculated for each pair of the second reduced images paired with the M sheets, and a super-resolution image corresponding to the second reduced image group having the minimum difference value is obtained. An optimization unit that outputs an optimal super-resolution image;
A super-resolution image optimizing device comprising:   The first LPF processing unit and the second LPF processing unit perform low-pass filter processing using a point spread function or a Gaussian function when parameters used to generate the plurality of super-resolution images are unknown. The super-resolution image optimizing device according to claim 1, wherein the super-resolution image optimizing device according to claim 1.   The super-resolution processing unit performs a super-resolution process using a point spread function or a Gaussian function when parameters used for generating the plurality of super-resolution images are unknown. The super-resolution image optimization apparatus according to claim 1.   The optimization unit calculates a difference value between the original image and a set of the second reduced image paired with the M sheets, and outputs an optimum super-resolution image for each area. The super-resolution image optimizing device according to any one of claims 1 to 3, wherein the super-resolution image optimizing device is characterized.   The program for functioning a computer as a super-resolution image optimization apparatus as described in any one of Claim 1 to 4.

Images