JP2018081461A - Super resolution apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve image quality of a super resolution image by considering a color sampling structure.SOLUTION: A super resolution apparatus 1 includes: a frequency decomposition section 11 for frequency decomposing a plurality of hierarchical levels with respect to an original image to generate a low frequency component image group and a high frequency component image group; a frequency component selection section 12 for selecting a high frequency component image having a Nyquist frequency equal to or lower than a Nyquist frequency of an imaging device in the high frequency component image group according to a color sampling structure; a registration section 13 for performing block matching between the original image and a low frequency component image of the same hierarchical level as the selected high frequency component image to generate registration information; a super resolution high frequency component generation section 14 for allocating the high frequency component image selected according to the registration information as a high frequency component exceeding the Nyquist frequency of the original image to generate a super resolution high frequency component image; and a frequency reconstruction section 15 for performing frequency reconstruction using the original image as a low frequency component and the super resolution high frequency component image as a high frequency component to generate a super resolution image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a super-resolution apparatus and program for super-resolution processing of a single frame original image.

  As a conventional technique for performing super-resolution processing using an original image (super-resolution image) of a single frame, for example, searching for similar locations in a single frame and performing alignment, and high frequency of the aligned block A super-resolution device that can generate a high-definition super-resolution image regardless of the correlation between frames by allocating the component as a super-resolution high-frequency component exceeding the Nyquist frequency of the original image is known ( For example, see Patent Document 1).

  Further, depending on the color sampling structure of the image sensor, the sampling frequency of the green signal of the image may be higher than that of the red signal and the blue signal. Therefore, when the similarity of the green signal is high when searching for a similar portion in a single frame, the high-frequency component of the red signal and the blue signal is super-resolved high-frequency component using the alignment information of the green signal. Is known (see, for example, Non-Patent Document 1).

JP2015-203952A

・ Y. Matsuo and S. Sakaida, "A Super-resolution Method Using Registration of Multi-scale Components on the Basis of Color-sampling Patterns of UHDTV Cameras", Proceedings of IEEE ICCE, p.435-436, Jan. 2016

  However, the prior art described in the prior art document does not select the frequency component (frequency band) of the image used for alignment in consideration of the color sampling structure of the original image. Therefore, depending on the color signal, a frequency component including a aliasing component may be used for alignment and allocation. Then, there is a problem that the image quality of the super-resolution image is deteriorated due to a decrease in alignment accuracy or a blur component or a aliasing component included when assigned to the super-resolution high-frequency component. .

  An object of the present invention made in view of such circumstances is to determine a frequency component to be used for alignment and allocation in consideration of a color sampling structure, and to improve the super-resolution image quality. To provide an apparatus and a program.

  In order to solve the above-described problem, a super-resolution apparatus according to the present invention is a super-resolution apparatus that generates a super-resolution image by performing super-resolution processing on an original image, and has a plurality of hierarchical frequencies for the original image. A frequency resolving unit that generates a low-frequency component image group that is a low-frequency component of the original image and a high-frequency component image group that is a high-frequency component of the original image by performing decomposition processing; According to the sampling structure, by the frequency component selection unit that selects a high-frequency component image having a Nyquist frequency equal to or lower than the Nyquist frequency of the imaging device, the original image, and the frequency component selection unit. A matching unit that performs block matching between the selected high-frequency component image and a low-frequency component image in the same hierarchy and generates alignment information indicating a corresponding positional relationship; and the alignment information. And assigning the high-frequency component image selected by the frequency component selection unit as a high-frequency component exceeding the Nyquist frequency of the original image to generate a super-resolution high-frequency component image that becomes a high-frequency component of the super-resolution image A high-frequency component generation unit; and a frequency reconfiguration unit that generates a super-resolution image by performing frequency reconstruction using the original image as a low-frequency component and the super-resolution high-frequency component image as a high-frequency component. And

  Further, in the super-resolution device according to the present invention, the frequency resolving unit further calculates a frequency spectrum power of the high-frequency component image, and the frequency component selecting unit is configured to include the imaging element in the high-frequency component image group. A high-frequency component image having a Nyquist frequency equal to or lower than the Nyquist frequency and having a frequency spectrum power equal to or lower than a threshold value is selected.

  Furthermore, in the super-resolution apparatus according to the present invention, the frequency resolving unit performs a multi-layer wavelet decomposition process with decimation on the original image.

  In order to solve the above problems, a program according to the present invention causes a computer to function as the super-resolution device.

  According to the present invention, when super-resolution processing is performed, the possibility of using blur components and aliasing components for alignment and allocation is reduced, so that the quality of a super-resolution image can be improved.

It is a block diagram which shows the structural example of the super-resolution apparatus which concerns on the 1st Embodiment of this invention. It is a figure explaining the process of the frequency decomposition part in the super-resolution apparatus which concerns on the 1st Embodiment of this invention. It is a figure explaining the process of the registration part in the super-resolution apparatus which concerns on the 1st Embodiment of this invention. It is a figure explaining the process of the super-resolution high frequency component production | generation part in the super-resolution apparatus which concerns on the 1st Embodiment of this invention. It is a figure explaining the process of the frequency reconstruction part in the super-resolution apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the super-resolution apparatus which concerns on the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of a super-resolution apparatus according to the first embodiment of the present invention. In the example shown in FIG. 1, the super resolving device 1 includes a frequency resolving unit 11, a frequency component selecting unit 12, a positioning unit 13, a super resolving high frequency component generating unit 14, and a frequency reconfiguring unit 15. Prepare.

  The super-resolution device 1 is a device that generates a super-resolution image by performing super-resolution processing on an original image of a single frame.

The frequency resolving unit 11 performs a frequency resolving process of a plurality of hierarchies (nth floor) on the original image to generate a frequency resolving image group. The frequency-resolved image group includes a low-frequency component image group LL ⁿ that is a low-frequency component of the original image and a high-frequency component image group that is a high-frequency component of the original image. The high frequency component image group includes a horizontal high frequency component image group LH ⁿ , a vertical high frequency component image group HL ⁿ , and a diagonal high frequency component image HH ⁿ . The frequency resolving unit 11 outputs the low-frequency component image group LL ⁿ to the alignment unit 13 and outputs the high-frequency component image groups LH ⁿ , HL ⁿ , HH ⁿ to the super-resolution high-frequency component generating unit 14.

  In the present embodiment, the frequency decomposition unit 11 will be described below as performing the n-th wavelet decomposition process with decimation as the frequency decomposition process. The frequency decomposition processing is preferably performed for each of the red signal, blue signal, and green signal of the original image. However, the frequency decomposition processing may be performed only on the luminance signal by obtaining the luminance signal of the original image.

FIG. 2 is a diagram illustrating an example of wavelet decomposition processing in the frequency decomposition unit 11. In the example shown in this figure, the frequency resolving unit 11 performs third-order wavelet decomposition on the original image, and first-, second-, and third-order frequency-resolved image groups {LL ⁿ , LH ⁿ , HL ⁿ , HH ⁿ | Generate n = 1, 2, 3}. Further, the frequency resolving unit 11 may perform wavelet decomposition processing on a plurality of floors after scaling the original image. In the example shown in this figure, the frequency decomposition unit 11 further reduces the original image by 0.75 times, and then performs the third-order wavelet decomposition to obtain the frequencies of the 1.5th, 2.5th, and 3.5th floors. A decomposed image group {LL ⁿ , LH ⁿ , HL ⁿ , HH ⁿ | n = 1.5, 2.5, 3.5} is generated.

  The frequency component selection unit 12 inputs color sampling structure information indicating a color sampling structure (a sampling structure of a red signal, a green signal, and a blue signal) of an imaging device that has captured the original image. Typical examples of the color sampling structure include a three-plate type, an F65 type, and a Bayer type.

The frequency component selection unit 12 has a Nyquist frequency equal to or lower than the Nyquist frequency of the imaging device that captured the original image in the high frequency component image group LH ⁿ , HL ⁿ , HH ⁿ generated by the frequency decomposition unit 11 according to the color sampling structure. A high frequency component image having a Nyquist frequency is selected (determined), frequency component selection information indicating the selection result is generated, and output to the alignment unit 13. High-frequency component images whose Nyquist frequency is less than or equal to the Nyquist frequency of the image sensor theoretically do not include aliasing components, so there is a possibility that aliasing components may be included in high-frequency component images used for super-resolution high-frequency component allocation processing described later. Can be reduced.

FIGS. 3A and 3B are diagrams for explaining the processing of the frequency component selection unit 12. FIG. 3A shows a three-plate type color sampling structure, and FIG. 3B shows an F65 type color sampling structure. FIG. 3C shows a Bayer type color sampling structure. In the figure, μ is a horizontal frequency, ν is a vertical frequency, μ _N is a horizontal Nyquist frequency, and ν _N is a vertical Nyquist frequency.

  As shown in FIG. 3A, when the color sampling structure is a three-plate type, the Nyquist frequency of the image sensor is the same for the red signal, the green signal, and the blue signal. If the aliasing component is not included in the optical characteristics of the camera lens and the low-pass filter characteristics of the image sensor, there is no possibility that the aliasing component is included in all of the red signal, the green signal, and the blue signal. Therefore, the frequency component selection unit 12 selects all frequency-resolved images (n ≧ 1) in the red signal, the green signal, and the blue signal.

  As shown in FIG. 3B, when the color sampling structure is F65 type, the Nyquist frequency of the image sensor is lowered in the oblique direction in the red signal and the blue signal. Therefore, it can be seen that there is a possibility that the aliasing component may be included in the high-frequency component images of the first floor and the 1.5th floor in the red signal and the blue signal. For this reason, the frequency component selection unit 12 selects the second or higher-order (n ≧ 2) frequency-resolved image in the red signal and the blue signal.

  As shown in FIG. 3C, when the color sampling structure is a Bayer type, the Nyquist frequency of the image sensor is lowered in the diagonal direction in the green signal, and is lowered in the horizontal and vertical directions in the red signal and the blue signal. Become. Therefore, it can be seen that there is a possibility that aliasing components may be included in the high-frequency component images of the first floor and the 1.5th floor in all of the red signal, the green signal, and the blue signal. For this reason, the frequency component selection unit 12 selects the second or higher-order (n ≧ 2) frequency-resolved image in all of the red signal, the green signal, and the blue signal.

The alignment unit 13 divides the original image into blocks of a predetermined size. Then, block matching is performed between each block of the original image and the low-frequency component image LL ^{k in the} same hierarchy k as the high-frequency component image selected by the frequency component selection information, and the block having the highest similarity (correlation) To decide. Then, alignment information (registration information) indicating the positional relationship of the corresponding blocks is generated and output to the super-resolution high-frequency component generation unit 14. Block matching is performed by a known method using an evaluation function such as an absolute value error sum (SAD) or a square error sum (SSD). In addition, block matching is performed with decimal pixel accuracy by, for example, interpolation processing using a parabolic fitting function. When the evaluation function value of SAD or SSD exceeds the threshold value, it may not be adopted as the alignment information.

The super-resolution high-frequency component generation unit 14 converts the high-frequency component solution images LH ^k , HL ^k , and HH ^k selected by the frequency component selection unit 12 according to the alignment information generated by the alignment unit 13 into the original image. Assigned as an unknown high-frequency component exceeding the Nyquist frequency, super-resolution high-frequency component images LH ^SR , HL ^SR , and HH ^SR that are high-frequency components of the super-resolution image are generated and output to the frequency reconstruction unit 15.

FIG. 4 is a diagram for explaining the processing of the alignment unit 13 and the super-resolution high-frequency component generation unit 14. In this figure, the hierarchy k of the high frequency component image selected by the frequency component selection information is only ^1, and block matching is performed between the original image I and the low frequency component image LL ¹ . A broken-line arrow from the original image I toward the low-frequency component image LL ¹ indicates the position of the corresponding block. By performing alignment between the original image and the low-frequency component image LL ¹ , when there are images having similar shapes and different sizes in the same frame, alignment is performed between similar images. Can do. Note that there is a high possibility that similar images exist in the same frame when viewed locally, so that the accuracy of alignment increases as the block size when performing block matching to the size of the original image is reduced. it can.

When assigning the high-frequency component images LH ¹ , HL ¹ , HH ¹ , the alignment information of the same phase position in the low-frequency component image LL ¹ is followed. If the block P having the low-frequency component image LL ¹ is similar to the block Q having the original image I, the block having the same phase position as the block P of each high-frequency component image LH ¹ , HL ¹ , HH ¹ is used. This is because there is a high possibility of being similar to the block Q.

  The frequency reconstruction unit 15 generates a super-resolution image by performing frequency reconstruction using the original image as a low-frequency component and the super-resolution high-frequency component image generated by the super-resolution high-frequency component generation unit 14 as a high-frequency component. Output to the outside.

FIG. 5 is a diagram illustrating an example of frequency reconstruction processing in the frequency reconstruction unit 15. In this example, the first-order wavelet reconstruction is performed using the original image I as a low-frequency component and the super-resolution high-frequency component images LH ^SR , HL ^SR , and HH ^SR as high-frequency components, thereby generating a super-resolution image I ^SR .

  As described above, in the super-resolution device 1 of the first embodiment, the frequency decomposition unit 11 generates the low-frequency component image group and the high-frequency component image group by frequency-decomposing the original image, and the frequency component selection unit 12. Thus, a high frequency component image having a Nyquist frequency equal to or lower than the Nyquist frequency of the image sensor is selected from the high frequency component image group according to the color sampling structure. Then, the alignment unit 13 performs block matching between the original image and the low-frequency component image in the same hierarchy as the selected high-frequency component image to generate alignment information, and the super-resolution high-frequency component generation unit 14 Accordingly, the super-resolution high-frequency component image is generated by allocating the high-frequency component image to the high-frequency component of the original image according to the alignment information. Finally, the frequency reconstruction unit 15 performs frequency reconstruction using the original image as a low-frequency component and the super-resolution high-frequency component image as a high-frequency component, thereby generating a super-resolution image.

  With this configuration, the super-resolution device 1 can reduce the frequency components used for alignment and allocation with a small amount of aliasing components in consideration of the color sampling structure. Therefore, the image quality of the super-resolution image can be improved.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. FIG. 6 is a block diagram illustrating a configuration example of the super-resolution apparatus according to the second embodiment. In the example shown in FIG. 6, the super resolving device 2 includes a frequency resolving unit 11 ′, a frequency component selecting unit 12 ′, an alignment unit 13, a super resolving high frequency component generating unit 14, and a frequency reconfiguring unit 15. With. Compared with the super-resolution device 1 of the first embodiment, the super-resolution device 2 of the second embodiment replaces the frequency resolution unit 11 and the frequency component selection unit 12 with a frequency resolution unit 11 ′ and a frequency. The difference is that a component selection unit 12 'is provided.

  Similar to the frequency decomposition unit 11, the frequency decomposition unit 11 ′ performs a plurality of layers (n floors) of frequency decomposition processing (for example, wavelet decomposition processing with decimation) on the original image. Further, the frequency resolving unit 11 'calculates the frequency spectrum power of each frequency component image, and outputs frequency spectrum power information, which is information indicating the frequency spectrum power, to the frequency component selecting unit 12'. The frequency spectrum power information is, for example, RMS (root mean square) of the frequency spectrum power.

The frequency component selection unit 12 ′ receives the frequency component selection information generated by the frequency decomposition unit 11 ′ in addition to the color sampling information. When the frequency spectrum power is very small, the high frequency component is hardly included in the frequency band. On the other hand, when the frequency spectrum power is very large, there is a high possibility that the frequency band includes many aliasing components. Therefore, the frequency component selection unit 12 ′ has a Nyquist frequency that is equal to or lower than the Nyquist frequency of the imaging device that captured the original image among the high-frequency component image groups LH ⁿ , HL ⁿ , and HH ⁿ generated by the frequency decomposition unit 11 ′. In addition, a high frequency component image having a frequency spectrum power equal to or lower than a threshold value is selected, and frequency component selection information indicating the selection result is generated and output to the alignment unit 13.

  The alignment unit 13, the super-resolution high-frequency component generation unit 14, and the frequency reconstruction unit 15 perform the same processing as in the first embodiment.

  That is, the super-resolution device 2 of the second embodiment generates a low-frequency component image group and a high-frequency component image group by frequency-decomposing the original image by the frequency resolving unit 11 ′, and the frequency component selecting unit 12 ′. From the high frequency component image group, a high frequency component image having a Nyquist frequency equal to or lower than the Nyquist frequency of the image sensor and having a frequency spectrum power equal to or lower than a threshold value is selected. Then, the alignment unit 13 performs block matching between the original image and the low-frequency component image in the same hierarchy as the selected high-frequency component image to generate alignment information, and the super-resolution high-frequency component generation unit 14 Thus, the super-resolution high-frequency component is generated by allocating the high-frequency component image as the high-frequency component of the original image according to the alignment information. Finally, the frequency reconstruction unit 15 performs frequency reconstruction using the original image as a low-frequency component and the super-resolution high-frequency component image as a high-frequency component, thereby generating a super-resolution image.

  The super-resolution device 2 uses the frequency component selection information in addition to the color sampling information to select a frequency component image to be used for alignment and allocation. Therefore, the super-resolution device 2 can further prevent the frequency component including the aliasing component from being used for alignment and allocation than the super-resolution device 1, and can improve the image quality of the super-resolution image.

  It should be noted that a computer can be suitably used to function as the above-described super-resolution devices 1 and 2, and such a computer is a program describing processing contents for realizing the functions of the super-resolution devices 1 and 2. Is stored in a storage unit of the computer, and the program is read and executed by the CPU of the computer. This program can be recorded on a computer-readable recording medium.

  Although the above embodiment has been described as a representative example, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiments can be combined into one, or one constituent block can be divided.

DESCRIPTION OF SYMBOLS 1, 2 Super-resolution apparatus 11, 11 'Frequency decomposition part 12, 12' Frequency component selection part 13 Positioning part 14 Super-resolution high frequency component production | generation part 15 Frequency reconstruction part

Claims (4)

A super-resolution device that generates a super-resolution image by super-resolution processing of an original image,
A frequency resolving unit that performs multi-level frequency decomposition processing on the original image to generate a low-frequency component image group that is a low-frequency component of the original image and a high-frequency component image group that is a high-frequency component of the original image;
A frequency component selection unit that selects a high-frequency component image having a Nyquist frequency equal to or lower than the Nyquist frequency of the image sensor from the high-frequency component image group according to the color sampling structure of the image sensor that captured the original image;
A positioning unit that performs block matching between the original image and a low-frequency component image in the same hierarchy as the high-frequency component image selected by the frequency component selection unit, and generates alignment information indicating a corresponding positional relationship; ,
According to the alignment information, the high-frequency component image selected by the frequency component selection unit is allocated as a high-frequency component exceeding the Nyquist frequency of the original image to generate a super-resolution high-frequency component image that becomes a high-frequency component of the super-resolution image A super-resolution high-frequency component generation unit,
The original image is a low frequency component, the super resolving high frequency component image is a high frequency component, frequency reconstruction is performed, and a frequency reconstructing unit that generates a super resolving image;
A super-resolution device comprising: The frequency resolving unit further calculates a frequency spectrum power of the high-frequency component image;
The frequency component selection unit selects a high-frequency component image having a Nyquist frequency equal to or lower than the Nyquist frequency of the imaging element and having a frequency spectrum power equal to or lower than a threshold value from the high-frequency component image group. The super-resolution device according to claim 1.   The super-resolution apparatus according to claim 1, wherein the frequency resolution unit performs a multi-level wavelet decomposition process with decimation on the original image.   The program for functioning a computer as a super-resolution apparatus as described in any one of Claim 1 to 3.

Images