JP6532148B2 - Super-resolution device and program - Google Patents

Description

  The present invention relates to a blur correction device, a super resolution device, and a program.

  As a technique for correcting a signal such as a blurred image, there is a technique for inversely correcting a point spread function of blur. As a method of inverse correction, there is a method using a Wiener filter, and a method of making a correction result converge on a sharp image by performing an iterative operation so that a blurred result of the temporary correction result matches the input signal.

On the other hand, in the case of converting the resolution of an image, the input image is convoluted with an interpolation function, and sampling is performed at a sampling period different from that of the input image. In particular, if the resolution of the output image is higher than the resolution of the input image, a super resolution technique may be used to obtain an output image with a sense of definition by compensating for signal components exceeding the Nyquist frequency of the input image. is there.
In the super-resolution technique, when an image is given in time series, motion compensation (alignment) is performed with subpixel accuracy for each image region, and the image of interest and other images close in time, There is a multi-frame super resolution technology that reproduces signal components exceeding the Nyquist frequency by superimposing the captured images.

In addition, there is also a method for improving the sense of definition in appearance by artificially combining signal components exceeding the Nyquist frequency from one image, and this method is called single frame super resolution technology.
For example, in the non-linear method, which is one of single-frame super resolution techniques, harmonics are generated by applying a non-linear function to the high frequency components of the input image. In this method, the signal gradient is enhanced by superimposing harmonics in the amplitude direction in order to compensate for blur in the spatial direction (see, for example, Patent Document 1).

  In addition, there is learning type super resolution as another single frame super resolution technology. In learning-type super resolution, a low resolution patch and a high resolution patch pair are made into a database in advance for a small area of an image, and the high resolution patch is referenced while referring to the database for each portion of the input low resolution image. Convert to By doing this, natural high definition can be achieved if a reasonable texture is registered in the database (see, for example, Patent Document 2).

Patent No. 5396626 gazette Patent No. 4140690

However, in the method of inversely correcting the point spread function of the blur, the point spread function needs to be known, and if there is a lag between the assumed point spread function and the actual point spread function, the restored result is an artifact. It occurs.
In addition, when there is a zero (null) in the frequency transfer characteristic of the point spread function of the blur, the component of the zero can not be restored because the inverse transfer characteristic is singular. Similarly, when the blur has a frequency transfer characteristic close to that of an ideal low-pass filter, it is difficult to reversely correct the high frequency component because the high frequency component becomes almost zero. In particular, when the image contains noise, the amplification effect of the noise component becomes more prominent than the correction effect of the signal component, which may result in the deterioration of the image quality.

Even in the case of multi-frame super resolution, high-frequency restoration is basically impossible if the input image does not include a component exceeding the Nyquist frequency of the image as aliasing distortion. Also, with respect to a set of motion vectors obtained from a plurality of image pairs, a set of vectors consisting of the decimal part of the horizontal component and the decimal part of the vertical component is an open area (0,1) × (0,1) (where the operator is There is also a problem that high-range restoration can not be performed with good balance (small difference in definition between horizontal and vertical directions) unless x is uniformly distributed within the product.
In the case of the single frame super resolution non-linear method, harmonics are superimposed in the amplitude direction for the purpose of compensating for blurring in the spatial direction. Therefore, there is a problem that artifacts such as overshoot, undershoot, and ringing are easily generated in the signal waveform.

  The effectiveness of learning-type super-resolution depends on the quality and quantity of databases. The method is useless for textures not registered in the database in advance. In learning type super resolution, a database construction is performed by machine learning using pairs of low resolution images and high resolution images for learning in advance. If the textures in the learning image at this time are biased, or if the amount of the learning image is insufficient, generalization of learning is not performed. When a database that is not generalized is used, an image output from learning type super resolution becomes an unnatural image.

  The present invention has been made in view of such circumstances, and provides a blur correction device, a super-resolution device, and a program that can more naturally correct blur.

(1) The present invention has been made to solve the problems described above, and one aspect of the present invention is an expansion operation unit that performs expansion operation of morphological operation on an input signal, and morphology on the input signal. A contraction calculation unit performing contraction calculation of the calculation, and a first signal representing a calculation result by the expansion calculation unit when the first condition is satisfied, and a second signal by the contraction calculation unit when the second condition is satisfied And a switching unit configured to output a second signal representing a calculation result.

(2) Further, another aspect of the present invention is the blur correction device according to (1), wherein when the switching unit satisfies a third condition, the calculation result by the expansion calculating unit and the calculation result A third signal representing a value between the contraction calculation unit and the calculation result is output.

(3) Further, another aspect of the present invention is the blur correction device according to (2), wherein the third signal is the input signal.

(4) Further, another aspect of the present invention is the blur correction device according to any one of (1) to (3), wherein the first condition is a calculation result by the expansion calculation unit and the input. The difference between the signal and the value represented by the signal is smaller than the difference between the calculation result by the contraction calculation unit and the value represented by the input signal, and the second condition is the calculation result by the expansion calculation unit and the input The difference between the value represented by the signal and the value represented by the signal is larger than the difference between the calculation result of the contraction calculation unit and the value represented by the input signal.

(5) Moreover, the other aspect of this invention interpolates with respect to the input signal, The interpolation part which makes the resolution of the said input signal high, The signal which the said interpolation part made the resolution high And the blurring correction device according to any one of (1) to (4).

(6) Further, according to another aspect of the present invention, there is provided an expansion operation unit that performs expansion operation of morphology operation on an input signal, a contraction operation unit that performs contraction operation of morphology operation on the input signal, A switching unit that outputs a first signal representing the calculation result by the expansion calculation unit when the condition of 1 is satisfied, and outputs a second signal representing the calculation result by the contraction calculation unit when the second condition is satisfied Is a program to function as

(7) Moreover, the other aspect of this invention interpolates with respect to the input signal to a computer, The interpolation part which makes the resolution of the said input signal high, The said interpolation part makes resolution high It is a program for functioning as a blur correction device given in either of (1) to (4) which uses the input signal as the input signal.

  According to the present invention, blur can be corrected more naturally.

FIG. 1 is a schematic block diagram showing the configuration of a blur correction device 1 according to a first embodiment of the present invention. It is a figure explaining the 1st example of field kernels K and J in the embodiment. It is a figure explaining the 2nd example of field kernels K and J in the embodiment. It is a schematic diagram explaining operation | movement of the blur correction apparatus 1 in the embodiment. It is a schematic block diagram explaining the structure of the super-resolution apparatus 2 by 2nd Embodiment of this invention. It is a figure which shows the example of the low resolution image L input into the super-resolution apparatus 2 in the embodiment. It is a figure which shows the example of the image by which the resolution was raised by the interpolation interpolation part 14 in the embodiment. It is a figure which shows the example of the correction | amendment image (high resolution image) H by the super-resolution apparatus 2 in the embodiment.

First Embodiment
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of the blur correction device 1 according to the present embodiment. The blur correction device 1 performs a process of correcting blur on the input image M (input signal), and outputs a corrected image H that is the result of the process. The blur correction device 1 includes an expansion calculation unit 11, a contraction calculation unit 12, and a comparison switching unit 13.

  A pixel value of image coordinates (x, y) in the input image M is denoted as M (x, y). The same applies to pixel values of other images. In the following, processing in the case where the input image M is a monochrome image will be described. When the input image M is a color image or a multiband image, similar processing may be performed independently for each color (or band).

  The expansion operation unit 11 applies expansion operation (Dilation) of morphology operation to the input image M, expands pixel values in the spatial direction, and outputs an expanded image D. For example, Equation 1 is used for the expansion calculation of the expansion calculation unit 11. In addition to (Equation 1), for example, in the region kernel K, the pixel value is multiplied by a weight that decreases as the distance from the center increases, and the pixel value of the largest pixel among them is selected. You may use a thing. The region (set) kernel K in (Equation 1) is the size and shape of spatial expansion. Details of the area kernel K will be described later.

  The contraction operation unit 12 applies contraction operation (Erosion) of morphology operation to the input image M, contracts a pixel value in the space direction, and outputs a contracted image E. For example, (Equation 2) is used for the contraction calculation of the contraction calculation unit 12. In addition to the equation (2), for example, in the region kernel J, the pixel value is multiplied by a weight that increases as the distance from the center increases, and the pixel value of the pixel having the minimum value among them is selected. You may use a thing. The region (set) kernel J in (Equation 2) is the size and shape of the spatial contraction. Details of the area kernel J will be described later.

  The comparison switching unit 13 outputs a corrected image H generated from the input image M (third signal), the expanded image D (first signal), and the contracted image E (second signal). The comparison switching unit 13 generates a corrected image H as follows. First, the comparison switching unit 13 compares the input image M, the dilation image D, and the contraction image E pixel by pixel. Specifically, for each pixel position (x, y), the comparison switching unit 13 calculates the difference A (x, y) between the D (x, y) and the M (x, y), and M (x, y) And the difference B (x, y) of E (x, y) are compared. The difference A (x, y) and the difference B (x, y) are obtained by (Equation 3).

Next, the comparison switching unit 13 determines the pixel value H (x, y) of the corrected image H according to the following rule according to the magnitude of the difference A (x, y) and the difference B (x, y).
(A) H (x, y) = D (x, y) if A (x, y) <B (x, y)
(B) H (x, y) = E (x, y) if A (x, y)> B (x, y)
(C) H (x, y) = E (x, y) if A (x, y) = B (x, y)
That is, the comparison switching unit 13 determines the pixel value H (x, y) of the corrected image H according to (Expression 4).

In the case of (c), the value of H (x, y) may be an intermediate value between D (x, y) and E (x, y), for example, D (x, y) May be M (x, y), for example, the average of D (x, y) and E (x, y), D (x, y) and E (x, y) Or E (x, y) or more and D (x, y) or less calculated from D (x, y), E (x, y) and M (x, y) May be
That is, the comparison switching unit 13 may determine the pixel value H (x, y) of the corrected image H according to any one of (Expression 5), (Expression 6), and (Expression 7).

  The comparison switching unit 13 generates and outputs the corrected image H by performing the above-described determination of the pixel value H (x, y) for all the pixel positions in the screen and imaging.

  Each of FIG. 2, FIG. 3 is a figure explaining the example of area | region kernel K and J. FIG. As the area kernel K and the area kernel J, an area including the target pixel and its peripheral pixels, for example, an area as shown in FIGS. 2 and 3 can be used. In FIG. 2 and FIG. 3, the horizontal axis and the vertical axis are i and j in (Expression 1) and (Expression 2), and the set of i, j belonging to the shaded area is the area kernel K, It is J. For example, in FIG. 2, the combination of (i, j) of the region kernels K and J is (−3, 1), (−3, 0), (−3, −1), (−2, 2) ), (−2, 1), (−2, 0), (−2, −1), (−2, −2), (−1, 3), (−1, 2), (−1, 1), (-1, 0), (-1, -1), (-1, -2), (-1, -3), (0, 3), (0, 2), (0, 1) ), (0, 0), (0, -1), (0, -2), (0, -3), (1, 3), (1, 2), (1, 1), (1, 1) 0), (1, -1), (1, -2), (1, -3), (2, 2), (2, 1), (2, 0), (2, -1), (( 2, 2), (3, 1), (3, 0), (3, -1). Also, in FIG. 3, the combination of (i, j) of the region kernels K and J is (0, 0), (1, 0), (0, 1), (-1, 0), (0) , -1). It is preferable to use the same area as the area kernel K and the area kernel J, but they may be different.

  FIG. 4 is a schematic view for explaining the operation of the blur correction device 1. Here, for the sake of simplicity, a one-dimensional signal is shown instead of the image. First, the expansion operation unit 11 converts the input image M100 into an expanded image D102 by applying the expansion operation using the area kernel K101. The contraction operation unit 12 converts the input image M100 into a contraction image E104 by applying contraction calculation using the region kernel J103. The comparison switching unit 13 performs comparison and switching according to (Expression 4) based on the input image M100, the expanded image D102, and the contracted image E104 to obtain a corrected image H105.

  By such an operation, the waveform of the corrected image H105 is steeper than the waveform of the input image M100, as in the regions 106 and 107 surrounded by the broken line in the corrected image H105 in FIG. 4. That is, the edge portion of the image can be sharpened to more naturally correct the blur.

Second Embodiment
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. FIG. 5 is a schematic block diagram for explaining the configuration of the super resolving device 2 in the present embodiment. The super resolution device 2 includes an interpolation unit 14 and a blur correction device 1. The super-resolution device 2 has a configuration in which an interpolation / interpolation unit 14 is provided at the front stage of the blur correction device 1.

The interpolation unit 14 performs interpolation on the input low resolution image L to generate an interpolation image with an increased number of pixels, and generates the generated interpolation image as an input image M to the blur correction device 1 Output as The interpolation function used for interpolation by the interpolation unit 14 is preferably one that does not impair the smoothness of the image signal. For example, bilinear interpolation, bicubic interpolation, spline interpolation, interpolation by Lanczos filter, or the like is preferable to zero-order interpolation (nearest neighbor interpolation).
For example, the interpolation unit 14 interpolates the low resolution image L twice in both horizontal and vertical directions by bilinear interpolation using (Equation 8).

  Also, for example, the interpolation unit 14 interpolates twice horizontally and vertically by the Lanczos-3 function using (Equation 9).

However, in (Expression 9), the same filter coefficient is applied to all sample points by shifting the sample position horizontally and vertically (by the pixel interval of the low resolution image) by 1⁄4 pixel.
The interpolation unit 14 inputs the interpolation image thus obtained to the blur correction device 1 as the input image M. As a result, from the blur correction device 1, it is possible to obtain a sharp corrected image (high resolution image) H in which the number of pixels is larger than that of the low resolution image L and blur generated in the interpolation unit 14 is corrected. .

  FIG. 6 shows an example L1 of the low resolution image L input to the super resolution device 2. FIG. 7 is an example M1 of an image in which the resolution is enhanced by the interpolation unit 14 when the example L1 of the low resolution image L is input to the super resolution device 2. FIG. 8 shows an example H1 of a corrected image (high resolution image) H by the super resolution device 2 when the example L1 of the low resolution image L is input to the super resolution device 2. In the example M1 of FIG. 7, blurring is seen at the edge portion, but in the example H1 of FIG. 8, sharp results are obtained. In the example H1, the region kernel K is used for the example of FIG. 2 as the region kernel J, the equation (6) for the comparison switching unit 13, and the equation 9 for the interpolation unit 14 It was.

  In the present embodiment, the sizes of the area kernels K and J in the blur correction device 1 may be changed according to the enlargement ratio of the number of pixels in the predetermined direction in the interpolation unit 14. Furthermore, it is desirable that the radius of the area kernels K and J be 1/2 or less of this enlargement factor. By doing this, only the pixels included in the area kernels K and J have a common pixel that is the basis for calculating the pixel value by interpolation, and therefore it is possible to prevent blurring other than by interpolation. The effect of blur correction can be suppressed. For example, the expansion operation unit 11 sets 1⁄4 of the enlargement ratio of the number of pixels in the horizontal direction in the interpolation unit 14 as the radius of the region kernel K, and the contraction operation unit 12 generates the horizontal direction in the interpolation unit 14. Let 1⁄4 of the enlargement ratio of the number of pixels be the radius of the area kernel J. Also, the radius of the area kernels K, J may be changed depending on the direction. For example, assuming that 1⁄4 of the enlargement ratio of the number of pixels in the horizontal direction in the interpolation unit 14 is the radius in the horizontal direction of the area kernels K and J, 1 of the enlargement ratio of the number of pixels in the vertical direction Alternatively, / 4 may be the longitudinal radius of the region kernel K.

  In each of the above-described embodiments, the blur correction device 1 and the super-resolution device 2 that process signals representing an image are exemplified. However, the blur correction device 1 and the super-resolution device 2 may be voices, three-dimensional models, etc. Signals representing something other than an image may be processed. Also, the signal may be a one-dimensional signal or a signal of two or more dimensions.

  In addition, the program for realizing the functions of the blur correction device 1 in FIG. 1 and the super-resolution device 2 in FIG. 5 is recorded in a computer readable recording medium, and the program recorded in the recording medium is The blur correction device 1 and the super-resolution device 2 may be realized by reading and executing. Here, the “computer system” includes an OS and hardware such as peripheral devices.

  The term "computer-readable recording medium" refers to a storage medium such as a flexible disk, a magneto-optical disk, a ROM, a portable medium such as a ROM or a CD-ROM, or a hard disk built in a computer system. Furthermore, “computer-readable recording medium” dynamically holds a program for a short time, like a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, the volatile memory in the computer system which is the server or the client in that case, and the one that holds the program for a certain period of time is also included. The program may be for realizing a part of the functions described above, or may be realized in combination with the program already recorded in the computer system.

In addition, each functional block of the blur correction device 1 in FIG. 1 and the super-resolution device 2 in FIG. 5 may be individually chipped, or a part or all of them may be integrated and chipped. Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. Either hybrid or monolithic may be used. Some of the functions may be implemented by hardware and some may be implemented by software.
Further, when a technology such as integrated circuit substitution for LSI emerges with the advance of semiconductor technology, it is also possible to use an integrated circuit according to the technology.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and design changes and the like within the scope of the present invention are also included.

Reference Signs List 1 blur correction device 2 super resolution device 11 expansion calculation unit 12 contraction calculation unit 13 comparison switching unit 14 interpolation interpolation unit

Claims (5)

What is claimed is: 1. A super-resolution device comprising: an interpolation unit that interpolates an input signal; and a blur correction device that corrects the interpolated input signal,
The interpolation unit shifts the sample position at the time of the interpolation in each of horizontal and vertical directions by 1⁄4 pixel by the pixel interval of the low resolution image by the input signal, and interpolates using a predetermined function. ,
The blur correction device
An expansion operation unit that performs expansion operation of morphological operation on the input signal interpolated by the interpolation unit;
A contraction operation unit that performs contraction operation of morphology operation on the input signal;
When the first condition is satisfied, the first signal representing the calculation result by the expansion calculation unit is output, and when the second condition is satisfied, the second signal representing the calculation result by the contraction calculation unit is output. A super-resolution device characterized by comprising: When the third condition is satisfied, the switching unit outputs a third signal representing a value between the calculation result of the expansion calculation unit and the calculation result of the contraction calculation unit. The super resolution device according to 1 . The super resolution device according to claim 2, wherein the third signal is an input signal interpolated by the interpolation unit . The first condition is that the difference between the calculation result by the expansion calculation unit and the value represented by the input signal interpolated by the interpolation unit is the calculation result by the contraction calculation unit and the input interpolated by the interpolation unit. It must be smaller than the difference between the signal and the value it represents,
The second condition is that the difference between the calculation result by the expansion calculation unit and the value represented by the input signal interpolated by the interpolation unit is the input obtained by the calculation result by the contraction calculation unit and the interpolation unit The super-resolution device according to any one of claims 1 to 3, characterized in that it is larger than the difference from the value represented by the signal. Computer,
Program for functioning as a super-resolution device according to 請 Motomeko 1 to one of any claims 4.