JP6532151B2 - Super-resolution device and program - Google Patents

Description

  The present invention relates to a blur correction device, a super resolution device, and a program.

  As a technique for correcting a signal such as a blurred image, there is a technique for inversely correcting a point spread function of blur. As a method of inverse correction, there is a method using a Wiener filter, and a method of making a correction result converge on a sharp image by performing an iterative operation so that a blurred result of the temporary correction result matches the input signal.

On the other hand, in the case of converting the resolution of an image, the input image is convoluted with an interpolation function, and sampling is performed at a sampling period different from that of the input image. In particular, if the resolution of the output image is higher than the resolution of the input image, a super resolution technique may be used to obtain an output image with a sense of definition by compensating for signal components exceeding the Nyquist frequency of the input image. is there.
In the super-resolution technique, when an image is given in time series, motion compensation (alignment) is performed with subpixel accuracy for each image region, and the image of interest and other images close in time, There is a multi-frame super resolution technology that reproduces signal components exceeding the Nyquist frequency by superimposing the captured images.

In addition, there is also a method for improving the sense of definition in appearance by artificially combining signal components exceeding the Nyquist frequency from one image, and this method is called single frame super resolution technology.
For example, in the non-linear method, which is one of single-frame super resolution techniques, harmonics are generated by applying a non-linear function to the high frequency components of the input image. In this method, the signal gradient is enhanced by superimposing harmonics in the amplitude direction in order to compensate for blur in the spatial direction (see, for example, Patent Document 1).

  In addition, there is learning type super resolution as another single frame super resolution technology. In learning-type super resolution, a low resolution patch and a high resolution patch pair are made into a database in advance for a small area of an image, and the high resolution patch is referenced while referring to the database for each portion of the input low resolution image. Convert to By doing this, natural high definition can be achieved if a reasonable texture is registered in the database (see, for example, Patent Document 2).

Patent No. 5396626 gazette Patent No. 4140690

However, in the method of inversely correcting the point spread function of the blur, the point spread function needs to be known, and if there is a lag between the assumed point spread function and the actual point spread function, the restored result is an artifact. It occurs.
In addition, when there is a zero (null) in the frequency transfer characteristic of the point spread function of the blur, the component of the zero can not be restored because the inverse transfer characteristic is singular. Similarly, when the blur has a frequency transfer characteristic close to that of an ideal low-pass filter, it is difficult to reversely correct the high frequency component because the high frequency component becomes almost zero. In particular, when the image contains noise, the amplification effect of the noise component becomes more prominent than the correction effect of the signal component, which may result in the deterioration of the image quality.

Even in the case of multi-frame super resolution, high-frequency restoration is basically impossible if the input image does not include a component exceeding the Nyquist frequency of the image as aliasing distortion. Also, with respect to a set of motion vectors obtained from a plurality of image pairs, a set of vectors consisting of the decimal part of the horizontal component and the decimal part of the vertical component is an open area (0,1) × (0,1) (where the operator is There is also a problem that high-range restoration can not be performed with good balance (small difference in definition between horizontal and vertical directions) unless x is uniformly distributed within the product.
In the case of the single frame super resolution non-linear method, harmonics are superimposed in the amplitude direction for the purpose of compensating for blurring in the spatial direction. Therefore, there is a problem that artifacts such as overshoot, undershoot, and ringing are easily generated in the signal waveform.

  The effectiveness of learning-type super-resolution depends on the quality and quantity of databases. The method is useless for textures not registered in the database in advance. In learning type super resolution, a database construction is performed by machine learning using pairs of low resolution images and high resolution images for learning in advance. If the textures in the learning image at this time are biased, or if the amount of the learning image is insufficient, generalization of learning is not performed. When a database that is not generalized is used, an image output from learning type super resolution becomes an unnatural image.

  The present invention has been made in view of such circumstances, and provides a blur correction device, a super-resolution device, and a program that can more naturally correct blur.

(1) This invention was made in order to solve the subject mentioned above, and one mode of the present invention is a signal at the time of scanning a predetermined scanning direction in the signal position concerned about each of a plurality of signal positions of an input signal. A correction pattern determination unit that determines a correction pattern related to a signal value in the vicinity of the signal position based on a sign change of a second-order difference value of the value sequence and a sign of a first-order difference value of the signal value sequence; And a correction unit that adds the correction pattern determined by the correction pattern determination unit to the input signal for each of the signal positions.

(2) Another aspect of the present invention is the blur correction device according to (1), wherein the correction pattern determined by the correction pattern determination unit has a positive value along the scanning direction, It is characterized in that it includes a correction pattern in which values are arranged in the order of negative values, and a correction pattern in which values are arranged in the order of positive values and negative values along the reverse direction of the scanning direction.

(3) Further, another aspect of the present invention is the blur correction device according to (1) or (2), wherein the correction pattern determination unit determines the amplitude of the correction pattern as the first-order difference value. It is characterized in that the value corresponds to the size.

(4) Moreover, the other aspect of this invention interpolates with respect to the input signal, The interpolation part which makes the resolution of the said input signal high, The signal which the said interpolation part made the resolution high And the blur correction device according to any one of (1) to (3), wherein the input signal is the input signal.

(5) Further, another aspect of the present invention is the sign change of the second-order difference value of the signal value sequence when the computer is scanned in a predetermined scanning direction at a plurality of signal positions of the input signal. And a correction pattern determination unit that determines a correction pattern related to signal values in the vicinity of the signal position based on a sign of a first-order difference value of the signal value sequence, the correction pattern determination unit for each of the plurality of signal positions Is a program for functioning as a correction unit that adds the determined correction pattern to the input signal.

(6) Moreover, the other aspect of this invention interpolates with respect to the input signal to the computer, The interpolation part which makes the resolution of the said input signal high, The said interpolation part makes resolution high The program according to any one of claims 1 to 3, wherein the input signal is a signal that has been input.

  According to the present invention, blur can be corrected more naturally.

FIG. 1 is a schematic block diagram showing a configuration of a blur correction device according to a first embodiment of the present invention. It is a figure which shows the pixel used for calculation of the correction value of pixel position (i, j) in the embodiment. It is a figure which shows the example of the correction | amendment pattern in the embodiment. It is a schematic block diagram explaining the structure of the super-resolution apparatus 2 by 2nd Embodiment of this invention. It is an example L1 of the low resolution image L input to the super resolution device 2 in the same embodiment. When an example L1 of the low resolution image L in the same embodiment is input to the super resolution device 2, an example M1 of the image whose resolution is enhanced by the interpolation unit 20 is shown. It is an example H1 of a corrected image (high resolution image) H by the super resolution device 2 when an example L1 of the low resolution image L in the same embodiment is input to the super resolution device 2.

First Embodiment
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing a configuration of a blur correction device according to a first embodiment of the present invention. The blur correction device 1 performs a process of correcting blur on the input image M (input signal), and outputs a corrected image H that is the result of the process. The blur correction device 1 includes a horizontal difference calculation unit 11, a correction value calculation unit 12, a vertical difference calculation unit 13, a correction value calculation unit 14, and a correction unit 15.

A pixel value of image coordinates (x, y) in the input image M is denoted as M (x, y). The same applies to pixel values of other images. The resolutions (number of pixels) of the input image M and the corrected image H are X pixels in the horizontal direction and Y pixels in the vertical direction, and i∈ {0, 1,..., X−1} and j∈ {0, 0 Pixel values are defined on image coordinates (i, j) such as 1,.
In the following, processing in the case where the input image M is a monochrome image will be described. When the input image M is a color image or a multiband image, similar processing may be performed independently for each color (or band).

The blur correction device 1 performs correction processing for each pixel position (i, j) of the input image M, and sets the result as the pixel value H (i, j) at the pixel position (i, j) of the corrected image H The corrected image H is generated by performing the operation for all pixel positions (i, j). The horizontal difference calculation unit 11 calculates second-order difference values and first-order difference values in the horizontal direction (x-axis direction) at each pixel position (i, j) of the input image M. The correction value calculation unit 12 calculates a reinforcement value V ^(k) (i, j) for each pixel position (i, j) of the input image M, and calculates these reinforcement values V ^(k) (i, j) A reinforcement image V ^(k) having a pixel value is generated.

A method of calculating the reinforcement value V ^(k) (i, j) by the correction value calculation unit 12 will be described. When the correction value calculation unit 12 calculates the reinforcement value V ^(k) (i, j), among the second-order difference value and the first-order difference value calculated by the horizontal difference calculation unit 11, shown in (Expression 1) Two second-order difference values P and Q and one first-order difference value R are used.

Note that the calculation of these two second-order difference values P and Q and one first-order difference value R is based on the image coordinates (i, j) from the two pixels on the left to the one pixel on the right The pixel values of the four pixels A, B, C, and D are used. FIG. 2A shows these four pixels. M (i-2, j) in (Expression 1) is the pixel value of the pixel A, M (i-1, j) is the pixel value of the pixel B, and M (i, j) is , M (i + 1, j) is the pixel value of the pixel D.

The correction value calculation unit 12 uses the above-described two second-order difference values P and Q and one first-order difference value R to calculate the reinforcement value V ^(k) (i, j) as a recurrence formula of (Expression 2) Calculate using Correction value calculation unit 12, the calculation of the reinforcement value ^{V (k) (i, j} ), by performing for every pixel position (i, j), the image V for correcting the blur of the input image M ^{( k)} Generate (a reinforced image). Note that the superscript (k) is an index indicating the number of repetitions when deriving the reinforcement image by the iterative operation. Note that V ^(-1) (i) is applied to all pixel positions (i, j) where i ⁻¹ {0, 1, ..., X-1} and j ∈ {0, 1, ..., Y-1}. , J) = 0.

  Here, α is a predetermined positive constant. FIG. 3A shows an example of the update amount (correction pattern) according to (Expression 2). In the example of FIG. 3A, in the case of (1) of (Expression 2), that is, P (i, j)> 0, Q (i, j) <0, and R (i, j)> 0. It is an example of time. As described above, the correction value calculation unit 12 (correction pattern determination unit) determines positive and negative pulses (correction pattern) to be added to the pixel on the left and the pixel with respect to the pixel position (i, j). Do. In the case of (2) of (Expression 2), α in FIG. 3A is −α, and −α is α. That is, in the case of (1), it is a correction pattern in which the values are arranged in the order of the positive value and the negative value along the scanning direction, and in the case of (2), the positive value along the reverse direction of the scanning direction It is a correction pattern in which values are arranged in the order of negative values.

The correction value calculator 12 may make the magnitudes (amplitudes) of positive and negative pulses to be added proportional to the first-order difference value R. For example, the correction value calculation unit 12 may calculate the reinforcement value V ^(k) (i, j) using the recurrence formula of (Expression 3).

The vertical difference calculation unit 13 calculates second-order difference values and first-order difference values in the vertical direction (y-axis direction) at each pixel position (i, j) of the input image M. The correction value calculation unit 14 calculates a reinforcement value V ^(k) (i, j) for each pixel position (i, j) of the input image M, and calculates these reinforcement values V ^(k) (i, j) A reinforcement image V ^(k) having a pixel value is generated.

A method of calculating the reinforcement value V ^(k) (i, j) by the correction value calculation unit 14 will be described. When the correction value calculation unit 14 calculates the reinforcement value V ^(k) (i, j), the second-order difference value and the first-order difference value calculated by the vertical difference calculation unit 13 are represented by (Expression 4) Use two second-order difference values S and T and one first-order difference value U.

Note that the calculation of these two second-order difference values S and T and one first-order difference value U is based on the image coordinates (i, j) by the pixel two pixels above and one pixel below The pixel values of the four pixels E, F, C, and G are used. FIG. 2 (b) is a diagram showing these four pixels. M (i, j-2) in (Expression 4) is the pixel value of the pixel E, M (i, j-1) is the pixel value of the pixel F, and M (i, j) is , M (i, j + 1) are pixel values of the pixel G.

The correction value calculator 14 uses the above-described two second-order difference values S and T and one first-order difference value U to calculate the reinforcement value V ^(k) (i, j) as a recurrence equation of (Equation 5) Calculate using The correction value calculation unit 14 performs the calculation of the reinforcement value V ^(k) (i, j) for all pixel positions (i, j) to obtain the image V for correcting the blur of the input image M ^{( k)} Generate (a reinforced image). V ^(XY-1) (i, j) is a reinforcement value calculated by the correction value calculation unit 12.

  Here, α is a predetermined positive constant. The example of the update amount by (Formula 5) is illustrated in figure 3 (b). In the example of FIG. 3B, in the case of (1) in (Expression 5), that is, S (i, j)> 0, T (i, j) <0, and U (i, j)> 0. It is an example of time. As described above, the correction value calculation unit 14 (correction pattern determination unit) determines positive and negative pulses (correction pattern) to be added to the pixel adjacent on the upper side with respect to the pixel position (i, j) and the corresponding pixel. Do. In the case of (2) of (Expression 5), α in FIG. 3B is −α, and −α is α.

The correction value calculator 14 may make the magnitudes of the positive and negative pulses to be added proportional to the first-order difference value U. For example, the correction value calculator 14 may calculate the reinforcement value V ^(k) (i, j) using the recurrence formula of (Expression 6).

The correction value calculation unit 14 finally calculates V ^(2XY-1) which is the final value of the reinforcement image (the value after scanning all the pixels of the input image M) as shown in (Expression 7). The image is output as a reinforced image V.

The horizontal difference calculating unit 11 and the vertical difference calculating unit 13 may be replaced with each other, and the correction value calculating unit 12 and the correction value calculating unit 14 may be replaced with each other.
Furthermore, the reinforcement image by only the horizontal difference calculation unit 11 and the correction value calculation unit 12 and the reinforcement image by only the vertical difference calculation unit 13 and the correction value calculation unit 14 may be calculated independently. In this case, the two reinforcement images may be sequentially applied in the correction unit 15 described later, or the sum of the two reinforcement images is used as a final reinforcement image, and the final reinforcement image is described in the correction unit 15 described below. May apply.

  The correction unit 15 obtains the sum of the pixel values of each pixel of the input image M and the reinforcement image V, and sets the result as the pixel value of the corresponding pixel position of the corrected image H. That is, the correction unit 15 performs the operation shown in (Expression 8).

The correction unit 15 generates and outputs a corrected image H by performing determination of the pixel value H (i, j) for all pixel positions in the screen and imaging.

For example, in (Equation 2), the pixel position which becomes (1) is a point where the pixel value is increasing and the curve on which the pixel value is plotted changes from convex downward to convex. Such a pixel position is considered to be an outline of a subject or the like, and the blur correction device 1 adds a negative pulse to the pixel value adjacent on the left and adds a positive pulse to the pixel value. That is, the curve in which the pixel values are plotted is made to be steep. The pixel position (2) is a point at which the pixel value decreases and the curve plotting the pixel value changes from convex upward to convex downward. Such a pixel position is regarded as an outline of a subject or the like, and the blur correction device 1 adds a positive pulse to the pixel value adjacent on the left and adds a negative pulse to the pixel value. That is, the curve in which the pixel values are plotted is made to be steep.
As described above, even when the outline of the subject is blurred, the blur correction device 1 can make the pixel value change in the outline or the like steep and correct the blur more naturally.

Second Embodiment
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. FIG. 5 is a schematic block diagram for explaining the configuration of the super resolving device 2 in the present embodiment. The super resolution device 2 includes an interpolation unit 20 and a blur correction device 1. The super-resolution device 2 has a configuration in which an interpolation / interpolation unit 20 is provided at the front stage of the blur correction device 1.

Interpolation The interpolation unit 20 performs interpolation on the input low resolution image L to generate an interpolation image having an increased number of pixels, and generates the generated interpolation image as an input image M to the blur correction device 1. Output as The interpolation function used for interpolation by the interpolation unit 20 is preferably one that does not impair the smoothness of the image signal. For example, bilinear interpolation, bicubic interpolation, spline interpolation, interpolation by Lanczos filter, or the like is preferable to zero-order interpolation (nearest neighbor interpolation).
For example, the interpolation unit 20 interpolates the low resolution image L twice in both horizontal and vertical directions by bilinear interpolation using (Expression 9).

  Also, for example, the interpolation unit 20 interpolates twice horizontally and vertically by the Lanczos-3 function using (Equation 10).

However, in the equation (10), the same filter coefficient is applied to all sample points by shifting the sample position horizontally and vertically (by the pixel interval of the low resolution image) by 1⁄4 pixel.
The interpolation unit 20 inputs the interpolation image thus obtained to the blur correction device 1 as the input image M. As a result, from the blur correction device 1, it is possible to obtain a sharp corrected image (high resolution image) H in which the number of pixels is larger than that of the low resolution image L and blur generated in the interpolation unit 20 is corrected. .

  FIG. 5 is an example L1 of the low resolution image L input to the super resolution device 2. FIG. 6 is an example M1 of an image in which the resolution is enhanced by the interpolation unit 20 when the example L1 of the low resolution image L is input to the super resolution device 2. FIG. 7 shows an example H1 of a corrected image (high resolution image) H by the super resolution device 2 when the example L1 of the low resolution image L is input to the super resolution device 2. Although blurring is seen in the edge portion in FIG. 6, sharp results are obtained in FIG. In the correction image H of FIG. 7, the correction value calculation unit 12 uses (Expression 3), and the correction value calculation unit 14 uses (Expression 6), and the constant β is β = 0.75. This is an example of the case of

  In the above-described embodiments, the scanning direction in the correction value computing units 12 and 14 has been described as the x-axis direction and the y-axis direction, but other directions may be used. For example, the scanning direction may be the negative direction of the x axis and the negative direction of the y axis.

  In each of the above-described embodiments, the blur correction device 1 and the super-resolution device 2 that process signals representing an image are exemplified. However, the blur correction device 1 and the super-resolution device 2 may be voices, three-dimensional models Etc. may be processed to represent something other than an image. Also, the signal may be a one-dimensional signal or a signal of two or more dimensions.

  In addition, the program for realizing the functions of the blur correction device 1 in FIG. 1 and the super-resolution device 2 in FIG. 4 is recorded in a computer readable recording medium, and the program recorded in the recording medium is The blur correction device 1 and the super-resolution device 2 may be realized by reading and executing. Here, the “computer system” includes an OS and hardware such as peripheral devices.

  The term "computer-readable recording medium" refers to a storage medium such as a flexible disk, a magneto-optical disk, a ROM, a portable medium such as a ROM or a CD-ROM, or a hard disk built in a computer system. Furthermore, “computer-readable recording medium” dynamically holds a program for a short time, like a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, the volatile memory in the computer system which is the server or the client in that case, and the one that holds the program for a certain period of time is also included. The program may be for realizing a part of the functions described above, or may be realized in combination with the program already recorded in the computer system.

In addition, each functional block of the blur correction device 1 in FIG. 1 and the super resolution device 2 in FIG. 4 may be individually chipped, or a part or all of them may be integrated and chipped. Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. Either hybrid or monolithic may be used. Some of the functions may be implemented by hardware and some may be implemented by software.
Further, when a technology such as integrated circuit substitution for LSI emerges with the advance of semiconductor technology, it is also possible to use an integrated circuit according to the technology.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and design changes and the like within the scope of the present invention are also included.

Reference Signs List 1 blur correction device 2 super resolution device 11 horizontal difference calculation unit 12 correction value calculation unit 13 vertical difference calculation unit 14 correction value calculation unit 15 correction unit 20 interpolation interpolation unit

Claims (4)

What is claimed is: 1. A super-resolution device comprising: an interpolation unit that interpolates an input signal; and a blur correction device that corrects the interpolated input signal,
The interpolation unit shifts the sample position at the time of the interpolation in each of horizontal and vertical directions by 1⁄4 pixel by the pixel interval of the low resolution image by the input signal, and interpolates using a predetermined function. ,
The blur correction device
For each of the plurality of signal positions of the input signal interpolated by the interpolation unit, the sign change of the second-order difference value of the signal value sequence when the signal position is scanned in a predetermined scanning direction, and the signal value sequence A correction pattern determination unit that determines a correction pattern related to signal values around the signal position based on the sign of the first-order difference value;
Wherein the plurality of signal positions each, the correction the correction pattern pattern determination unit has determined, the super-resolution device, characterized by comprising a correction portion for adding the input signal. The correction pattern determined by the correction pattern determination unit includes a correction pattern in which values are arranged in the order of positive value and negative value along the scanning direction, and a positive value and negative value along the reverse direction of the scanning direction. The super-resolution device according to claim 1, further comprising: a correction pattern in which the values are arranged in the order of the values of. The super-resolution device according to claim 1 or 2, wherein the correction pattern determination unit sets the amplitude of the correction pattern to a value corresponding to the magnitude of the first-order difference value. Computer,
Program for functioning as a super-resolution device according to 請 Motomeko 1 to one of any claims 3.