JP7122155B2 - Image super-resolution device and its program, and parameter learning device and its program - Google Patents

Description

The present invention relates to an image super-resolution apparatus and program for increasing the resolution of an image using a convolutional neural network, and a parameter learning apparatus and program for learning parameters of the convolutional neural network used in the image super-resolution apparatus.

Conventionally, as a method to improve the resolution of an image, after performing orthogonal transformation such as wavelet transformation on the input image, the spatial high-frequency spectrum of the high-resolution image is estimated, and inverse orthogonal transformation is performed with the input image. discloses a technique for increasing the resolution of an input image (see Patent Document 1).

This method assumes that there is self-similarity between the input image and the image after resolution enhancement. Generate interpolation.
In this method, the input image is an image obtained by reducing the resolution of the original image in advance, and the spectrum power representative value for each band obtained by octave-decomposing the original image is input from the outside as known information. Alternatively, in this method, on the premise of self-similarity, the spectral power representative value for each band obtained by octave-decomposing the input image is directly doubled in the horizontal and vertical directions to obtain the spectral power representative value for the band.
In this method, the spectrum of the input image and the spatial high-frequency spectrum are corrected to the spectral power representative value input from the outside or the spectral power representative value obtained from the input image for each band.
Then, in this method, an inverse orthogonal transform is performed on the corrected spectrum of the input image and the spatial high-frequency spectrum to generate a high-resolution image.

JP 2012-59138 A

In the conventional method described in Patent Document 1, in order to estimate the spatial high-frequency spectrum, the spectral power representative value for each band obtained by octave decomposition of the original image that generated the input image is used as known information, and the spatial high-frequency spectrum Estimate the spectrum.
However, such information on the original image is not necessarily obtained. Therefore, this method has a problem that a high-resolution image cannot be generated from an image that does not have an original image.

As another method, the conventional method estimates a spatial high-frequency spectrum using spectral power representative values for each band obtained by octave-decomposing an input image on the premise of self-similarity.
However, in this case, the conventional method can estimate the spatial high-frequency spectrum only by adjusting the spectral power. Thus, there is a limit to adjusting the spectrum in fine spatial frequency units only by adjusting the spectral power using the spectral power representative value. Therefore, there has been a demand for higher image quality than the conventional method.

The present invention has been made in view of such problems and demands, and uses a trained convolutional neural network to generate a high-quality, high-resolution image without the original image. It is an object of the present invention to provide an image super-resolution device and its program, and a parameter learning device and its program for learning the parameters of the convolutional neural network.

In order to solve the above-mentioned problems, the image super-resolution device according to the present invention is a low-frequency component that is low-frequency in both the horizontal direction and the vertical direction obtained by wavelet decomposition of an image. An image super-resolution device for improving the resolution of an input image by using a convolutional neural network with pre-learned parameters for estimating high-frequency components both of which are high-frequency components, comprising block segmentation means and convolutional neural network means. , wavelet reconstruction means, and block arrangement means.

In such a configuration, the image super-resolution device sequentially cuts out blocks of a predetermined size from the input image to be increased in resolution by the block cutout means.
Then, the image super-resolution apparatus uses the convolutional neural network means to estimate the high-frequency component corresponding to the block using the convolutional neural network, with the extracted block as the low-frequency component.

Then, the image super-resolution device uses wavelet reconstruction means to perform wavelet reconstruction of the block that is the low-frequency component and the high-frequency component estimated by the convolutional neural network means, and super-resolves the block. to generate This will produce a double resolution image (super-resolution block) in the horizontal and vertical direction of the block.

Then, the image super-resolution device rearranges the super-resolution blocks according to the positions where the blocks are cut out by the block arrangement means. As a result, the image super-resolution device generates a high-resolution image (super-resolution image) in which super-resolution blocks are arranged over the entire image.
Note that the image super-resolution apparatus can be operated with an image super-resolution program for causing a computer to function as each means described above.

Further, in order to solve the above problems, a parameter learning device according to the present invention is a parameter learning device for learning parameters of a convolutional neural network used in an image super-resolution device, comprising: block extraction means; wavelet decomposition means; The configuration includes learning convolutional neural network means and error calculation means.

In such a configuration, the parameter learning device sequentially cuts out from the input image blocks having twice the resolution in the horizontal direction and the vertical direction of the image to be input to the convolutional neural network by the block cutting means.
Then, the parameter learning device uses the wavelet decomposition means to wavelet decompose the block into a low-frequency component that is low-frequency in both the horizontal direction and the vertical direction, and a high-frequency component in either or both of the horizontal direction and the vertical direction. generates high-frequency components.

Then, the parameter learning device inputs the low-frequency component generated by the wavelet decomposition means by the learning convolutional neural network means, and propagates it forward in the convolutional neural network to estimate the high-frequency component.
Further, the parameter learning device uses the error computing means to compute the error between the high frequency component generated by the wavelet decomposition means and the high frequency component estimated by the learning convolutional neural network means.

Then, the parameter learning device uses the convolutional neural network means for learning to propagate the error calculated by the error calculation means in the reverse direction in the convolutional neural network by the error backpropagation method, thereby setting the connection weight coefficient of the convolutional neural network. learn.
Thereby, the parameter learning device learns connection weight coefficients, which are parameters of the convolutional neural network used by the image super-resolution device.
The parameter learning device can operate a computer with a parameter learning program for functioning as each of the means described above.

ADVANTAGE OF THE INVENTION This invention has the outstanding effect shown below.
According to the image super-resolution apparatus of the present invention, a super-resolution image can be generated by synthesizing high-frequency components with respect to an input image using a convolutional neural network. This high-frequency component is obtained by learning high-frequency components of various waveforms. Therefore, the present invention can generate a high-quality super-resolution image because it does not only adjust the power of high-frequency components as in the conventional art.
According to the parameter learning device of the present invention, learning images can be used to learn the parameters of the convolutional neural network used by the image super-resolution device. Therefore, according to the present invention, the image for learning can be changed according to the image targeted by the image super-resolution device, and the convolutional neural network used by the image super-resolution device can be optimized.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic diagram for explaining the outline of the present invention, in which (a) is a diagram showing an outline of processing by an image super-resolution device, and (b) is a diagram showing an outline of processing by a parameter learning device; 1 is a block configuration diagram showing the configuration of an image super-resolution device according to an embodiment of the present invention; FIG. 2 is a block configuration diagram showing the configuration of a wavelet reconstruction means for color images of the image super-resolution device according to the embodiment of the present invention; FIG. 4 is a flow chart showing the operation of the image super-resolution device according to the embodiment of the present invention; 1 is a block configuration diagram showing a specific example (part 1) of an image super-resolution device according to an embodiment of the present invention; FIG. 2 is a block configuration diagram showing a specific example (No. 2) of an image super-resolution device according to an embodiment of the present invention; FIG. 1 is a block configuration diagram showing the configuration of a parameter learning device according to an embodiment of the present invention; FIG. 3 is a block configuration diagram showing the configuration of a wavelet decomposition means for color images of the parameter learning device according to the embodiment of the present invention; FIG. 4 is a flow chart showing the operation of the parameter learning device according to the embodiment of the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<Overview of the invention>
First, an outline of the present invention will be described with reference to FIG. FIG. 1(a) is a diagram showing an overview of the processing of the image super-resolution device 1 (FIG. 2) of the present invention using a convolutional neural network (CNN). FIG. 1(b) is a diagram showing an overview of the processing of the parameter learning device (FIG. 7) for learning CNN parameters used in the image super-resolution device 1 (FIG. 2) of the present invention.

The image super-resolution device 1 (FIG. 2) increases the resolution of an image L (low-resolution image) to an image H (super-resolution image) that is doubled in the horizontal and vertical directions.
As shown in FIG. 1A, the image super-resolution device 1 sequentially cuts out blocks B (for example, 8×8 pixels) of an image L, and converts the blocks B into two-dimensional wavelet decomposition both horizontally and vertically. Assume that the LL image (LL ₁ ) is a region component. In addition, the image super-resolution apparatus 1 uses the CNN that has learned the parameter Pa in advance from the LL image (LL ₁ ) to set the horizontal high-frequency component and the vertical component corresponding to the LL image (LL ₁ ) as three high-frequency components An HL image (HL ₁ ̂) that is a low-frequency component, an LH image (LH ₁ ̂) that has a horizontal low-frequency component and a vertical high-frequency component, and an HH image (HH ₁ ̂) that has both horizontal and vertical high-frequency components ^) is estimated.

Then, the image super-resolution device 1 performs wavelet reconstruction on the LL image (LL ₁ ), the HL image (HL ₁ ̂), the LH image (LH ₁ ̂), and the HH image (HH ₁ ̂). , a super-resolution block S(LL ₀ ̂) corresponding to block B is generated.
In this manner, the image super-resolution apparatus 1 generates a high-resolution (super-resolution) image H from a low-resolution image L by performing resolution enhancement using CNN for each block B. FIG.

The parameter learning device 2 ( FIG. 7 ) learns the CNN parameter Pa used by the image super-resolution device 1 .
As shown in FIG. 1B, the parameter learning device 2 sequentially cuts out blocks E (for example, 16×16 pixels) of the learning image D. As shown in FIG. Then, the parameter learning device 2 converts block E (LL ₀ ′) into LL image (LL ₁ ′), HL image (HL ₁ ′), and LH image (LH ₁ ′) by two-dimensional wavelet decomposition. HH image (HH ₁ ').
Then, the parameter learning device 2 inputs the LL image (LL ₁ ') to the CNN, and outputs the HL image (HL ₁ ̂), the LH image (LH ₁ ̂) and the HH image (HH ₁ ̂), CNN parameter Pa is learned by error backpropagation so as to eliminate the error with the HL image (HL ₁ '), LH image (LH ₁ '), and HH image (HH ₁ ') after wavelet decomposition, which is the correct data. do.

Note that the parameter learning device 2 uses, as the learning image D, an image containing features (patterns, etc.) of an image to be increased in resolution by the image super-resolution device 1. can optimize the CNN used by For example, the parameter learning device 2 may perform parameter learning using, as the learning image D, a low-resolution image L to be increased in resolution.
Also, if a general-purpose image is used as the learning image D, the parameter learning device 2 can learn the CNN used by the image super-resolution device 1 for increasing the resolution of the general-purpose image.
Note that the learning image D does not have to be one, and a plurality of images may be used.
The configurations and operations of the image super-resolution device 1 and the parameter learning device 2 will be described in detail below.

<Configuration of image super-resolution device>
The configuration of the image super-resolution device 1 will be described with reference to FIG. Here, it is assumed that the resolution of the image L input to the image super-resolution apparatus 1 is horizontal A _x pixels and vertical A _y pixels. The resolution of the image H output by the image super-resolution device 1 is assumed to be horizontal 2A _x pixels and vertical 2A _y pixels obtained by doubling the image L in the horizontal and vertical directions.
As shown in FIG. 2 , the image super-resolution device 1 includes block clipping means 10 , block scanning means 11 , convolutional neural network means 12 , wavelet reconstruction means 13 and block placement means 14 .

The block clipping means 10 clips a block which is a partial image of an input image (image L). Hereinafter, the pixel value of the c-th color component at the image coordinates (x, y) of the image L is denoted as L(x, y, c). Here, when the image L is a monochrome image, c = 0, and when the image L is a C primary color image, c is 0 or more and less than C (C is an integer of 2 or more, for example, in the case of an RGB image, C = 3).
The block cutout means 10 cuts out from the image L a block of a rectangular area of horizontal P pixels and vertical Q pixels (P×Q pixels). Here, both P and Q are natural numbers, and P×Q is 2 or more. For example, P=8 and Q=8.

Here, the block cutout means 10 cuts out based on the cutout coordinates (p, q) specified by the block scanning means 11, which will be described later. For example, when the block scanning means 11 designates the cutout coordinates (p, q), the block cutout means 10 sets the image coordinates (p, q) and the image coordinates (p+P−1, q+Q−1) to the diagonal A pixel value string of the image L within the rectangle (including the boundary) with two points is cut out as a partial image (block).
The block extraction means 10 outputs the extracted blocks to the convolutional neural network means 12 and the wavelet reconstruction means 13 .

Note that the block extraction means 10 may normalize the pixel values (coefficient α _c , offset β _c ) for each color component c along with the block extraction.
Specifically, the block cut-out means 10 performs normalization using the following equation (1) to obtain the pixel values (x, y, c) of the block B.

For example, the image L is a color image (C=3) expressed by luminance and color difference, and for c=0 (luminance), the pixel value ranges from 16 to 235, c=1 and c=2 (color difference). , when the pixel values are in the range of 16 to 240, α ₀ =1/219, β ₀ =−16/219, α ₁ =α ₂ =1/224, β ₁ =β ₂ =−16/224 and

The block scanning means 11 sequentially generates cut-out coordinates (p, q) that serve as references for the block cut-out means 10 to cut out blocks. The block scanning means 11 scans the coordinates (p, q).

Also, the binary operator % is defined as a %b that obtains the remainder when a non-negative integer a is divided by a positive integer b. Also, _Bx is the number of cut-out blocks in the horizontal direction.
Alternatively, the block scanning means 11, for example, at time u (u is an integer equal to or greater than 0), according to the following equation (3), P/2 pixels in the horizontal direction and P/2 pixels in the vertical direction so that the clipped images overlap before and after the time. Coordinates (p,q) may be generated in raster scan order at intervals of Q/2 pixels.

The block scanning means 11 outputs the generated cutout coordinates (p, q) to the block cutout means 10 and the block arrangement means 14 .

The convolutional neural network means 12 inputs the blocks cut out by the block cutout means 10 and executes processing by a pre-learned convolutional neural network. The convolutional neural network means 12 generates three channels of blocks having the same number of samples as the input block. That is, the convolutional neural network means 12 outputs the data of the number of samples three times the number of samples of the block.
For example, when a rectangular block of P.times.Q pixels is input from the block extraction means 10, the convolutional neural network means 12 outputs an image of P.times.Q pixels for three channels.
The convolutional neural network means 12 can have, for example, a configuration in which one or more convolution means 120 and one or more activation function application means 121 are alternately connected in cascade.
As shown in FIG. 2, the convolutional neural network means 12 includes L convolution means 120 (120 ₁ , 120 ₂ , . . . , 120 _L ) and L activation function application means 121 (121 ₁ , 121 ₂ , . . . , 121 _L ).

The convolution means 120 performs a convolution operation using a kernel having learned connection weighting coefficients (parameters) of a predetermined size.
The convolution means 120 _i ( _i is an integer of ₁ or more and L or less _{) converts K i types} (T _i ⁽⁰⁾ (r,s,t) to T _i ^(Ki−1) (r,s,t)) and of size P×Q×K _i−1 3rd order tensor I _i−1 (r,s,t) t) is subjected to a convolution operation and output as a third-order tensor J _i (r, s, t) of size P×Q×K _i .
Specifically, the convolution means 120 _i calculates J _i (r, s, t) by the following equation (4).

Note that the tensor T _i ^(k) (ρ, σ, τ) (k is an integer greater than or equal to ⁰ and less than K _i ) is an integer ρ greater than or equal to _ri (0) and less than or equal to _ri ⁽¹ ), _si ⁽⁰⁾ or greater s _i ⁽¹ ) or less, and an integer τ of 0 or more and less than K _i−1 are defined.

In addition, the convolution means 120 _i refers to I _i−1 (r−ρ, s−σ, τ) in Equation (4), r−ρ<0, r−ρ≧P, s−σ <0 or s−σ≧Q (when referring to outside the domain of the tensor), the value is, for example, I _i−1 (r−ρ, s−σ, τ)=0 (zero padding) Use the value defined as Alternatively, the convolution means 120 _i may use the value of the closest element within the domain (zero-order extrapolated value).
For r _i ⁽⁰⁾ , r _i ⁽¹⁾ , s _i ⁽⁰⁾ and s _i ⁽¹⁾ , for example, values defined by the following equations (5) or (6) are used.

For example, if M _i =5, N _i =5, both equations (5) and (6) yield r _i ⁽⁰⁾ =−2, r _i ⁽¹⁾ =+2, s _i ⁽⁰⁾ =-2 and s _i ⁽¹⁾ =+2.
Also, for example, when M _i =4 and N _i =4, according to equation (5), r _i ⁽⁰⁾ =−1, r _i ⁽¹⁾ =+2, s _i ⁽⁰⁾ =−1, s _i ⁽¹⁾ =+2, and according to equation (6), r _i ⁽⁰⁾ =−2, r _i ⁽¹⁾ =+1, s _i ⁽⁰⁾ =−2, s _i ⁽¹⁾ =+1 becomes.

The input to the first-stage convolution means 120 ₁ is a third-order tensor I ₀ (r, s, t) of size P×Q×K _{0 .} defines K ₀ =1, and K ₀ =C for a C-channel color image (where C is the number of primary colors, eg C=3 for a typical color image such as an RGB image).
Also, I ₀ (r, s, t), which is input to the convolution means 120 ₁ , is represented by the following equation (7), the block B (r, s, t).

On the other hand, when the input image _L is a monochrome image, the number of types of convolution calculators (not shown) in the convolution means 120 _L at the final stage is K _L =3, C channels (C is the number of primary colors, typically For a color image with C=3), define K _L =3C

The activation function applying means 121 performs calculation using the activation function on the output of the convolution means 120 .
Activation function application means 121 _i (i is an integer of 1 or more and L or less), as shown in the following equation (8), a third-order tensor of size P×Q×K _i input from convolution means 120 _i To each component of J _i (r, s, t), apply an activation function φ and denote the result of the application as a 3rd order tensor I _i (r, s, t) of size P×Q×K _i Output.

The activation functions φ _i,t applied to each component of tensor J need not all be the same for either i or t, or both, or be the same for all combinations of i and t. It doesn't matter if there is. A typical example is to set the same activation function for all combinations of i and t, or to set an activation function that is not necessarily the same for each i.
For example, for i = 1, 2, . (13)) and the like. A function φ, which is a specific example of the activation function applied by the activation function applying means 121, is shown below.
For example, the function φ can use ReLU shown in the following equation (9).

A sigmoid function shown in the following equation (10) can be used as the function φ.

Also, the function φ can use the hyperbolic tangent function shown in the following equation (11).

Also, the function φ can use the softsign function shown in the following equation (12).

Also, as the function φ, the identity map (without applying the activation function) shown in the following equation (13) may be used.

As shown in this equation (13), when the activation function is not applied to all the components of tensor J, the activation function applying means 121 _i itself may be omitted from the configuration.
Note that the activation function applying means 121 connected to the convolution means 120 other than the convolution means _120L at the final stage is provided with a nonlinear activation function (equation (13) ) should be used.
Activation function application means 121 _L connected to the rear stage of convolution means 120 _L in the final stage has an activation function (for example, the formula Either the hyperbolic tangent function of (11), the soft sine function of formula (12)) is used, the function that does not apply the activation function (formula (13)) is used, or the activation function applying means 121 _L itself is omitted. It shall be.
The convolutional neural network means 12 outputs to the wavelet reconstruction means 13 the operation result _JL of the final stage that has been processed by the convolutional neural network.

FIG. 3 shows the configuration of the wavelet reconstruction means 13 for color images of the image super-resolution device 1 according to the embodiment of the present invention.
The wavelet reconstruction means 13 has a _first wavelet reconstruction means 13-1, a second wavelet reconstruction means _13-2 , and a _third wavelet reconstruction means 13-3 that perform wavelet reconstruction for each color component, Wavelet reconstruction is performed based on the block B cut out by the block cutout means 10 and the data J _L whose number of samples is three times that of the block B calculated by the convolutional neural network means 12, and the super-resolution block S is generated. It is something to do. In addition, below, the pixel value of the color component c at the coordinates (x, y) of the super-resolution block S is expressed as S(x, y, c). However, when the input image L is a monochrome image, the color component c is only c=0. In this case, the wavelet reconstruction means 13 may have a single configuration as shown in FIG.

The wavelet reconstruction means 13 can use any basis function for wavelet reconstruction, but for example, a Haar basis can be used.
For example, when the input image L is a monochrome image and the basis function is the Haar basis, the wavelet reconstruction means 13 converts the block B (r, s, 0) which is the output of the block extraction means 10 and the convolutional neural network means 12 Based on the outputs J _L (r, s, 0), J _L (r, s, 1) and J _L (r, s, 2), the super-resolution block S to generate

Further, for example, when the input image L is a color image and the basis functions are Haar basis, the wavelet reconstruction means 13 generates the super-resolution block S by the following equation (15).

The wavelet reconstruction means 13 outputs the generated super-resolution block S to the block arrangement means 14 .

Based on the cut-out coordinates (p, q) of the block B generated by the block scanning means 11, the block placement means 14 selects the super-resolution block S generated by the wavelet reconstruction means 13 corresponding to the block B. arranged to generate a super-resolution image.

In addition, when the coordinates generated by the block scanning means 11 are the cutout coordinates (p, q) where the blocks do not overlap, the block arrangement means 14 arranges the super-resolution block S according to the cutout coordinates (p, q) to generate a super-resolution image.
Further, when the coordinates generated by the block scanning unit 11 are taken as the cutout coordinates (p, q) where the blocks overlap, the block arrangement unit 14 blends the super-resolution block S according to the cutout coordinates (p, q). Specifically, in the case of cut-out coordinates in which the blocks do not overlap (see the above formula (2)), the block arrangement means 14 uses the following formula (16) to generate a super-resolution image by combining A super-resolution block S is arranged in a super-resolution image H in correspondence with coordinates (p, q) according to scanning by the block scanning means 11 .

Note that when the input image L is a monochrome image, C=1, and the third argument c of the output image H is only c=0.
In addition, in the case of cut coordinates where blocks overlap (see the above formula (3)), the block placement means 14 adds a predetermined weight W _p,q (ρ, σ, c) according to the following formula (17) to , and super-resolution image H is synthesized by blending the overlapping portions.

As shown in equation (17), the block placement means 14 adds spatial weights W _{p , q} is added. Here, one time point before is a time point where the time point u used by the block scanning means 11 in the calculation of the formula (3) is set to u-1. Note that the output image H before scanning is set to all "0" as an initial value.
The weight W _p,q is the product of the horizontal factor W _p,q ^(Hor) and the vertical factor W _p,q ^(Ver) , as shown in the following equations (18) and (19). can be used.

By using the weights of the above equation (18), the block arranging means 14 assigns the maximum weighting to the central portion of the block and the weighting to linearly attenuate the overlapping portion of the block in each of the horizontal and vertical directions. , to blend blocks. This allows the block placement means 14 to make the boundaries between blocks inconspicuous.
When the block scanning unit 11 finishes scanning the input image L, the block arranging unit 14 outputs an output image (super-resolution image) H having four times (horizontal double, vertical double) the resolution of the input image L. can be generated.

By configuring the image super-resolution device 1 as described above, the image super-resolution device 1 generates a high-resolution image (super-resolution image) by a convolutional neural network using pre-learned parameters. be able to.
At this time, even if the input image L is generated by reducing the original image, the image super-resolution apparatus 1 obtains a wavelet-reconstructable spatial high-frequency spectrum for the input image L without referring to the original image. can be estimated and a super-resolution image H can be generated.
Note that the image super-resolution apparatus 1 can be operated by a program (image super-resolution program) for causing a computer to function as each means described above.

<Operation of image super-resolution device>
The operation of the image super-resolution apparatus 1 will be described with reference to FIG. 4 (see also FIG. 2 for the configuration). It is assumed that the connection weighting coefficients of the convolution means 120 of the convolution neural network means 12 are preset with parameters learned by the parameter learning device 2 (FIG. 7).

In step S1, the block scanning means 11 generates coordinates (p, q) as a cut-out position of a block of P×Q pixels in the input image L in the order of raster scanning.
The block cutout position may be a position where the blocks do not overlap or a position where the blocks overlap, and either one of the predetermined cutout positions may be used.

In step S2, the block clipping means 10 clips a block of P×Q pixels from the input image L using the coordinates (p, q) generated in step S1 as the clipping position. The block cut out by the block cutout means 10 corresponds to the LL image (LL ₁ ) of the block B shown in FIG. 1(a).

In step S3, the convolutional neural network means 12 inputs the block cut out in step S2, and performs an operation by a convolutional neural network (CNN) composed of the convolution means 120 and the activation function application means 121. Output data with the number of samples three times the number of samples in the block.
The data output from the convolutional neural network means 12 correspond to the HL image (HL ₁ ̂), LH image (LH ₁ ̂) and HH image (HH ₁ ̂) shown in FIG. 1(a).

In step S4, the wavelet reconstruction means 13 wavelet-reconstructs the block (LL image) cut out in step S2 and the data (HL image, LH image and HH image) generated in step S3, and super-resolution block to generate

In step S5, the block placement unit 14 places the super-resolution block generated in step S4 at coordinates (2p, 2q) on the output image H with respect to the coordinates (p, q) generated in step S1. place in position. In step S1, when the block cutout position is set to a position where the blocks do not overlap, the block placement means 14 places the super-resolution block at the position of the coordinates (2p, 2q) on the output image H as it is. On the other hand, in step S1, when the block cutout position is set to a position where the blocks do not overlap, the block arrangement means 14 places the already arranged super-resolution block at the position of the coordinates (2p, 2q) on the output image H. Blend the overlapping parts.

In step S6, the block scanning means 11 determines whether or not all blocks of the input image L have been scanned.
Here, if all the blocks of the input image L have not been scanned (No in step S6), the image super-resolution apparatus 1 returns to step S1 and continues its operation.

On the other hand, when all blocks of the input image L have been scanned (Yes in step S6), in step S7, the image super-resolution apparatus 1 outputs an output image (super-resolution image) H in which super-resolution blocks are arranged. do.
By the operation described above, the image super-resolution device 1 can generate a high-resolution image (super-resolution image) using a convolutional neural network.

(Concrete example of convolutional neural network)
An example of the convolutional neural network used by the image super-resolution device 1 will now be described.
FIG. 5 is a block configuration diagram showing a specific example of the image super-resolution device ₁ , showing an example of a 5-layer CNN as the convolutional neural network N1. In FIG. 5, Conv. (5, 5, 16) indicates that the convolution means 120 has 16 kinds of 5×5 kernels. ReLU also indicates activation function application means 121 using a normalized linear function.
Conv. (5, 5, 3) indicates that three types of 5×5 kernels are provided. Also, here, an example is shown in which the activation function applying means 121 is not used in the final stage. In addition, Conv. is given as a parameter Pa from the parameter learning device 2 (FIG. 7).

Conv. By using three types of (5, 5, 3) kernels, the convolutional neural network N ₁ has three types of HL images (HL ₁ ^), LH images (LH ₁ ^) and HH images (HH ₁ ^). image.
As a result, the image super-resolution device ₁ uses the block B as an LL image (LL ₁ ), and the HL image (HL ₁ ^), LH image (LH ₁ ^) and HH image ( HH ₁ ̂) and wavelet reconstruction can generate a super-resolution block S(LL ₀ ̂).

FIG. 6 is a block configuration diagram showing another specific example of the image super-resolution device 1, showing an example of an 8-layer CNN as the convolutional neural network _N2 .
In this case, in addition to the _number of layers being different from that of the convolutional neural network N1 in FIG. It is configured.
Thus, the convolutional neural network _N2 may be configured as a ResNet (Residual Network). As a result, it is possible to configure a convolutional neural network with higher estimation accuracy by increasing the depth of layers even with a small number of layers.

<Configuration of parameter learning device>
Next, the configuration of the parameter learning device 2 will be described with reference to FIG. Here, the resolution of the image D input to the parameter learning device 2 is assumed to be horizontal D _x pixels and vertical D _y pixels.
As shown in FIG. 7, the parameter learning device 2 includes block extraction means 20, block scanning means 21, wavelet decomposition means 22, learning convolutional neural network means 23, error calculation means 24, and parameter output means 25. And prepare.

The block cutout means 20 cuts out a block which is a partial image of the input image (image D). Hereinafter, the pixel value of the c-th color component at the image coordinates (x, y) of the image D is denoted as D(x, y, c). Here, when the image D is a monochrome image, c = 0, and when the image D is a C primary color image, c is 0 or more and less than C (C is an integer of 2 or more, for example, in the case of an RGB image, C = 3).

The block cutout means 20 cuts out from the image D a rectangular block of 2P pixels in the horizontal direction and 2Q pixels in the vertical direction (2P×2Q pixels). Here, both P and Q are natural numbers, and P×Q is 2 or more. Note that P and Q are the same as the number of horizontal pixels (P) and the number of vertical pixels (Q) of the block cut out by the block cutout means 10 (FIG. 2) of the image super-resolution device 1 . For example, P=8 and Q=8.

Here, the block cutout means 20 cuts out based on the cutout coordinates (p, q) specified by the block scanning means 21, which will be described later. For example, when the block scanning means 21 designates cut-out coordinates (p, q), the block cut-out means 20 converts the image coordinates (p, q) and the image coordinates (p+2P-1, q+2Q-1) diagonally. A pixel value string of the image D within the rectangle (including the boundary) with two points is cut out as a partial image (block).
The block cutting means 20 outputs the cut blocks to the wavelet decomposition means 22 .

Note that the block extraction unit 20 may normalize the pixel values (coefficient α _c , offset β _c ) for each color component c along with the block extraction.
Specifically, the block cut-out means 20 performs normalization according to the following equation (20) to obtain the pixel values (x, y, c) of the block E.

For example, the image D is a color image (C=3) expressed by luminance and color difference, and for c=0 (luminance), the pixel values are in the range of 16 to 235, c=1 and c=2 (color difference). , when the pixel values are in the range of 16 to 240, α ₀ =1/219, β ₀ =−16/219, α ₁ =α ₂ =1/224, β ₁ =β ₂ =−16/224 and

The block scanning means 21 sequentially generates cut-out coordinates (p, q) that serve as references for the block cut-out means 10 to cut out blocks.
The block scanning means 21 may perform raster scanning within the image D at predetermined pixel intervals, or may generate coordinates (p, q) from random numbers.
When the block scanning means 21 generates coordinates (p, q) from random numbers, for example, p is a uniform random number of 0 or more (D _x −2P) or less, and q is a uniform random number of 0 or more (D _y −2Q) or less. Uniform random numbers. The uniform random numbers may be pseudo-random numbers that approximate them.
The block scanning means 21 outputs the generated clipping coordinates (p, q) to the block clipping means 20 .

FIG. 8 shows the configuration of the wavelet decomposition means 22 for color images of the parameter learning device 2 according to the embodiment of the present invention.
The wavelet decomposition means 22 has a _first wavelet decomposition means 22-1, a _second wavelet decomposition means 22-2, and a _third wavelet decomposition means 22-3 that perform wavelet decomposition for each color component. Input the extracted block and perform wavelet decomposition. Note that when the input image D is a monochrome image, the wavelet decomposition means 22 may have a single configuration as shown in FIG.

By applying two-dimensional wavelet decomposition to the input block, the wavelet decomposition means 22 generates an LL image that is both horizontal and vertical low-frequency components, and an HL image that is horizontal high-frequency components and vertical low-frequency components. , an LH image with horizontal low-frequency components and vertical high-frequency components, and an HH image with both horizontal and vertical high-frequency components. The LL, HL, LH and HH images all have a resolution of P×Q pixels.

The basis functions used for two-dimensional wavelet decomposition are preferably the same (for example, Haar basis) as the basis functions used by the wavelet reconstruction means 13 (FIG. 2) of the image super-resolution apparatus 1 .
For example, when Haar bases are used as the basis functions, the wavelet decomposition means 22 extracts the LL image (LL (r, s, t)), HL Generate images (HL(r,s,t)), LH images (LH(r,s,t)) and HH images (HH(r,s,t)).

However, when the input image D is a monochrome image, C=1.
The wavelet decomposition means 22 outputs the generated LL image to the learning convolutional neural network means 23 and outputs the HL, LH and HH images to the error calculation means 24 .

The learning convolutional neural network means 23 inputs the LL image generated by the wavelet decomposition means 22, and performs the convolutional neural network so that the output is the HL image, the LH image and the HH image generated by the wavelet decomposition means 22. It learns parameters (coupling weight coefficients of kernels).
As shown in FIG. 7, the learning convolutional neural network means 23 includes L convolution means 230 (230 ₁ , 230 ₂ , . . . , 230 _L ) and L activation function application means 231 (231 ₁ , 231 ₂ , . . . , 231 _L ). The convolution means 230 and activation function application means 231 have the same connection configuration as the convolution means 120 and activation function application means 121 of the convolution neural network means 12 ( FIG. 2 ) of the image super-resolution apparatus 1 .

The convolution means 230 performs a convolution operation using connection weight coefficients (parameters) that are successively learned. Further, the convolution means 230 updates the coupling weight coefficients by error backpropagation based on the error input from the latter stage of the convolutional neural network, and propagates the error to the previous stage.
The convolution means 230 _i (i is an integer of 1 or more and L or less) performs a convolution operation on the input of the third order tensor I _i−1 (r, s, t) of size P×Q×K _i−1 . and output as a third-order tensor J _i (r, s, t) of size P×Q×K _i (see equation (4) above).
It is assumed that the size and type of kernel used by the convolution means 230 _i are the same as those used by the convolution means 120 _i (FIG. 2).

here. The input to the first-stage convolution means 230 ₁ is a third-order tensor I ₀ (r, s, t) of size P×Q×K ₀ , where K ₀ is K if the input image D is a monochrome image. Define ₀ = 1 and K ₀ =C for a color image with a C channel (where C is the number of primary colors, eg C=3 for a typical color image such as an RGB image).
Also, the LL image input from the _wavelet decomposition means 22 ( _LL (r, s, t)).

The activation function applying means 231 performs calculation using the activation function on the output of the convolution means 230 . Furthermore, the activation function applying means 231 also propagates the error inputted from the latter stage of the convolutional neural network to the former stage.
Activation function application means 231 _i ( _i is an integer of 1 or more and L or less _{) receives} each To the components, we apply an activation function φ and output the result as a third-order tensor I _i (r, s, t) of size P×Q×K _i . It is assumed that the activation function used by the activation function application means _231i is the same as that used by the activation function application means _121i .

Similarly to the convolutional neural network means 12 (FIG. 2), the learning convolutional neural network means 23 propagates the tensor forward from the convolution means ₂₃₀₁ to the activation function application means _231L to obtain a tensor of size P×Q× Compute the third-order tensor J _L (r, s, t) of 3C. The initial values of the connection weight coefficients (parameters) of the convolution means 230 ₁ to 230 _L are set randomly or intentionally in advance. For example, the initial values of the connection weight coefficients can be generated and set using uniform random numbers or pseudo-random numbers that approximate them.
The learning convolutional neural network means 23 outputs the calculated third order tensor J _L to the error calculation means 24 .

Further, the learning convolutional neural network means 23 repetitively updates the connection weighting coefficients by the error backpropagation method each time an error is successively input from the error computing means 24 . The number of repetitions may be a predetermined number (for example, one million times), or the learning convolutional neural network means 23 monitors the degree of change in the connection weighting coefficients of the convolution means 230 and may fall below a predetermined threshold. Alternatively, the number of repetitions may exceed a predetermined number of times and the degree of change in the connection weighting coefficients may be less than the threshold.
The learning convolutional neural network means 23 outputs the connection weight coefficients of the respective convolution means 230 to the parameter output means 25 after completing the update of the connection weight coefficients by the error backpropagation method (learning completion).

The error computing means 24 computes the errors between the 3rd order tensor J _L computed by the learning convolutional neural network means 23 and the HL, LH and HH images generated by the wavelet decomposition means 22 .
As shown in the following equation (23), the error calculation means 24 calculates the third-order tensor J _L (r, s, t), the HL image (HL (r, s, t)), the LH image (LH (r , s, t)) and the HH image (HH (r, s, t)), an error tensor Δ, which is a third-order tensor value of size P×Q×3C, is calculated, and sent to the learning convolutional neural network means 23 Output.

The parameter output means 25 outputs, as output parameters, the connection weighting coefficients of the respective convolution means 230 output after the learning convolution neural network means 23 completes learning.
The parameters output by the parameter output means 25 are set in the _convolution means 120 (120 ₁ , 120 ₂ , . Thus, the image super-resolution device 1 can be operated in an optimal state.

By configuring the parameter learning device 2 as described above, the parameter learning device 2 can learn the parameters of the convolutional neural network for increasing the resolution of the image by the image super-resolution device 1 .
The parameter learning device 2 can be operated by a program (parameter learning program) for causing a computer to function as each means described above.

<Operation of parameter learning device>
9 (for the configuration, the operation of the parameter learning device 2 will be described with reference to FIG. 7 as needed. Note that the connection weight coefficients of the convolution means 120 of the convolution neural network means 12 are set in advance by the parameter learning device 2 ( It is assumed that the parameters learned by FIG. 7) are set.

In step S10, the block scanning means 21 generates coordinates (p, q) that are positions for cutting out a block of 2P×2Q pixels in the input image D by raster scanning or randomly.

In step S11, the block clipping means 20 clips a block of 2P×2Q pixels from the input image D using the coordinates (p, q) generated in step S10 as the clipping position. The block cut out by the block cutout means 20 corresponds to the LL image (LL ₀ ′) of block E shown in FIG. 1(b).

In step S12, the wavelet decomposition means 22 applies two-dimensional wavelet decomposition to the blocks cut out in step S11. As a result, the wavelet decomposition means 22 extracts, from the blocks, an LL image that is both horizontal and vertical low-frequency components, an HL image that is horizontal high-frequency components and vertical low-frequency components, a horizontal low-frequency component and a vertical high-frequency image. An LH image, which is a component, and an HH image, which is both horizontal and vertical high frequency components, are generated. The images after wavelet decomposition correspond to the LL image (LL ₁ '), HL image (HL ₁ '), LH image (LH ₁ ') and HH image (HH ₁ ') shown in FIG. 1(b).

In step S13, the learning convolutional neural network means 23 inputs the LL image generated in step S12, and performs an operation by a convolutional neural network (CNN) composed of the convolution means 230 and the activation function application means 231. By doing so, the data of the number of samples three times the number of samples of the block is output. The output of this learning convolutional neural network means 23 corresponds to the HL image (HL ₁ ̂), the LH image (LH ₁ ̂), and the HH image (HH ₁ ̂) shown in FIG. 1(b).

In step S14, the error computing means 24 computes the HL image (HL ₁ '), the LH image (LH ₁ '), and the HH image (HH ₁ ') generated by wavelet decomposition in step S12, and the CNN in step S13. Errors with the generated HL image (HL ₁ ̂), LH image (LH ₁ ̂) and HH image (HH ₁ ̂) are calculated.

In step S15, the learning convolutional neural network means 23 updates the connection weight coefficients of the convolutional neural network (CNN) by error backpropagation based on the error calculated in step S14.
In step S16, the learning convolutional neural network means 23 determines whether or not learning has been completed based on a predetermined number of iterations or the like.
Here, if the learning is not completed (No in step S16), the parameter learning device 2 returns to step S10 and continues the operation.

On the other hand, when the learning is completed (Yes in step S16), in step S17, the parameter output means 25 outputs the connection weighting coefficient in the convolution means 230 of the learning convolutional neural network means 23 as an output parameter. Thus, the parameter learning device 2 can learn the parameters of the convolutional neural network used by the image super-resolution device 1 .

Note that parameter learning in the parameter learning device 2 may be performed before the image super-resolution device 1 is manufactured, and parameters after learning may be reflected in the image super-resolution device 1 .
Further, after the image super-resolution device 1 is manufactured, the parameters of the image super-resolution device 1 may be reset by performing parameter learning in the parameter learning device 2 at an appropriate time.
Further, for example, when the parameter learning device 2 performs learning with the same input image as the image super-resolution device 1, at an appropriate time during the operation of the image super-resolution device 1 (for example, each time an input image is input) , the parameter learning device 2 may be operated, and the parameters after learning may be set in the image super-resolution device 1 .

REFERENCE SIGNS LIST 1 image super-resolution device 10 block extraction means 11 block scanning means 12 convolution neural network means 120 convolution means 121 activation function application means 13 wavelet reconstruction means 14 block placement means 2 parameter learning device 20 block extraction means 21 block scanning means 22 wavelet decomposition means 23 learning convolutional neural network means 230 convolution means 231 activation function application means 24 error calculation means 25 parameter output means

Claims (9)

An image super-resolution device that increases the resolution of an input image using a convolutional neural network that estimates high-frequency components of the image from low-frequency components obtained by wavelet decomposition of the image,
a block extraction means for extracting a block of a predetermined size from the input image;
Convolutional neural network means for estimating the high-frequency component corresponding to the block using the convolutional neural network, with the block as the low-frequency component;
wavelet reconstruction means for wavelet reconstruction of the block and the high-frequency component to generate a super-resolution block by super-resolving the block;
a block placement unit that rearranges the super-resolution blocks according to the positions from which the blocks are cut out to generate a super-resolution image for the input image;
An image super-resolution device comprising: 2. The image super-resolution apparatus according to claim 1, wherein said convolutional neural network estimates high-frequency components for said channels from images for one or more channels regarding color. The block cutout means cuts out the blocks for the channels from the images for the channels and inputs them to the convolutional neural network means,
The wavelet reconstruction means generates super-resolution blocks for the channels from the blocks for the channels and high-frequency components for the channels estimated by the convolutional neural network means,
3. The super-resolution image according to claim 2, wherein the block arranging unit arranges the super-resolution blocks for each channel to generate a super-resolution image corresponding to the number of channels. Device. The block clipping means clips the input image so that regions overlap,
4. The image super-resolution according to any one of claims 1 to 3, wherein the block arrangement means generates the super-resolution image by synthesizing overlapping regions of the super-resolution blocks. image device. A parameter learning device for learning parameters of a convolutional neural network used in the image super-resolution device according to any one of claims 1 to 4,
a block extraction means for sequentially extracting blocks having twice the resolution in the horizontal direction and the vertical direction of the input image of the convolutional neural network from the input image;
Wavelet decomposition means for generating a low-frequency component and a high-frequency component by wavelet-decomposing the block;
learning convolutional neural network means for estimating high-frequency components by inputting the low-frequency components and propagating them forward in the convolutional neural network;
an error calculation means for calculating an error between the high frequency component generated by the wavelet decomposition means and the high frequency component estimated by the learning convolutional neural network means;
The learning convolutional neural network means learns the connection weighting coefficient of the convolutional neural network as the parameter by propagating the error backward in the convolutional neural network by an error backpropagation method. Parameter learning device. 6. The parameter learning apparatus according to claim 5, wherein the convolutional neural network estimates high-frequency components for the channels from images for one or more channels regarding color. The block cutout means cuts out blocks for the channel from the image for the channel,
7. The parameter learning apparatus according to claim 6, wherein said wavelet decomposition means generates low frequency components and high frequency components for said channels from blocks for said channels. An image super-resolution program for causing a computer to function as the image super-resolution device according to any one of claims 1 to 4. A parameter learning program for causing a computer to function as the parameter learning device according to any one of claims 5 to 7.

Images