JP6832252B2 - Super-resolution device and program - Google Patents

Description

The present invention relates to super-resolution devices and programs.

As a method of improving the resolution of an image, there is a method of interpolating between pixels. There is also a super-resolution technology that improves resolution while supplementing or restoring signal components that exceed the Nyquist frequency. As super-resolution technology, for example, there are a method based on Bayesian estimation and a method based on machine learning.

In the method based on Bayesian estimation, a high-resolution image to be output is obtained by obtaining a high-resolution image that maximizes the posterior probability based on the product of the likelihood and the prior probability being proportional to the posterior probability. Specifically, for example, the likelihood is defined as quantifying whether or not the result of lowering the resolution of the assumed high-resolution image does not contradict the input low-resolution image. Further, for example, a prior probability is defined as quantifying whether the assumed high-resolution image is a natural image (for example, whether the pixel value is spatially smooth). In the method based on Bayesian inference, a method using the successive Monte Carlo method for the solution has been developed. Patent Document 1 describes a technique for performing Bayesian estimation using the sequential Monte Carlo method.

As a method by machine learning, a method using a convolutional neural network (CNN) has been proposed. In the CNN method, a deep CNN with a low-resolution image on the input side and a high-resolution image on the output side is constructed, and network learning (neural network) is performed using a pair of a low-resolution image and a high-resolution image in advance. (Adjusting the weight of synaptic connections) is the mainstream.
Non-Patent Document 1 describes a method in which CNN is applied to prior probability calculation from a high-resolution image as a method combining machine learning and Bayesian estimation.

Japanese Patent No. 5405389

Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens: “Pixel Recursive Super Resolution,” arXiv: 1702.00783v1 [cs.CV], 2 Feb. 2017.

In the high resolution method by interpolation, since the frequency component exceeding the Nyquist frequency of the input low resolution image is lacking, only a blurred output high resolution image without a sense of fineness can be obtained.

Among the super-resolution techniques, the method based on Bayesian estimation still has a problem in efficient modeling of a high-dimensional space when an image is expressed in a high dimension. In the method of Non-Patent Document 1, a method is used in which the occurrence probability of the pixel value of the target pixel is sequentially reconstructed from the already restored adjacent pixel value sequence by the chain rule of the conditional probability. However, in the chain rule method, image distortion may occur due to modeling and learning errors and pixel scanning order.

The present invention has been made based on the above-mentioned problem recognition, and is a super-resolution device capable of obtaining a visually fine high-resolution image based on an input low-resolution image. It is intended to provide a program.

[1] In order to solve the above problems, the super-resolution device according to one aspect of the present invention is a super-resolution device that obtains a high-resolution image from a low-resolution image by iterative calculation, and is a time point in the iterative calculation. A high-resolution image candidate storage unit that stores high-resolution image candidates corresponding to the above, a posterior probability map storage unit that stores a posterior probability map corresponding to a time point in the iterative calculation, and a prior time point in the iterative calculation. A sample generation unit that probabilistically generates a temporary high resolution image from a high resolution image candidate according to a predetermined probability model, and each local region between the temporary high resolution image and the low resolution image. A likelihood map calculation unit that evaluates the consistency and outputs the spatial distribution of the consistency as a likelihood map, and a space that evaluates the degree to which the prior high resolution image is a natural image and is a natural image. The prior probability map calculation unit that outputs the distribution as a prior probability map and the high resolution image candidate are updated and stored in the high resolution image candidate storage unit, and the posterior probability map is updated and the posterior posterior It includes a sample update unit to be stored in the probability map storage unit, and a representative value calculation unit to determine and output a value at each pixel position of the high-resolution image from the sequence at the time of the high-resolution image candidate. , The sample update unit is the product of the value at each pixel position of the likelihood map output by the likelihood map calculation unit and the value at the corresponding pixel position of the prior probability map output by the prior probability map calculation unit. And, based on the value at the corresponding pixel position of the posterior probability map at the previous time point and the predetermined random number value, the pixel value at the pixel position of the prior high resolution image or the high resolution image at the previous time point. Any of the pixel values of the candidate at the pixel position is determined as the pixel value of the pixel position of the high resolution image candidate at the present time, and the value at each pixel position of the likelihood map output by the likelihood map calculation unit. And the product of the values at the corresponding pixel positions of the prior probability map output by the prior probability map calculation unit, and the value of the same pixel position of the probability map at the corresponding pixel positions of the posterior probability map at the previous time. , The product at the current corresponding pixel position, or the posterior probability map at the previous time point, based on the random number value used to determine the current pixel value of the high resolution image candidate. It is characterized in that any one of the values at the corresponding pixel positions of is determined as the value at the corresponding pixel position of the posterior probability map at the present time.

[2] Further, in one aspect of the present invention, in the above-mentioned super-resolution device, the advance probability map calculation unit is composed of a neural network, and is an example of an image that should be a high-resolution image. It is characterized in that the neural network has been trained in advance using a negative example which is an example of an image which should not be a high-resolution image.

[3] Further, in one aspect of the present invention, in the above-mentioned super-resolution device, the advance probability map calculation unit increases the amount of change in the pixel value in the spatial direction at the pixel position of the provisional high-resolution image as small as possible. It is characterized in that the degree to which the provisional high-resolution image is a natural image is highly evaluated, and the advance probability map is output.

[4] Further, in one aspect of the present invention, in the super-resolution device, the prior probability map calculation unit has a total change amount of pixel values within a predetermined range of pixel positions of the provisional high-resolution image. The smaller the value, the higher the degree to which the provisional high-resolution image is a natural image is evaluated, and the pre-probability map is output.

[5] Further, one aspect of the present invention is a super-resolution device in which a computer obtains a high-resolution image from a low-resolution image by an iterative calculation, and a high-resolution image corresponding to a time point in the iterative calculation. Predetermined from the high-resolution image candidate storage unit that stores the candidates, the posterior probability map storage unit that stores the posterior probability map corresponding to the time point in the iterative calculation, and the high-resolution image candidate at the previous time point in the iterative calculation. According to the probabilistic model of, the sample generator that probabilistically generates a pseudo-high resolution image and the consistency of each local region between the temporary high-resolution image and the low-resolution image are evaluated and the consistency is achieved. A likelihood map calculation unit that outputs the spatial distribution of the above as a likelihood map, and an evaluation of the degree to which the provisional high-resolution image is a natural image, and outputs the spatial distribution of the degree of the natural image as a prior probability map. A sample in which the prior probability map calculation unit and the high-resolution image candidate are updated and stored in the high-resolution image candidate storage unit, and the posterior probability map is updated and stored in the posterior probability map storage unit. The sample update unit includes an update unit and a representative value calculation unit that determines and outputs a value at each pixel position of the high-resolution image from a sequence at the time of the high-resolution image candidate. The product of the value at each pixel position of the likelihood map output by the likelihood map calculation unit and the value at the corresponding pixel position of the prior probability map output by the prior probability map calculation unit, and the post-event of the previous time. Based on the value at the corresponding pixel position of the probability map and the predetermined random number value, the pixel value at the pixel position of the temporary high resolution image or the pixel value at the pixel position of the high resolution image candidate at the previous time point. Is determined as the pixel value of the pixel position of the high-resolution image candidate at the present time, and the value at each pixel position of the likelihood map output by the likelihood map calculation unit and the prior probability map calculation unit. The product of the values at the corresponding pixel positions of the prior probability map output by, the value of the same pixel position of the probability map at the corresponding pixel position of the posterior probability map at the previous time, and the high resolution image at the present time. Based on the random number value used to determine the pixel value of the candidate pixel position, the product at the current corresponding pixel position or the value at the corresponding pixel position of the prior probability map at the previous time. This is a program for functioning as a super-resolution device, which determines one of them as a value at a corresponding pixel position of the prior probability map at the present time.

According to the present invention, a high-definition and reliable high-resolution image can be efficiently obtained based on an input low-resolution image.

It is a block diagram which shows the schematic functional structure of the super-resolution apparatus according to 1st Embodiment of this invention. It is a functional block diagram which shows the schematic structure of the storage part 300 by the same embodiment. It is a schematic diagram which shows the schematic structure of the neural network used by the prior probability map calculation unit 15 by the same embodiment. It is the schematic which shows the example of the positive example and the negative example suitable for use in the learning process of the neural network in the same embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing a schematic functional configuration of the super-resolution device according to the present embodiment. As shown in the figure, the super-resolution device 1 multiplies the delay unit 10, the delay unit 11, the sample generation unit 12, the switching unit 13, the likelihood map calculation unit 14, and the prior probability map calculation unit 15. It is composed of a unit 16, a sample update unit 18, a representative value calculation unit 19, an initialization unit 20, and a storage unit 300. Each of these functional parts is realized by using, for example, an electronic circuit. In addition, the storage unit 300, and if necessary, other functional units include storage means such as a semiconductor memory and a magnetic hard disk device. In addition, each function may be realized by a computer and software.

This super-resolution device 1 is a device that performs Bayesian estimation from a low-resolution image to a high-resolution image by a method applying the solution method by the Metropolis-Hastings method of the Markov chain Monte Carlo method. The super-resolution device 1 obtains a high-resolution image from a low-resolution image by iterative calculation. The "application" of "applying the solution method by the Metropolis-Hastings method" is, in particular, a method configured so that the pixel values of the individual pixels constituting the image are sampled independently and probabilistically. Means. Further, the super-resolution device 1 that performs Bayesian estimation obtains a high-resolution image by the following equation (1).

In the formula (1), Y is an input low-resolution image, and X is a high-resolution image corresponding to the low-resolution image. Further, P (X) is a prior probability indicating whether or not the high-resolution image X is suitable as a high-resolution image. Further, P (Y | X) is a conditional probability (likelihood) indicating the degree to which the high-resolution image X becomes a low-resolution image Y when the high-resolution image X is reduced in resolution. Further, X ^ (hat on X) is a high-resolution image that the super-resolution device 1 should obtain for the input Y. In obtaining the high-resolution image X ^ (hat on top of X) for the input Y, P (Y) is constant without depending on X, and can be treated as a constant.
That is, the super-resolution device 1 obtains a high-resolution image that maximizes the posterior probability P (X | Y) based on the product of the likelihood and the prior probability being proportional to the posterior probability.

The input to the super-resolution device 1 is the input low-resolution image Y. The pixel position in the image is represented by r. The pixel position r is a numerical value for specifying the coordinates in the image. For example, r may be a two-dimensional vector value representing the coordinate value itself, or a one-dimensional index value having a one-to-one correspondence with each sample position of the image. The pixel value of the image Y at the pixel position r is written as Y (r). In the following, for images other than Y (the same applies to the probability map described later), the reference to the pixel value at the pixel position r is represented by adding "(r)" in the same manner. Further, the pixel values of various images may be scalars, or may be vectors representing luminance and the like (luminance, color difference, hue, saturation, etc.) for each color and wavelength.

The output from the super-resolution device 1 is set as the output high-resolution image X ^ (hat on top of X). The super-resolution device 1 derives an output high-resolution image X ^ (a hat on X) by iterative calculation. The super-resolution device 1 stores high-resolution image candidates in the high-resolution image candidate storage unit 302 in the storage unit 300, which will be described later. The t time in the iterative calculation (t is a natural number of 1 or more) represents a high-resolution image candidate after repeating the X _t.

Further, super-resolution device 1, the posterior probability map storage unit 304 in the storage unit 300 described later, an image (this representing the likelihood of the pixel value of each pixel of the high resolution image candidate X _t, posterior probability map ) P _t is stored. The posterior probability map P _t may be a map that records a value proportional to the posterior probability P (X _t (r) | Y (r)) even if the posterior probability itself is not recorded. For example, the simultaneous probabilities P (X _t (r), Y (r)) _{for each pixel value pair of X t and Y for each pixel position, that is, P (X t} (r) | Y (r)) P (Y). (R)) is the posterior probability map P _t
It may be memorized as.

Next, each part constituting the super-resolution device 1 will be described.

Delay unit 10, together with the stores the high resolution image candidates X _t in the high-resolution image candidate storage unit 302, 1 and the time delay (i.e., the t-1 against t) high resolution image candidate X _{t- 1} is read from the high-resolution image candidate storage unit 302 and output. _{The initial value X 0} of the high-resolution image candidate may be any image.
The delay unit 11 stores the posterior probability map P _{t in the posterior probability map storage unit 304, and stores the posterior probability map P t-1} delayed by one time point (that is, t-1 with respect to t) in the posterior probability map storage unit 304. Read from unit 304 and output. The initial value P ₀ of the posterior probability map is a map having an arbitrary value of 0 or less for all pixel positions.

The sample generation unit 12 generates and outputs a temporary high-resolution image X'(also referred to as a "temporary high-resolution image") from the high-resolution _{image candidate X t-1.} That is, a tentative high-resolution image is stochastically generated from the high-resolution image candidates at the previous time point in the iterative operation according to a predetermined probability model.
Specifically, the sample generation unit 12 generates a temporary high-resolution image X'by adding noise to, for example, the high-resolution image candidate X _t-1. For example, when N (μ, Σ) is a normal distribution of the mean value μ and the variance-covariance matrix Σ (dispersion value in the case of one dimension), the sample generation unit 12 uses the following equation (2). Generates a temporary high-resolution image X'.

That is, the sample generation unit 12 adds noise to the high-resolution image candidate, which has an average of zero and the variance-covariance matrix follows a normal distribution of Σ, at each time point and each pixel, thereby tentatively high-resolution image X. 'Generate. Here, Σ is a semi-positive definite matrix (a non-negative real constant in the case of one dimension).

The switching unit 13 selects either the input from the sample generation unit 12 or the input from the initialization unit 20, and outputs the selected signal. For example, the switching unit 13 selects the input from the initialization unit 20 only at the start of the iterative calculation process (first time) of the super-resolution device 1, and selects the input from the sample generation unit 12 at the subsequent times. To do. That is, the switching unit 13 switches between outputting the temporary high-resolution image X'and outputting the initial value thereof. The output from the switching unit 13 is supplied to the likelihood map calculation unit 14 and the prior probability map calculation unit 15.

The likelihood map calculation unit 14 evaluates for each pixel position whether or not the provisional high-resolution image X'(however, its initial value in the initial processing) matches the input low-resolution image Y. The evaluation value is output as an image (likelihood map L). For example, the likelihood map calculation unit 14 reduces the resolution to the resolution of the input low-resolution image Y based on the temporary high-resolution image X', and inputs the temporary high-resolution image X'with the reduced resolution. The likelihood map L is obtained based on the error image with the low resolution image Y. That is, the likelihood map calculation unit 14 evaluates the consistency of each local region between the temporary high resolution image and the low resolution image, and outputs the spatial distribution of the consistency as a likelihood map.
For example, m _x magnification in the horizontal direction with respect to the provisional high-resolution image X 'input low-resolution image Y, and assumed to be m _y times the resolution in the vertical direction. Then, the image whose resolution is reduced to the size of the input low-resolution image Y by inputting the temporary high-resolution image X'is referred to as Y'.
Specifically, for example, the likelihood map calculation unit 14 first obtains an image Y'by performing subsampling according to the following equation (3) based on the provisional high-resolution image X'.

Alternatively, the likelihood map calculation unit 14 subsamples the result of convolving the thinning filter D into the temporary high-resolution image X'instead of subsampling by the equation (3) to obtain the image Y'. You may try to get it. For example, a two-dimensional Lanczos-3 filter may be used as the thinning filter D. In this case, the likelihood map calculation unit 14 performs subsampling according to the following equations (4) and (5).

In equation (4), δ is a Dirac delta function, and R ² is a real two-dimensional vector space. Further, when the pixel position r is outside the image or other than the grid points, it is defined as X'(r) = 0. Equation (5) is a discrete representation of Equation (4).

Further, in the equations (4) and (5), the floor function and the ceiling function are used. Equation (6) below expresses the floor function and returns the maximum integer that does not exceed the real number z. Further, the following equation (7) represents the ceiling function and returns the smallest integer not smaller than the real number z.

When the image Y'(low resolution image based on the tentative high resolution image) is obtained as described above, the likelihood map calculation unit 14 subsequently bases the image Y'and the input low resolution image Y on. The likelihood map L is derived. The resolution of L is the same as the resolution of the provisional high-resolution image X'.

Specifically, for example, the likelihood map calculation unit 14 obtains the likelihood map L by up-converting the error map E obtained based on the error between the input low-resolution image Y and the image Y'. The likelihood map calculation unit 14 obtains the error map E by, for example, the following equation (8).

In equation (8), σ is a positive real constant, for example, σ = 1. In defining the error map E, the smaller the absolute value of the difference between Y (r) and Y'(r), the larger the value of E (r). Equation (8) also defines the error map E in this way.

The likelihood map calculation unit 14 obtains the likelihood map L by up-converting the error map E.
As an example, the likelihood map calculation unit 14 obtains the likelihood map L by the following equation (9).

The floor function described above is also used in equation (9). Equation (9) obtains the likelihood map L by up-converting the error map E by zero-order interpolation.
Alternatively, as another example, the error map E may be up-converted to obtain the likelihood map L by convolving the interpolation filter to obtain the pixel value at the decimal pixel position. For example, the likelihood map calculation unit 14 uses the Lanczos-3 filter as the interpolation filter (referred to as F) and up-converts according to the following equation (10) to obtain the likelihood map L.

The prior probability map calculation unit 15 obtains and outputs the prior probability map R based on the tentative high-resolution image X'. The prior probability map R is a numerical map that quantifies how natural the tentative high-resolution image X'is (the degree of being a natural image) for each pixel. The degree to which an image is natural is also called "image-likeness". That is, the prior probability map calculation unit 15 evaluates the degree to which the temporary high resolution image is a natural image, and outputs the spatial distribution of the degree to which the temporary high resolution image is a natural image as a prior probability map.

In the present embodiment, the natural image is not characterized only by, for example, the degree of change in the pixel value in the spatial direction. As an example, an image of an urban landscape may contain many linear components of a building or the like. At this time, the degree of change in pixel value in the spatial direction (gradient, etc.) may be large in the local region near the straight line, but such an image represents a real landscape and is a natural image. It can be said that the degree is high. It is difficult to properly evaluate whether or not a change in pixel value between pixels in a specific region is a "natural image" simply by analyzing it by simple arithmetic. In this embodiment, the following method is used in order to appropriately evaluate the degree of being a natural image.
That is, the prior probability map calculation unit 15 according to the present embodiment quantifies the degree to which the local pixel value sequence included in the temporary high-resolution image X'is a pattern that easily occurs as a natural image, and preliminarily. The numerical value for each pixel of the probability map R is obtained.

Specifically, the prior probability map calculation unit 15 uses, for example, a discriminator constructed by machine learning to determine whether or not the above-mentioned local pixel value sequence is a pattern that easily occurs as a natural image. Quantify. For example, the prior probability map calculation unit 15 may use a neural network as a discriminator constructed by using the machine learning. As the neural network, a convolutional neural network (CNN) can be typically used. The configuration of the discriminator using the convolutional neural network will be described later with reference to another figure.

The multiplication unit 16 multiplies the likelihood map L obtained by the likelihood map calculation unit 14 and the prior probability map R obtained by the prior probability map calculation unit 15 for each pixel. The multiplication unit 16 outputs an image composed of the result of multiplication for each pixel as a temporary posterior probability map P'(also referred to as a "temporary posterior probability map"). Specifically, the multiplication unit 16 obtains a tentative posterior probability map P'by the following equation (11).

The sample update unit 18 updates the high-resolution image candidate X _t-1 to output the high-resolution image candidate X _t , updates the posterior probability map P _t-1 , and outputs the posterior probability map P _t. ..

Sample updating unit 18, with respect to the high resolution image candidates, based on the ratio between the pixel value P _t-1 of the 'pixel value P' (r) and _{P t-1 P (r)} , the provisional high-resolution It is stochastically determined whether or not the pixel value X'(r) of the image _{is adopted as the pixel value X t (r) of the high-resolution image candidate at time t.} Then, the sample update unit 18 updates the high-resolution image candidate X _t-1 to generate the high-resolution image candidate X _t and outputs it. Therefore, the sample update unit 18 includes the high-resolution image candidate X _{t-1 output from the delay unit 10, the posterior probability map P t-1} output from the delay unit 11, and the provisional output from the switching unit 13. The high-resolution image X'of the above and the temporary posterior probability map P'output from the multiplication unit 16 are acquired.

The sample update unit 18 adopts the above according to the same rules as the selection / rejection of the sample in the solution method by the Metropolis-Hastings method of the Markov chain Monte Carlo method. That is, the sample update unit 18, P _', the _{P t-1,} and on the basis of the sample values _u t (r) according to the uniform distribution U (0, 1), each pixel position r of the high-resolution image candidate _{X t} It is determined _{whether X'(r) is adopted as the value X t} (r) or X'(r) is rejected and _{the value of X t-1} (r) is maintained.

_{For example, the sample update unit 18 determines the pixel value X t} (r) of the high-resolution image candidate for each pixel position r by the following equation (12).

In the equation (12), u t _(r) is (each t) every time, each time (for generating a random number) to the specimen at each pixel position (each r) and a value obtained by. Therefore, the sample update unit 18 uses a random number generator or a pseudo random number generator. Moreover, the uniform distribution U according to (0,1) sampled value u _{t (r),} for example, may be approximated generated by a pseudo-random number generator such as a Mersenne Twister. That is, the uniform distribution U (0,1) may be approximated by a pseudo random number generator or the like.

Further, the sample update unit 18, for posterior probability map, based on the ratio between the pixel value P _t-1 of the 'pixel value P' (r) and _{P t-1 P (r)} , the temporary posterior probability map It is probabilistically determined whether or not the pixel value P'(r) of _{is adopted as the pixel value P t (r) of the posterior probability map at time t.} Specifically, the sample update unit 18 based on the _{P '(r), P t} -1 (r), and uniform distribution according to U (0, 1) sample value _u t (r), below _{The value P t} (r) at each pixel position r of the _{posterior probability map P t} to be output is determined by the equation (13).

_{As the ut} (r) in the equation (13), the numerical value of the _ut (r) generated when the calculation by the equation (12) is performed is used as it is. In other words, once the t and r, and _u t (r) in the _u t (r) and formula (12) in equation (12), the same value.

In summary, the sample update unit 18 updates the high-resolution image candidates, stores the updated high-resolution image candidates in the high-resolution image candidate storage unit, and updates the posterior probability map. The posterior probability map after the update is stored in the posterior probability map storage unit.
In that process, the sample update unit 18 is the product of the value at each pixel position of the likelihood map output by the likelihood map calculation unit 14 and the value at the corresponding pixel position of the prior probability map output by the prior probability map calculation unit 15. Based on (the product output by the multiplication unit 16), the value at the corresponding pixel position of the posterior probability map at the previous time point, and the predetermined random number value, the pixel value or the previous time point at the pixel position of the prior high resolution image. Any one of the pixel values at the pixel position of the high-resolution image candidate is determined as the pixel value at the pixel position of the current high-resolution image candidate.
Further, the sample update unit 18 is the product (multiplication) of the value at each pixel position of the likelihood map output by the likelihood map calculation unit 14 and the value at the corresponding pixel position of the priorior probability map output by the priorior probability map calculation unit 15. (Product output by unit 16), the value of the same pixel position of the probability map at the corresponding pixel position of the posterior probability map at the previous time, and the pixel value of the pixel position of the high resolution image candidate at the present time. Based on the random value used in (the above "predetermined random value" is reused in the processing of the pixel position at the current time), the product at the corresponding pixel position at the present time, or the posterior probability at the previous time. Any of the values at the corresponding pixel positions of the map is determined as the value at the corresponding pixel positions of the current posterior probability map.

The representative value calculation unit 19 calculates a representative value viewed in the time direction for each pixel position r with respect to the _{high-resolution image candidate X t} output in time series from the sample update unit 18, and based on the result. , Generates output high resolution image X ^ (hat on top of X).
_{Specifically, for example, the representative value calculation unit 19 is a pixel value sequence (X t} ) of a high-resolution image between time S and time T (S and T are natural numbers of 1 or more, respectively, and S ≦ T). (R)) The following equation (14) is calculated based on _{t = S, S + 1, S + 2, ..., T-2, T-1,1.}

That is, when the equation (14) is used, the representative value calculation unit 19 performs the pixel value sequence (X _t (r)) _{t = S, S + 1, S + 2 in the time interval from the time S to the time T for each pixel position r. , ..., T-2, T-1,1} Expected values (mean values) are obtained. As a result, the representative value calculation unit 19 determines the output high-resolution image X ^ (hat on top of X).

In the iterative calculation by the super-resolution device 1, the high-resolution image candidate _Xt is strongly influenced by the initial value in the relatively early stage of the iteration. Therefore, it is preferable to set the above time S at a time when the influence of the initial value is sufficiently eliminated. As an example, let S = 100. Further, the high-resolution image candidate _Xt is a sample from the probability density function representing the certainty as a high-resolution image. Therefore, a reliable output high-resolution image X ^ (hat on X) cannot be obtained unless the number of samples (TS + 1) for which the arithmetic mean is taken in the equation (14) is sufficiently large. For example, in order to obtain the expected value from 100 samples, T = 199 may be set when S = 100.

Alternatively, for example, the representative value calculation unit 19 may use the pixel value sequence (X _t (r)) _{t = S, S + 1, S + 2, ..., T-2, T-1 of the high-resolution image for each pixel position r. , 1} may be detected, and the arithmetic mean may be taken after excluding those out-of-range values. As a result, the representative value calculation unit 19 operates to obtain the pixel value X ^ (r) (where "X ^" is a hat on X) of the output high-resolution image. In this case, the representative value calculation unit 19 _{determines whether or not X t} (r) is an out-of-range value based on _{λ t calculated by the following equation (15).}

In the equation (15), μ _{X (r)} and σ _{X (r)} are pixel value strings (X _t (r)) _{t = S, S + 1, S + 2, ..., T-2, T, respectively.} Expected value and standard deviation of _{-1,1. Then, when | λ t} | exceeds a predetermined threshold value θ (for example, θ = 3), the representative value calculation unit 19 _{treats the corresponding X t} (r) as an out-of-region value. As is clear from the calculation process, λ _t is a unique value for each pixel position r.
That is, the representative value calculation unit 19 obtains the output high-resolution image X ^ (hat on X) based on the expected value excluding the out-of-region value by performing the calculation according to the following equation (16).

Further, for example, the representative value calculation unit 19 has a pixel value sequence (X _t (r)) _{t = S, S + 1, S + 2, ..., T-2, T-1 of a high-resolution image for each pixel position r.} The output high-resolution image X ^ (hat on top of X) may be obtained based on the median value (med) and the mode value (mod) of _{1, 1.} For example, when the output high resolution image X ^ (hat on X) is obtained by the median value (med), the representative value calculation unit 19 performs the calculation by the following equation (17).

Further, when the output high resolution image X ^ (hat on top of X) is obtained by the mode, the representative value calculation unit 19 changes the “med” in the equation (17) to “mod” (statistics). (A function that returns the most frequent value).

Further, for example, the representative value calculation unit 19 clusters _{the pixel value sequence (X t} (r)) _{t = S, S + 1, S + 2, ..., T-2, T-1,1 of the high-resolution image candidate.} The process may be performed. In this case, by obtaining the average value of the pixel value strings belonging to the largest cluster among the obtained clusters, the representative value calculation unit 19 calculates the pixel value of the output high-resolution image X ^ (hat on X). obtain. For the clustering process here, for example, the k-means method can be used.

That is, the representative value calculation unit 19 determines and outputs the value at each pixel position of the high-resolution image from the sequence at the time of the high-resolution image candidate.

The initialization unit 20 generates and supplies an initial value of a temporary high-resolution image X'. Specifically, for example, the initialization unit 20 sets the input low-resolution image Y up-converted to the resolution of the temporary high-resolution image X'as the initial value X'. Here, the up-conversion executed by the initialization unit 20 may be, for example, by nearest neighbor interpolation or by a method using an interpolation filter. Examples of methods that use interpolation filters include bilinear interpolation, bicubic interpolation, Lanczos interpolation (for example, Lanczos-3 interpolation and Lanczos-4 interpolation), and a method based on an interpolation filter in which the sinc function is cut off with a finite tap length. Can be used.

The storage unit 300 has a function of storing data at least temporarily in the process in which each unit included in the super-resolution device 1 performs the above-described processing. The details of the storage unit 300 will be described below with reference to FIG.

FIG. 2 is a functional block diagram schematically showing an example of the configuration of the storage unit 300 according to the present embodiment. As shown in the figure, the storage unit 300 includes an input low-resolution image storage unit 301, a high-resolution image candidate storage unit 302, a provisional high-resolution image storage unit 303, an ex post facto probability map storage unit 304, and a provisional post-mortem. It includes a probability map storage unit 305, a prior probability map storage unit 306, a likelihood map storage unit 307, and an output high-resolution image storage unit 320.

The input low resolution image storage unit 301 stores the input low resolution image Y. The input low-resolution image Y stored in the input low-resolution image storage unit 301 is read out by the initialization unit 20 and the likelihood map calculation unit 14.
The high-resolution image candidate storage unit 302 stores the high-resolution image candidate _Xt in chronological order. As described above, the initial value X ₀ of the high-resolution image candidate may be an arbitrary image, and this image X ₀ is appropriately written in the high-resolution image candidate storage unit 302. _{Further, X t} in t ≧ 1 is written by the sample update unit 18. It should be noted that each unit constituting the super-resolution device 1 _{can read X t} with respect to an arbitrary t from the high-resolution image candidate storage unit 302. That is, the high-resolution image candidate storage unit 302 stores the high-resolution image candidate corresponding to the time point in the iterative operation.
The temporary high-resolution image storage unit 303 stores the temporary high-resolution image X'at least temporarily. The tentative high-resolution image X'is written by the sample generation unit 12. However, the initial value of the temporary high-resolution image X'is written by the initialization unit 20. Then, the temporary high-resolution image X'is read out by the likelihood map calculation unit 14, the prior probability map calculation unit 15, and the sample update unit 18.
Posterior probability map storage unit 304, the posterior probability map P _t, least delay unit 11 stores in the period to delay. As described above, the initial value P ₀ of the posterior probability map is a map having an arbitrary value of 0 or less for all pixel positions, and this map P ₀ is appropriately written in the posterior probability map storage unit 304. _{Further, P t} in t ≧ 1 is written by the sample update unit 18. That is, the posterior probability map storage unit 304 stores the posterior probability map corresponding to the time point in the iterative operation.

The temporary posterior probability map storage unit 305 stores the temporary posterior probability map P'at least temporarily. The tentative posterior probability map P'is written in the tentative posterior probability map storage unit 305 by the multiplication unit 16. Further, the temporary posterior probability map P'is read from the temporary posterior probability map storage unit 305 by the sample update unit 18.
The prior probability map storage unit 306 stores the prior probability map R at least temporarily. The prior probability map R is written in the prior probability map storage unit 306 by the prior probability map calculation unit 15. Further, the prior probability map R is read from the prior probability map storage unit 306 by the multiplication unit 16.
The likelihood map storage unit 307 stores the likelihood map L at least temporarily. The likelihood map L is written in the likelihood map storage unit 307 by the likelihood map calculation unit 14. Further, the likelihood map L is read from the likelihood map storage unit 307 by the multiplication unit 16.
The output high-resolution image storage unit 320 stores the output high-resolution image X ^ (hat on top of X) obtained by the representative value calculation unit 19 at least temporarily.
When the super-resolution device 1 is configured to be processed by a downstream processing process immediately after the data is generated, the configuration may not have a storage unit for storing the data. Further, when the output high-resolution image X ^ (hat on X) is generated and immediately output to the outside of the super-resolution device 1, the configuration does not have the output high-resolution image storage unit 320. Good.

Next, the details of the neural network constituting the prior probability map calculation unit 15 will be described.
FIG. 3 is a schematic diagram showing a schematic configuration of a neural network used in this embodiment. As shown in the figure, the neural network that is a part of the prior probability map calculation unit 15 has a layer structure, and has an input layer 30, an intermediate layer 34 (first component), and an intermediate layer 35 (second component). ) And the output layer 40. As a more general configuration, the neural network may include an input layer, an intermediate layer of 0 or more layers, and an output layer. Each layer has a matrix structure corresponding to the pixel arrangement of the image. Values can be set for cells (1 partition) in the matrix. The neuron calculates a weighted sum for the values of a plurality of cells and outputs a signal corresponding to the values. The weight value in the neuron can be adjusted by learning described later.

The connection between each layer is in the form of a convolutional neural network. In the illustrated example, the neuron 32 and the neuron 33 calculate the weighted sum with respect to the pixel value sequence of the local region 31 of the input layer 30 (in this example, a block of 3 pixels × 3 pixels). Then, the neuron 32 and the neuron 33 output the result of applying the activation function (for example, the sigmoid function) to the respective weighted sums. Each neuron output value is set as each component value of the corresponding pixel in the intermediate layer. In this example, each pixel in the intermediate layer holds a two-dimensional value (first and second components). More specifically, the output value 36 (first component) calculated by the neuron 32 based on the pixel value sequence of the local region 31 is set as the component value of the corresponding pixel in the intermediate layer 34 (first component). To. Further, the output value 37 (second component) calculated by the neuron 33 based on the pixel value sequence of the local region 31 is set as the component value of the corresponding pixel in the intermediate layer 35 (second component).

When performing sequential processing for each pixel, the neural network propagates the signal value from the input layer to the intermediate layer while moving the local region in the input layer and the corresponding pixel position in the intermediate layer in the image. Similarly, the signal value is propagated from the upstream intermediate layer to the downstream intermediate layer (if there are a plurality of intermediate layers). The same applies to the propagation of signal values from the intermediate layer to the output layer.

The parameters in each neuron that make up the convolutional neural network are variable. This parameter can be updated during training. This parameter is used as the weight value when calculating the weight sum.

The convolutional neural network uses, for example, a training data set consisting of positive and negative examples, and performs training by an error backpropagation method.
_{As a positive example, for example, a high-resolution learning image X train} , which is a fine image obtained by photographing (or drawing) an actual thing or the like, is used. As a positive example, the learning high-resolution image X _train does not include meaningless noise, a blurry image, an image with coding deterioration exceeding a predetermined reference, and the like. A high-resolution image for learning X _train as a positive example may include a photograph, a video image frame, an illustration, CG (computer graphics), or a line art.
_{Further, a learning prior probability map R train in} which the entire screen has a predetermined value ρ (for example, ρ = 1) is used in correspondence with the learning high-resolution image X _{train as a positive example.}
As a positive example, one or more pairs of _{the above X train} and R _{train are used.}
On the other hand, as a negative example, for example, a learning high-resolution image X _train'which is an image that is not a fine image is used. As a negative example, the learning high-resolution image X _{train'includes} meaningless noise, a blurred image, an image with coding deterioration of a predetermined reference or more, and the like.
Further, in correspondence with the learning high-resolution image X _{train'as a negative example, the learning prior probability map R train' in} which the entire screen has a predetermined value ρ'(for example, ρ'= 0) is used.
As a negative example, one or more pairs of _{the above X train'and} R _{train' are used.}
By performing the training process using such a training data set in advance, the parameters in the neural network are appropriately adjusted. Then, by using the trained neural network, the prior probability map calculation unit 15 evaluates how natural the input temporary high-resolution image X'is as an image as a numerical value for each pixel position. It is possible to output the prior probability map R.

The neural network itself and the learning process itself of the neural network by the error back propagation method can be realized by the existing technology.

FIG. 4 is a schematic diagram showing examples of positive and negative examples suitable for use in the learning process of the neural network according to the present embodiment.
In the illustrated example, a positive sample _is a pair _{(X train1, R} train1) pairs, and _{(X train2, R train2).} X _{train 1} and X _{train 2} are fine and natural images, respectively. Further, R _{train 1} and R _{train 2} are images (all white) in which 1 is retained corresponding to all pixels.
Also, negative cases _are pairs of _{(X train3, R} train3) pairs, and _{(X train4, R train4).} X _{train 3} is a non-fine, very blurry image of a car. Further, X _{train 4} is an image having a pattern that cannot be said to be a natural image. Further, R _{train 3} and R _{train 4} are images (all black) in which 0 is held corresponding to all pixels.

As described above, according to the present embodiment, the likelihood map calculation unit 14 evaluates the consistency between the provisional high-resolution image and the low-resolution image which is the input image. Further, the prior probability map calculation unit 15 evaluates whether or not the provisional high-resolution image has image features that should be as a high-resolution image. Using these two evaluation values (maps), the sample generation unit 12 and the sample update unit 18 probabilistically generate a highly evaluated high-resolution image. This probabilistic generation is a highly efficient sample generation process similar to the Markov chain Monte Carlo method, and the representative value calculation unit 19 obtains the representative value of the stochastically generated high-resolution image group, which is the most probable super. The resolution result can be obtained quickly.

Further, according to the present embodiment, in order to quantify whether or not the prior probability map calculation unit 15 has image features that should be as a high-resolution image, a machine that imitates a neural network of an organism. We have introduced a learning method (neural network). Therefore, the super-resolution device according to the present embodiment is expected to realize response characteristics that approximate the human visual system (visual nervous system of the eye or brain). That is, in the present embodiment, it is possible to realize super-resolution with a visually fine feeling.

Further, according to the present embodiment, a high-resolution image can be obtained with better processing efficiency than the method of the prior art (such as the method described in Non-Patent Document 1).

[Second Embodiment]
Next, the second embodiment will be described. The matters already described in the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described.
The configuration of the super-resolution device 101 according to the present embodiment is such that the prior probability map calculation unit 15 in the functional configuration shown in FIG. 1 is replaced by the prior probability map calculation unit 115.

The prior probability map calculation unit 115 obtains and outputs the prior probability map R based on the tentative high-resolution image X'. The prior probability map R is a numerical map that quantifies how natural the tentative high-resolution image X'is (the degree of being a natural image) for each pixel.
Specifically, the prior probability map calculation unit 115 determines the degree to which the temporary high-resolution image X'is a natural image when the amount of change in the pixel value in the spatial direction (for example, the brightness gradient) is relatively small. Highly evaluated, the pixel value of the prior probability map R is determined.

As an example, the prior probability map calculation unit 115 calculates the prior probability map R by the following equation (18).

The operator nabla included in the exponential function exp on the right side of equation (18) represents the gradient at the pixel position r of the tentative high-resolution image X'. Further, τ is a positive real constant, for example, τ = 1. When the prior probability map R is calculated by the equation (18), the larger the absolute value of the gradient at the pixel position r, the smaller the value of the prior probability map R at the pixel position r.

In the present embodiment, the multiplication unit 16 performs multiplication using the prior probability map R (r) output by the prior probability map calculation unit 115. The processing of each part other than the prior probability map calculation unit 115 is the same as the processing of the corresponding parts in the first embodiment.

[Third Embodiment]
Next, the third embodiment will be described. The matters already explained up to the previous embodiment may be omitted below. Here, the matters peculiar to the present embodiment will be mainly described.
The configuration of the super-resolution device 201 according to the present embodiment is such that the prior probability map calculation unit 15 in the functional configuration shown in FIG. 1 is replaced by the prior probability map calculation unit 215.

The prior probability map calculation unit 215 obtains and outputs the prior probability map R based on the tentative high-resolution image X'. The prior probability map R is a numerical map that quantifies how natural the tentative high-resolution image X'is (the degree of being a natural image) for each pixel.
Specifically, in the prior probability map calculation unit 215, when the total amount of change in the local pixel value (for example, total variation norm) is relatively small, the temporary high-resolution image X'is natural. The pixel value of the prior probability map R is determined by highly evaluating the degree of the image.

As an example, the prior probability map calculation unit 215 calculates the prior probability map R by the following equation (19).

In equation (19), the operator nabla represents the gradient of the tentative high-resolution image X'at the pixel position (r + x), as in equation (18). Further, x is a vector along the pixel plane. That is, the definite integral included in the right side of the equation (19) is for calculating the total variation of the pixel region having a radius ρ or less around the pixel position r. The radius ρ is a positive real constant, and for example, ρ = 2. Further, φ is a positive real constant, and for example, φ = 1. When the prior probability map R is calculated by the equation (19), the larger the absolute value of the gradient at the pixel position r and its surroundings, the smaller the value of the prior probability map R at the pixel position r.

In the present embodiment, the multiplication unit 16 performs multiplication using the prior probability map R (r) output by the prior probability map calculation unit 215. The processing of each part other than the prior probability map calculation unit 215 is the same as the processing of the corresponding parts in the first embodiment.

[Modification example]
In the first embodiment, the second embodiment, and the third embodiment described above, the prior probability map calculation units 15, 115, and 215 have calculated the prior probability map, respectively. As a modification, the index indicating "how natural" may be quantified by combining a plurality of methods by the prior probability map calculation units 15, 115, and 215.
For example, a map based on a weighted average of the prior probability map obtained by the method by the prior probability map calculation unit 15 and the prior probability map obtained by the method by the prior probability map calculation unit 115 may be used.
Further, for example, a map based on a weighted average of the prior probability map obtained by the method by the prior probability map calculation unit 115 and the prior probability map obtained by the method by the prior probability map calculation unit 215 may be used.
Further, for example, a map based on a weighted average of the prior probability map obtained by the method by the prior probability map calculation unit 215 and the prior probability map obtained by the method by the prior probability map calculation unit 15 may be used.
Further, the prior probability map may be calculated by another method.

It should be noted that at least a part of the functions of the super-resolution device in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The "computer-readable recording medium" is a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a DVD-ROM, or a USB memory, or a storage device such as a hard disk built in a computer system. Say that. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above-mentioned program may be for realizing a part of the above-mentioned functions, and may be further realized by combining the above-mentioned functions with a program already recorded in the computer system.

According to at least one embodiment or a modification thereof described above, the super-resolution device can quickly obtain a high-definition and reliable high-resolution image based on the low-resolution image. ..

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and the design and the like within a range not deviating from the gist of the present invention are also included.

The present invention is available in any industry that processes images or video.

1 Super-resolution device 10 Delay unit 11 Delay unit 12 Sample generation unit 13 Switching unit 14 Likelihood map calculation unit 15 Prior probability map calculation unit 16 Multiplication unit 18 Sample update unit 19 Representative value calculation unit 20 Initialization unit 30 Input layer 31 Local region of interest in the input layer 32 Neuron 33 Neuron 34 Intermediate layer (first component)
35 Intermediate layer (second component)
36 Neuron output value (first component)
37 Neuron output value (second component)
38 Local area of interest in the middle layer 39 Neuron 40 Output layer 41 Neuron output value 101 Super-resolution device 115 Priorior probability map calculation unit 201 Priorior probability map calculation unit 215 Priorior probability map calculation unit 300 Storage unit 301 Input low-resolution image storage unit 302 High-resolution image candidate storage unit 303 Temporary high-resolution image storage unit 304 Posterior probability map storage unit 305 Priorior probability map storage unit 306 Prior probability map storage unit 307 Probability map storage unit 320 Output high-resolution image storage unit

Claims (5)

A super-resolution device that obtains a high-resolution image from a low-resolution image by iterative calculation.
A high-resolution image candidate storage unit that stores high-resolution image candidates corresponding to time points in the iterative calculation,
A posterior probability map storage unit that stores a posterior probability map corresponding to a time point in the iterative operation,
A sample generation unit that probabilistically generates a provisional high-resolution image from a high-resolution image candidate at a previous time in the iterative operation according to a predetermined probability model.
A likelihood map calculation unit that evaluates the consistency of each local region between the provisional high-resolution image and the low-resolution image and outputs the spatial distribution of the consistency as a likelihood map.
A prior probability map calculation unit that evaluates the degree to which the temporary high resolution image is a natural image and outputs a spatial distribution of the degree of being a natural image as a prior probability map.
A sample update unit that updates the high-resolution image candidate and stores it in the high-resolution image candidate storage unit, and updates the posterior probability map and stores it in the posterior probability map storage unit.
A representative value calculation unit that determines and outputs a value at each pixel position of the high-resolution image from the sequence at the time of the high-resolution image candidate, and
Equipped with
The sample update unit
The product of the value at each pixel position of the likelihood map output by the likelihood map calculation unit and the value at the corresponding pixel position of the prior probability map output by the advance probability map calculation unit, and the above-mentioned thing at the previous time. Based on the value at the corresponding pixel position of the post-probability map and the predetermined random number value, the pixel value at the pixel position of the provisional high-resolution image or the pixel at the pixel position of the high-resolution image candidate at the previous time point. One of the values is determined as the pixel value of the pixel position of the high-resolution image candidate at the present time, and is also determined.
The product of the value at each pixel position of the likelihood map output by the likelihood map calculation unit and the value at the corresponding pixel position of the advance probability map output by the advance probability map calculation unit, and the above-mentioned thing at the previous time. Based on the value of the same pixel position of the probability map at the corresponding pixel position of the post-probability map and the random value used to determine the pixel value of the pixel position of the high-resolution image candidate at the present time. Either the product at the corresponding pixel position of, or the value at the corresponding pixel position of the posterior probability map at the previous time point is determined as the value at the corresponding pixel position of the posterior probability map at the present time.
A super-resolution device characterized by this. The advance probability map calculation unit is composed of a neural network, and uses a positive example which is an example of an image which should be a high-resolution image and a negative example which is an example of an image which should not be a high-resolution image. The neural network has been trained in advance.
The super-resolution device according to claim 1. The prior probability map calculation unit highly evaluates the degree to which the temporary high resolution image is a natural image as the amount of change in the pixel value in the spatial direction at the pixel position of the temporary high resolution image is small. Output pre-article probability map,
The super-resolution device according to claim 1 or 2. The pre-probability map calculation unit highly evaluates the degree to which the temporary high-resolution image is a natural image as the total amount of change in the pixel values within a predetermined range of the pixel positions of the temporary high-resolution image is smaller. , Output the pre-probability map,
The super-resolution device according to any one of claims 1 to 3, wherein the super-resolution device is characterized. Computer,
A super-resolution device that obtains a high-resolution image from a low-resolution image by iterative calculation.
A high-resolution image candidate storage unit that stores high-resolution image candidates corresponding to time points in the iterative calculation,
A posterior probability map storage unit that stores a posterior probability map corresponding to a time point in the iterative operation,
A sample generation unit that probabilistically generates a provisional high-resolution image from a high-resolution image candidate at a previous time in the iterative operation according to a predetermined probability model.
A likelihood map calculation unit that evaluates the consistency of each local region between the provisional high-resolution image and the low-resolution image and outputs the spatial distribution of the consistency as a likelihood map.
A prior probability map calculation unit that evaluates the degree to which the temporary high resolution image is a natural image and outputs a spatial distribution of the degree of being a natural image as a prior probability map.
A sample update unit that updates the high-resolution image candidate and stores it in the high-resolution image candidate storage unit, and updates the posterior probability map and stores it in the posterior probability map storage unit.
A representative value calculation unit that determines and outputs a value at each pixel position of the high-resolution image from the sequence at the time of the high-resolution image candidate, and
Equipped with
The sample update unit
The product of the value at each pixel position of the likelihood map output by the likelihood map calculation unit and the value at the corresponding pixel position of the prior probability map output by the advance probability map calculation unit, and the above-mentioned thing at the previous time. Based on the value at the corresponding pixel position of the post-probability map and the predetermined random number value, the pixel value at the pixel position of the provisional high-resolution image or the pixel at the pixel position of the high-resolution image candidate at the previous time point. One of the values is determined as the pixel value of the pixel position of the high-resolution image candidate at the present time, and is also determined.
The product of the value at each pixel position of the likelihood map output by the likelihood map calculation unit and the value at the corresponding pixel position of the advance probability map output by the advance probability map calculation unit, and the above-mentioned thing at the previous time. Based on the value of the same pixel position of the probability map at the corresponding pixel position of the post-probability map and the random value used to determine the pixel value of the pixel position of the high-resolution image candidate at the present time. Either the product at the corresponding pixel position of, or the value at the corresponding pixel position of the posterior probability map at the previous time point is determined as the value at the corresponding pixel position of the posterior probability map at the present time.
A program to function as a super-resolution device.