JP2009146231A - Super-resolution method and super-resolution program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a super-resolution method, capable of automatically detecting a shielding object to attain accurate super-resolution. <P>SOLUTION: A shielding pattern is defined as a probability variable which takes a discrete value for each pixel of an observation image, and it is assumed that an observation image that does not correspond to an original image is obtained in an area where a shielding object or the like is present. A probability distribution of the probability variables expressing the shielding patterns is modeled as spin glass, and the shielding object or the like is expressed as a continuous area of an optional shape by reflecting the property of the shielding object or the like in which it is frequently connected in a spatially smooth manner. Since the presence/absence of shielding is stochastically estimated by the probability distribution expressing a certainty factor of estimation (posterior distribution of the shielding patterns when observed), estimation can be performed according to the certainty factor, and a high-resolution image can be estimated further accurately, compared with a method for deterministically identifying a shielding area. Since the estimation of the shielding patterns and the estimation of the high-resolution image have a strong dependency for the respective estimation results, these are simultaneously estimated without treating them by independent estimation methods. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a super-resolution method and a super-resolution program that restores an original image from a degraded image with high accuracy using a plurality of low-resolution images, and in particular, an obstruction is reflected in an observed image. Super solution that can automatically remove a shielding object or the like (hereinafter referred to collectively as a shielding object or a shielding area in this specification). The present invention relates to an imaging method and a super-resolution program.

  Super-resolution is a computer-based image processing method that specifically captures multiple images and integrates the information to generate an image with an increased number of pixels and improved image quality. Means to increase the resolution of the observed image beyond the physical limit inherent in an imaging system composed of a lens such as a camera or video, or a CCD (Charge Coupled Device). In an imaging device such as a camera or a video, the quality of components such as a lens and a CCD determines the resolution of an image to be captured. In addition, due to blur caused by lens aberration, sub-sampling that depends on CCD density, and noise that is mixed in observation, such as CCD shot noise, the captured image is more than the true observation target image (original image). It becomes deteriorated.

  In order to obtain high-resolution images that break through the physical limitations of imaging devices, software super-resolution processing has been studied to obtain high-resolution images that exceed the capabilities of imaging devices. It is known that if a plurality of images obtained by slightly different positions and angles can be prepared, a higher resolution image can be generated by statistically processing the information of the observed images. Yes.

  Depending on the application site, there may be a case where a shield or the like is reflected in the observed image. However, the conventional super-resolution technology as described above is designed on the premise that only the target object is captured consistently in all images, so it can cope with unexpected shielding, etc. It is not possible to do so, and the quality is disturbed by shielding or the like. For example, as a super-resolution method that can cope with an unexpected disturbance, a fast and robust super-resolution method by Farsiu et al. Is known (Non-Patent Document 1). This does not explicitly model the shielding object, but assumes a noise probability distribution based on the L1 norm, which is uniform for each pixel and is robust to the shielding. Therefore, automatic detection of the shielding area is out of scope.

  In general, image processing methods for detecting an occlusion object include a background difference method in which a background image is prepared in advance, an inter-frame difference method in which a moving object is detected without a background image, and an image in which shielding is removed adaptively. There are known robust background estimation methods for estimating. In addition, there is a method in which an image is segmented into a large number of regions by image processing, and a shielding region is labeled manually.

International Publication No. WO2007 / 122838 S. Farsiu, D. Robinson, and P. Milanfar, "Fast and Robust Multiframe Super Resolution," IEEE Trans. Imag. Proc., 13 (10), pp. 1327-1344, 2004.

  The conventional super-resolution technique is based on the premise that only the target object is always captured in all observed images, and it is not assumed that a non-target shielding object or the like is reflected. However, in reality, there are many situations in which a shielding object or the like can be reflected, for example, a cloud in a satellite photograph or a person in a street photograph. Further, the shielding object or the like can vary for each of a plurality of images. In such a situation, the conventional super-resolution technique is affected by a fluctuating shield or the like, and a problem arises that a very high-quality “high-resolution” degree image is output. In order for conventional super-resolution technology to deal with occlusion, etc., it is necessary to specify the occlusion area for each observation image in advance by manual operation by a human or completely independent image processing technique, and to teach super-resolution method. The super-resolution method had to be redesigned to super-resolve except for the taught shielding area.

  In view of the above problems, an object of the present invention is to provide a super-resolution method and a super-resolution program that can automatically detect a shielding object or the like and realize high-precision super-resolution. To do.

In order to achieve the above object, the super-resolution method of the first aspect of the present invention is:
(1) inputting a plurality of low-resolution observation images;
(2) expressing the shielding pattern as a random variable that takes discrete values for each pixel of a plurality of low-resolution observation images and defining the probability distribution;
(3) estimating the posterior distribution of the shielding pattern so that the free energy is minimized;
(4) using the estimated posterior distribution of the shielding pattern, identifying the shielding pattern and excluding it from the low-resolution observation image.

According to the super-resolution method of the first aspect of the present invention, a shield or the like can be automatically detected.
The shielding pattern is defined as a random variable that takes a discrete value for each pixel of the observation image, and it is assumed that an observation image that does not correspond to the original image is obtained in an area where a shielding object or the like exists. Since the probability of the presence or absence of occlusion is estimated probabilistically by the probability distribution (the posterior distribution of the occlusion pattern when an observation is given) The shielding pattern can be estimated with higher accuracy than the method of identifying the.

The super-resolution method according to the second aspect of the present invention is a method for generating a high-resolution image from a plurality of low-resolution observation images related to an original image,
(1) inputting a plurality of low-resolution observation images;
(2) defining a prior distribution of the high resolution image;
(3) inputting a known shielding pattern;
(4) excluding the shielding pattern from the low-resolution observation image;
(5) estimating a posterior distribution of the high resolution image so that free energy is minimized;
(6) A step of generating a high resolution image using the posterior distribution of the high resolution image estimated in the step (5) is provided.

  According to the super-resolution method of the second aspect of the present invention, an observation image that does not correspond to the original image in a region where a shielding object or the like is detected by using a known shielding pattern and automatically detecting the shielding object or the like. Is obtained from the observed image, and the posterior distribution of the high-resolution image is estimated so as to minimize the free energy, thereby realizing high-precision super-resolution.

The super-resolution method according to the third aspect of the present invention is a method for generating a high-resolution image from a plurality of low-resolution observation images related to an original image,
(1) inputting a plurality of low-resolution observation images;
(2) defining a prior distribution of the high resolution image;
(3) expressing the shielding pattern as a random variable that takes discrete values for each pixel of a plurality of low-resolution observation images and defining the probability distribution;
(4) estimating the posterior distribution of the high resolution image and the posterior distribution of the shielding pattern so that the free energy is minimized;
(5) A step of generating a high-resolution image using the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern estimated in step (4) is provided.

  It is a general situation that an observation image can be obtained by adding positional deviation, rotation, blur, low resolution, noise, and the like to the original image. The resolution to restore the original image from the observed image degraded in this way is an irreversible process, so even if the degradation process is known, it is not possible directly, but there are multiple observed images. When the degradation process differs for each observation image, it can be estimated by using a kind of majority vote for the plurality of observation images. In this majority decision, by using a Bayesian estimation method, which is a kind of statistical estimation method, for example, minimization of free energy, super-resolution capable of high-precision resolution is realized.

Here, conventionally, as the model of the observation process, it is assumed that only the movement between frames, the blur, the number of pixels is reduced, and the noise, the super-resolution method of the present invention, in addition to these, We assumed the possibility that there were various shapes of shields in the model.
That is, the shielding pattern is defined as a random variable that takes a discrete value for each pixel of the observation image. That is, it is assumed that the pixel value expected from the original image can be greatly different in the region where the shielding object or the like exists. Judgment of occluded area and non-occluded area is made probabilistically by the probability distribution (the posterior distribution of the occluded pattern when observation is given) expressing the certainty of estimation. As a result, it is possible to generate a high-resolution image with higher accuracy than a method of deterministically identifying the occluded region.

  Here, the probability distribution that takes a discrete value for each pixel of the observed image is preferably a spin glass model. By creating a spin glass model, it is possible to reflect the property that shielding objects are often connected (connected) smoothly and spatially, and the shielding objects are represented as continuous regions of any shape. be able to.

  In the step of estimating the free energy to be minimized, it is preferable to estimate the parameters of the degradation process of the original image so that the free energy is minimized.

  Since it is impossible to strictly calculate the true posterior probability distribution, the true distribution is approximated. The approximation is achieved by bringing the true distribution closer to a distribution (test probability distribution) that can be calculated based on some criterion such as minimization of free energy. By approximately obtaining the posterior distribution of the high-resolution image and the shielding pattern, both the removal of the shielding object and the super-resolution by the estimation of the shielding pattern can be realized automatically and simultaneously. The estimation of the occluded area and non-occluded area and the estimation of the high-resolution image are strongly dependent on each other's estimation results, so they can be estimated at the same time without using an independent estimation method. Contributes to generation.

The super-resolution method of the fourth aspect of the present invention is a method for generating a high-resolution image from a plurality of low-resolution observation images related to an original image,
(1) inputting a plurality of low-resolution observation images;
(2) defining a prior distribution of the high resolution image;
(3) expressing the shielding pattern as a random variable that takes discrete values for each pixel of the plurality of low-resolution observation images and defining the probability distribution;
(4) estimating the posterior distribution of the high resolution image and the posterior distribution of the shielding pattern so that the free energy is minimized;
(5) repeating step (4) until the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern converge;
(6) A step of generating a high resolution image using the posterior distribution of the high resolution image and the posterior distribution of the shielding pattern converged in the step (5) is provided.

  In the super-resolution method of the present invention, the high-resolution image and the shielding pattern are hidden variables, their probability distribution (parameters) and parameters such as positional deviation are optimized with an appropriate cost function (free energy). (Variation EM algorithm) is used. These probability distributions or the parameters of the probability distribution can be optimized simultaneously or alternately. In either case, the parameters are converged so that the free energy for these probability distributions or the parameters of the probability distribution is minimized.

Next, a super-resolution program according to the first aspect of the present invention is stored in a computer.
(1) a procedure for inputting a plurality of low-resolution observation images;
(2) A procedure for expressing a shielding pattern as a random variable having a discrete value for each pixel of a plurality of low-resolution observation images and defining the probability distribution;
(3) a procedure for estimating the posterior distribution of the shielding pattern so that the free energy is minimized;
(4) Using the estimated posterior distribution of the shielding pattern, the procedure for identifying the shielding pattern and excluding it from the low-resolution observation image is executed.

  A program for causing a computer to execute the super-resolution method according to the first aspect of the present invention described above, causing an observation image data file to be read by a computer, and a shielding pattern to be obtained for each pixel of a plurality of low-resolution observation images. The estimation process and the exclusion process are performed using a probability distribution definition file expressed as a random variable.

Next, a super-resolution program according to the second aspect of the present invention is stored in a computer.
(1) a procedure for inputting a plurality of low-resolution observation images;
(2) a procedure for defining a prior distribution of a high resolution image;
(3) a procedure for inputting a known shielding pattern;
(4) a procedure for excluding the shielding pattern from the low-resolution observation image;
(5) a procedure for estimating the posterior distribution of the high resolution image so that the free energy is minimized;
(6) A procedure for generating a high resolution image using the posterior distribution of the high resolution image estimated in the procedure (5),
Is to execute.

  A program for causing a computer to execute the super-resolution method according to the second aspect of the present invention described above, which reads an observed image data file into a computer and estimates a high-resolution image with a known shielding pattern.

Next, a super-resolution program according to the third aspect of the present invention is stored in a computer.
(1) a procedure for inputting a plurality of low-resolution observation images;
(2) a procedure for defining a prior distribution of a high resolution image;
(3) A procedure for expressing a shielding pattern as a random variable that takes discrete values for each pixel of a plurality of low-resolution observation images and defining a probability distribution thereof;
(4) a procedure for estimating the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern so that the free energy is minimized;
(5) A procedure for generating a high-resolution image using the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern estimated in step (4);
Is to execute.

  A program for causing a computer to execute the super-resolution method according to the third aspect of the present invention described above. Shielding pattern is defined as a random variable that takes discrete values for each pixel of the observed image, and it is assumed that an observed image that does not correspond to the original image will be obtained in an area where there is an obstacle, etc. A high-resolution image is estimated by probabilistic estimation using a probability distribution that expresses (the posterior distribution of the shielding pattern when observation is given). Since the estimation of the occlusion pattern and the estimation of the high-resolution image have a strong dependency relationship between the estimation results, they are estimated simultaneously without being handled by an independent estimation method.

Next, a super-resolution program according to a fourth aspect of the present invention is stored in a computer.
(1) a procedure for inputting a plurality of low-resolution observation images;
(2) a procedure for defining a prior distribution of a high resolution image;
(3) a procedure for expressing the shielding pattern as a random variable having a discrete value for each pixel of the plurality of low-resolution observation images and defining the probability distribution;
(4) a procedure for estimating the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern so that the free energy is minimized;
(5) repeating the procedure (4) until the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern converge;
(6) A procedure for generating a high resolution image using the posterior distribution of the high resolution image converged in the procedure (5) and the posterior distribution of the shielding pattern;
Is to execute.

  A program for causing a computer to execute the super-resolution method according to the fourth aspect of the present invention described above. Using high-resolution images and occlusion patterns as hidden variables, processing (variation EM algorithm) that optimizes the appropriate cost function (free energy) of those probability distributions (parameters) and positional deviation parameters To be executed. With respect to the probability distribution or the parameters of the probability distribution, the computer is caused to converge so that the free energy is minimized.

  According to the super-resolution method and the super-resolution program of the present invention, an explicit human shield area teaching method or a super-resolution independent shield detection method is not required, and a shield area and a non-shield area are automatically set. There is an effect that can be estimated. Furthermore, since the judgment of those shielding areas and non-shielding areas is performed probabilistically by the probability distribution (the posterior distribution of the shielding pattern when observation is given) expressing the certainty of estimation, estimation according to the certainty is performed. Therefore, there is an effect that a high resolution image can be generated with higher accuracy. A high-resolution image can also be estimated probabilistically in the form of a posterior probability distribution. For example, by obtaining an average of the probability distribution, a high-resolution image appropriately generated according to shielding can be obtained. In addition, the estimation of occluded areas and non-occluded areas and the estimation of high-resolution images are strongly dependent on each other's estimation results, so they can be estimated simultaneously without being handled by independent estimation methods. Contributes to image generation.

  Hereinafter, embodiments of the super-resolution method and super-resolution program of the present invention will be described with reference to the drawings.

The processing flow of the super-resolution method of the present invention will be described in order with reference to FIG.
(1) First, two or more low-resolution images are input (step S10). Here, the low-resolution image is applied to the original image by positional deviation, rotation, blur, noise, etc., sub-sampling depending on blur and CCD density caused by lens aberration, and CCD shot. It refers to an image that has been degraded by noise such as noise.

(2) Next, the prior distribution of the high-resolution image is defined as a distribution expressing local smoothness (each pixel has a correlation with a neighboring pixel) (step S20). For example, the prior distribution of the image is defined as a Gaussian distribution.

(2) The shielding pattern is defined as a random variable that takes a discrete value for each pixel of the plurality of low-resolution observation images (step S30). It is assumed that a plurality of low-resolution observation images mixed with noise are obtained as observation images due to the physical properties of the imaging device and the same scene is shot, but some shielding or the like occurs there. The shielding or the like may vary for each observation image or may be constant. For example, in a satellite image, a cloud that moves little by little appears as a shield, a landscape when shooting at a tourist spot, a situation where people come and go in front of a building, or a lens has dirt or scratches Etc. are assumed. Whether or not such a shield exists is represented by a binary variable for each pixel of each observation image, and its distribution is represented by a spin glass model.

  The occlusion pattern is defined as a random variable that takes discrete values for each pixel of the observed image, and it is assumed that an observed image that does not correspond to the original image is obtained in the area where the occluded object exists, and the random variable that expresses the occluded pattern By modeling the probability distribution of as a spin glass, it is possible to reflect the property that shielding objects are often connected (connected) smoothly and spatially, and the shielding objects can be continuously connected in any shape. It can be expressed as a region.

Hereinafter, the notation, x is means an original image having a _P number _H pixels _{(P H-dimensional} vector), _{y t} is, t th observation image _{(P L} dimension with _P number _L pixels It means a vector) of, z _t is intended to mean with respect to each pixel of t th observation image shielding pattern representing the presence or absence of blocking at -1 or +1 (P _{L-dimensional} vectors).
Shall T observed images y _t is obtained, representing the set of those observed image by the following expression.

It is assumed that Gaussian noise is added independently for each pixel in observation. The inverse dispersion β _{i of} Gaussian noise added to the pixel i is different depending on the presence or absence of shielding, and is expressed by the following equation. Here, z _ti is intended to indicate shielding by shielding the pixel i of observed image y _t is occurring, when z _ti is +1 means that reliable observations not occur shield, also When z _ti is −1, it means that observation is not reliable due to shielding. Here, β _H > β _L in order to express that the observation noise increases when there is shielding.

Here, the i-th diagonal matrix P _L × P _L of the diagonal elements and beta _i is represented as β (z _tt), probabilistic model of the observed process is expressed by the following equation.

W _t is the linear transformation of the _{P L} × _{P H} representing misalignment at t-th observation, blurring due to PSF, the downscale. Here, since the focus on shield removed, it will hereinafter be handled W _t avoid the complexity of positional displacement estimation as known.

  For the probability model of the shielding pattern, the prior distribution of the hidden variable z is given by the following Boltzmann machine model.

  The energy function E (z, J) is defined by the following equation.

However, i to j indicate that the pixel i and the pixel j are close to each other, and here, the pixels adjacent in the vertical and horizontal directions are close. The self-coupling J _i has an effect of biasing z _i to easily take +1 or −1, and expresses prior knowledge about an assumed shielding size. The inter-spin coupling J _ij represents the correlation between spins. Here, by making the coupling between the pixels i and j in the proximity relationship a positive constant (J _ij > 0), it becomes easy for z _i and z _j in the neighborhood relationship to have the same value, and the shielding is space. To express prior knowledge that is caused by continuous objects.

(3) Next, the posterior distribution of the high-resolution image, the posterior distribution of the shielding pattern, and the parameters of the degradation process of the original image are estimated (step S40). To directly calculate the posterior distribution, a sum of 2 ^Po terms must be obtained, but this is not possible in a realistic time.
The present invention uses a variational EM (Expectation-Maximization) algorithm in which the test distribution is in a form that is easy to calculate in order to calculate in a realistic time.
The variational EM algorithm is a calculation procedure for obtaining an approximate posterior distribution (test probability distribution) and a maximum likelihood parameter, and is an algorithm for optimizing free energy with respect to the test probability distribution and parameters. Using the variational learning algorithm, even if it is difficult to calculate the posterior distribution directly, it is possible to efficiently approximate by calculating the free energy assuming a test probability distribution that makes the calculation easier. Posterior distributions and maximum likelihood parameters can be obtained.

The variational EM algorithm is formulated as alternating minimization of free energy F [q, θ]. The free energy F [q, θ] is a functional of the probability density function q and a function of the model parameter θ, and is defined as the following equation.

In executing the variational EM algorithm, first, an initial value θ ⁰ of an appropriate model parameter is prepared. The calculation in the l (> 1) iteration of the variational EM algorithm consists of two steps of E step and M step represented by the following equation. The calculation of E step and M step is repeated until convergence, and θ ^l at the time of convergence is used as an estimated value of the model parameter (step S50).

  As explained above, the estimation based on the certainty is performed by probabilistically estimating the presence / absence of the occlusion by the probability distribution (the posterior distribution of the occlusion pattern when the observation is given) expressing the certainty of the estimation. As a result, it is possible to estimate a high-resolution image with higher accuracy than the method of deterministically identifying the occluded region. Occlusion pattern estimation and high-resolution image estimation are highly dependent on each other's estimation results, so they can be estimated at the same time without using an independent estimation method, which contributes to high-resolution image generation with high accuracy. It will be.

  The estimation of the posterior distribution of the high-resolution image and the shielding pattern in step S40 includes a sum composed of terms in the exponent order, and is difficult to calculate in a realistic time. Therefore, in the present invention, the variational EM algorithm can be executed by limiting the test distribution q to a simple form that allows calculation of free energy, instead of taking an arbitrary probability distribution. That is, the test distribution q (z, x) is a distribution represented by the following equation that is factorized independently for each dimension.

The test distribution defined by the above equation is independent of the high-resolution image x and the hidden variable z _ti that represents the presence or absence of shielding of each pixel of each observation image.

  Here, the optimum test distribution that minimizes the free energy is given as follows.

  However, the parameters of each optimum test distribution are as follows.

(4) Finally, in the flow shown in FIG. 1, a high-resolution image is generated using the estimated posterior distribution of the high-resolution image, the posterior distribution of the shielding pattern, and parameters (step S60).
Thereby, it is possible to restore a high-precision and high-resolution image from which the influence of shielding or the like has been removed.

  In the embodiment shown below, in order to verify the usefulness of automatically detecting a shielding pattern, which is a feature of this super-resolution method, a comparison experiment with a super-resolution method that does not assume shielding is performed.

As the original image, a Lenna standard image having a size of 512 × 512 shown in FIG. 2 is used. The observed image is artificially misaligned, blurred, downscaled (reduced to 1/4 horizontal and vertical), and given normal noise (shielding pattern) to the original image (Fig. 2). Fifteen low-resolution images of the size are generated. One of the 15 observation images is shown in FIG. The magnitude of noise differs for each pixel, and is changed between black (defined as a high noise region) and white (defined as a low noise region) according to the shielding pattern shown in FIG. Specifically, the shift amount is determined by a random number from a uniform distribution between −2 pixels and +2 pixels in both the vertical and horizontal directions, and a uniform distribution of −4 / (180π) radians to 4 / (180π) radians. The amount of rotation is determined by a random number from. A Gaussian PSF (PSF: Point Spread function) with a standard deviation of 2 (pixels) is used as a blur due to the lens, and then down-scale is applied so that the magnification rate r becomes r = 4. A hidden variable z _t representing presence / absence is generated as shown in FIG. 4, and an observation noise of 10 dB is added when z _ti = −1 and 40 dB is added when z _ti = + 1.

In FIG. 4, black represents a pixel with z _ti = −1, and white represents a pixel with z _ti = + 1. With respect to the observation image generated in this manner, a 512 × 512 high-resolution image was estimated using the super-resolution method (Ours) of the present invention. This estimation result and the result when uniform intensity observation noise is on the image without introducing the hidden variable z (Uniform), simply taking the average of the observation image and performing bilinear interpolation We compared the results (Obs. Mean). Here, parameters other than the estimated variables x and z are known.
As an evaluation standard for an output image by each method, a peak signal-to-noise ratio (PSNR) defined by the following equation is used.

Here, M is the maximum value of the pixel value, x is the true high resolution images, P _H is the number of pixels of the image. PSNR is a negative logarithm of the mean square error. The larger the PSNR, the better the performance. If the PSNR is different by 1 dB, the mean square error is 10 times different.
The mask pattern output by this super-resolution method is shown in FIG. It can be understood that the true mask pattern can be reproduced fairly accurately. The estimated image is shown in FIG. FIG. 7 is a partially enlarged view of the vicinity of the “right eye” in FIG.

  The upper center (FIG. 6 (2)) is the average value of 15 observation images. The upper right (FIG. 6 (3)) is a high-resolution image output by this super-resolution method. The lower three images (FIGS. 6 (4) to (6)) are high-resolution images output by a method that assumes uniform noise for all pixels without assuming occlusion. The strength of is different. The lower left (FIG. 6 (4)) is an output assuming the noise amount (high) in the shielding area, and the lower right (FIG. 6 (6)) is the output assuming the noise amount (low) in the non-shielding area. 6 (5)) is an output when the amount of noise is determined so that the PSNR with the true image is minimized.

  Compared with a partially enlarged view of the vicinity of the “right eye” in FIG. 7, the output by this super-resolution method (FIG. 7 (3)) is the output of the optimal uniform noise method (FIG. 7 (5)). In comparison, a high PSNR (and hence a low MSE) of 3.32 dB (= 31.38−28.06) is realized, and higher-definition image output can be performed by efficiently using information in a non-shielded area. Moreover, even when visually observed, the output by this method is a high-resolution image with sharper and less noise than the uniform method.

  In addition, Uniform's result assuming that uniform observation noise is present was obtained by changing the assumed observation noise intensity from 10 dB to 40 dB in various ways. However, as shown in FIG. 8, the PSNR is 29.51 at the maximum. Stayed in dB. From this, it can be seen that high-definition image output is possible by using the information of the shielding pattern that the noise is different for each pixel as in the super-resolution method of the present invention.

  In the second embodiment, an example of super-resolution in a situation where the shielding pattern is fixed will be shown. That is, in Example 1, a plurality of low-resolution images were generated using different shielding patterns, whereas in Example 2, a plurality of low-resolution images were generated using the same shielding pattern. I prepared something. The same shielding pattern is a model such as a case where a defect occurs in the CCD element of the camera.

  As the original image, a Lenna standard image having a size of 512 × 512 shown in FIG. As the observed image, the same shielding pattern as shown in FIG. 9 is given to the original image (FIG. 2), and 15 low-resolution images of 128 × 128 size are generated (see FIG. 10). The shielding pattern in FIG. 9 is represented in black (defined as a high noise region) and white (defined as a low noise region), as in the first embodiment.

Using the super-resolution method of the present invention, 512 × 512 images that were quadrupled in length and width were estimated from the 15 low-resolution images of 128 × 128 size. As an evaluation criterion for the output image, the peak signal-to-noise ratio (PSNR) was used as in the first embodiment.
FIG. 11 shows an image estimated by this super-resolution method, and FIG. 12 shows an estimated mask pattern. It can be understood that the estimated mask pattern (FIG. 12B) can reproduce the true mask pattern (FIG. 12A) fairly accurately. FIG. 13 is a partially enlarged view of the vicinity of the “right eye” in FIG.

  The upper center (FIG. 11 (2)) is the average value of 15 observation images. The upper right (FIG. 11 (3)) is a high-resolution image output by this super-resolution method. The lower three images (FIGS. 11 (4) to (6)) are high-resolution images that are output by a method that assumes uniform noise for all pixels without assuming occlusion. The strength of is different. The lower left (FIG. 11 (4)) is an output assuming the noise amount (high) in the shielding area, and the lower right (FIG. 11 (6)) is the output assuming the noise amount (low) in the non-shielding area. 11 (5)) is an output when the amount of noise is determined so that the PSNR with the true image is minimized.

  Compared with FIG. 11, the output by this super-resolution method (FIG. 11 (3)) is only 1.20 dB (= 30.06-28.86) compared with the output of the optimal uniform noise method (FIG. 11 (5)). A high PSNR (and hence a low MSE) is realized, and a higher definition image output can be performed by efficiently using information of an unshielded area. Moreover, even when visually observed, the output by this method is a high-resolution image with sharper and less noise than the uniform method.

Further, when comparing the vicinity of the “right eye” with FIG. 13 partially enlarged, the output by this super-resolution method (FIG. 13 (3)) is the output of the optimum uniform noise method (FIG. 13 (5)). In comparison, a high PSNR (and hence a low MSE) of 3.47 dB (= 31.58−28.10) is realized, and higher-definition image output can be performed by efficiently using information in a non-shielded area. Moreover, even when visually observed, the output by this method is a high-resolution image with sharper and less noise than the uniform method.
Note that the enlarged portion is an unshielded portion, and the PSNR of the output (FIG. 13 (6)) assuming a uniform noise amount (low) in the non-shielded region for all pixels without shielding is assumed. The value is also increasing.

Next, in the third embodiment, an example will be described in which an artificial noise is not added but an estimation is performed using an image captured by an actual camera. This embodiment is considered to be more suitable for real problems.
In the third embodiment, a case where a person crosses from the right to the left in front of the camera lens in a situation where the book shelf is photographed by the camera is considered. Here, a person becomes a shield. Although the figure of a person becomes a shielding pattern, this shielding pattern has a movement, and the pattern shape also changes somewhat (the walking state changes from moment to moment).
Here, the background image of the bookshelf is estimated. That is, an image obtained by removing a shield (person's figure) from an observation image that a person is crossing is estimated. FIG. 14 shows nine observation images crossed by a person. When the shielding pattern is estimated by the super-resolution method, a shielding pattern as shown in FIG. 15 is estimated for each observation image.

FIG. 16A is an image of the bookshelf estimated by the super-resolution method. For comparison, FIG. 16 (3) shows an image (observation average image) estimated from the average value of nine observation images. In the image of FIG. 16 (3), a so-called ghost image (an afterimage of the shielding object) appears on the center right side. FIG. 16 (4) shows an image (observed median image) estimated by using robustness and assuming noise uniformly. It can be confirmed that the books on the bookshelf are slightly blurred.
As can be seen from this example, by using this resolution method, it is possible to output information with higher definition by efficiently using the information of the unshielded area, and output even by visual observation. The image is a high-resolution image with sharper and less noise than other methods.

  The super-resolution method and super-resolution program of the present invention eliminates the reflection of a moving person from a photograph image taken at a sightseeing spot and restores only the target background object with high definition. It is related to the restoration of various images, from shooting with the digital camera used, to large-scale problems such as removing the cloud image mixed in the satellite image and restoring only the satellite image of the city with high definition. Applications in the industrial field are expected.

Flowchart of the super-resolution method of the present invention 512x512 Lenna standard image of the original image An example of a 128 x 128 observation image True shielding pattern Estimated shielding pattern Explanatory drawing for comparison of the method of automatically estimating the shield (the present invention) and the method that does not assume the shield Partial enlarged image near the right eye A graph showing the result (Uniform) when an observation noise of uniform strength appears on the image Shielding pattern in Example 2 15 observation images having a size of 128 × 128 in the second embodiment Explanatory drawing for the comparison of the method (invention) which estimates a shield automatically in Example 2, and the method which does not assume a shield The shielding pattern estimated as the true shielding pattern in Example 2 Partial enlarged image near the “right eye” in the second embodiment Observation image in Example 3 The estimated shielding pattern in Example 3 Explanatory drawing for the comparison of the method (this invention) and the other method of estimating the obstruction automatically in Example 3

Claims (12)

(1) inputting a plurality of low-resolution observation images;
(2) expressing the shielding pattern as a random variable that takes discrete values for each pixel of a plurality of low-resolution observation images and defining the probability distribution;
(3) estimating the posterior distribution of the shielding pattern so that the free energy is minimized;
(4) A step of identifying a shielding pattern using the estimated posterior distribution of the shielding pattern and excluding it from the low-resolution observation image. A method for generating a high-resolution image from a plurality of low-resolution observation images related to an original image,
(1) inputting a plurality of low-resolution observation images;
(2) defining a prior distribution of the high resolution image;
(3) inputting a known shielding pattern;
(4) excluding the shielding pattern from the low-resolution observation image;
(5) estimating a posterior distribution of the high resolution image so that free energy is minimized;
(6) A super-resolution method comprising: generating a high-resolution image using the posterior distribution of the high-resolution image estimated in step (5). A method for generating a high-resolution image from a plurality of low-resolution observation images related to an original image,
(1) inputting a plurality of low-resolution observation images;
(2) defining a prior distribution of the high resolution image;
(3) expressing the shielding pattern as a random variable that takes discrete values for each pixel of a plurality of low-resolution observation images and defining the probability distribution;
(4) estimating the posterior distribution of the high resolution image and the posterior distribution of the shielding pattern so that the free energy is minimized;
(5) A step of generating a high-resolution image using the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern estimated in the step (4). . A method for generating a high-resolution image from a plurality of low-resolution observation images related to an original image,
(1) inputting a plurality of low-resolution observation images;
(2) defining a prior distribution of the high resolution image;
(3) expressing the shielding pattern as a random variable that takes discrete values for each pixel of the plurality of low-resolution observation images and defining the probability distribution;
(4) estimating the posterior distribution of the high resolution image and the posterior distribution of the shielding pattern so that the free energy is minimized;
(5) repeating step (4) until the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern converge;
(6) A super-resolution method comprising: a step of generating a high-resolution image using the posterior distribution of the high-resolution image converged in the step (5) and the posterior distribution of the shielding pattern. .   5. The super-resolution method according to claim 3, wherein, in the step (3), the random variable and the probability distribution taking discrete values for each pixel of the observation image are a spin glass model.   5. The estimation according to claim 3, wherein in the step of estimating the free energy to be minimized, the parameter of the degradation process of the original image is also estimated so that the free energy is minimized. Super-resolution method. On the computer,
(1) a procedure for inputting a plurality of low-resolution observation images;
(2) A procedure for expressing a shielding pattern as a random variable having a discrete value for each pixel of a plurality of low-resolution observation images and defining the probability distribution;
(3) a procedure for estimating the posterior distribution of the shielding pattern so that the free energy is minimized;
(4) Using the estimated posterior distribution of the shielding pattern, identifying the shielding pattern and excluding it from the low-resolution observation image;
Super-resolution program for running On the computer,
(1) a procedure for inputting a plurality of low-resolution observation images;
(2) a procedure for defining a prior distribution of a high resolution image;
(3) a procedure for inputting a known shielding pattern;
(4) a procedure for excluding the shielding pattern from the low-resolution observation image;
(5) a procedure for estimating the posterior distribution of the high resolution image so that the free energy is minimized;
(6) A procedure for generating a high resolution image using the posterior distribution of the high resolution image estimated in the procedure (5),
Super-resolution program for running On the computer,
(1) a procedure for inputting a plurality of low-resolution observation images;
(2) a procedure for defining a prior distribution of a high resolution image;
(3) A procedure for expressing a shielding pattern as a random variable that takes discrete values for each pixel of a plurality of low-resolution observation images and defining a probability distribution thereof;
(4) a procedure for estimating the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern so that the free energy is minimized;
(5) A procedure for generating a high-resolution image using the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern estimated in step (4);
Super-resolution program for running On the computer,
(1) a procedure for inputting a plurality of low-resolution observation images;
(2) a procedure for defining a prior distribution of a high resolution image;
(3) a procedure for expressing the shielding pattern as a random variable having a discrete value for each pixel of the plurality of low-resolution observation images and defining the probability distribution;
(4) a procedure for estimating the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern so that the free energy is minimized;
(5) repeating the procedure (4) until the posterior distribution of the high-resolution image and the posterior distribution of the shielding pattern converge;
(6) A procedure for generating a high resolution image using the posterior distribution of the high resolution image converged in the procedure (5) and the posterior distribution of the shielding pattern;
Super-resolution program for running   The super-resolution program according to claim 9 or 10, wherein the random variable and probability distribution that take discrete values for each pixel of the observation image in the procedure (3) are a spin glass model.   11. The method according to claim 9, wherein in the procedure for estimating the free energy to be minimized, the parameter of the degradation process of the original image is also estimated so that the free energy is minimized. Super resolution program.