JP2013175003A - Psf estimation method, restoration method of deteriorated image by using method, program recording them, and computer device executing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that: in restoring an image deteriorated due to camera shake, it is important to estimate a PSF which is a locus of the camera shake, and conventionally there is a method for determining the PSF by performing pixel search from a cepstrum of a deteriorated image or the camera shake vector of a deteriorated image is obtained by operation of an operator.SOLUTION: A pseudo deteriorated image obtained by acting a candidate PSF to a template image is generated for a cepstrum of a deteriorated image, and a PSF is estimated by evolving the candidate PSF toward a true PSF by using a genetic algorithm with the similarity between the cepstrum of the pseudo deteriorated image and that of the deteriorated image as a degree of candidate PSF application to restore the deteriorated image.

Description

  The present invention relates to a technique for restoring an image deteriorated due to camera shake.

  In an apparatus such as an analog camera or a digital camera that captures a still image by pressing the shutter, the camera itself moves when the shutter is pressed unless a fixing jig such as a tripod is used. Then, deterioration called “blur” due to camera shake occurs in the image. As a method for compensating for this deterioration at the time of photographing, a method is known in which the movement of a camera shake is detected and the optical system of the camera is moved based on the detection signal (Patent Document 1).

  However, such a method requires a complicated system for the camera itself. Further, it is not possible to restore an image that has been deteriorated due to camera shake as a result of photographing.

  On the other hand, methods for estimating and correcting deterioration due to camera shake from the image itself are also being studied. Patent Document 2 discloses a camera shake image correction method that estimates a camera shake vector from a captured still image and performs blur correction according to the camera shake vector.

  An image deteriorated by camera shake can be considered as an image generated as a result of camera shake when an original image is taken. In this case, restoration of an image deteriorated due to camera shake is considered to obtain a motion (PSF) caused by camera shake and to apply an action opposite to the action caused by camera shake on the deteriorated image.

  The concept of the camera shake image correction method disclosed in Patent Document 2 is the same as described above. However, in the method of Patent Document 2, an operation by an operator is required in the step of determining the direction and size of a vector for obtaining a camera shake vector. That is, the skill level of the operator occupies a great deal of weight for restoring the degraded image.

  With the widespread use of digital cameras and camera-equipped mobile phones, the environment for handling image data tends to become more general in society. Considering that it is difficult to avoid camera shake when using a normal camera, it is necessary to more automatically correct a deteriorated image due to camera shake after shooting. Therefore, there is a need for a correction method that does not involve human manipulation.

  Non-Patent Document 1 discloses a method for estimating a PSF representing a camera shake locus from a deteriorated image by a calculation process. If the PSF can be estimated by a calculation process from the deteriorated image, the deteriorated image can be corrected without any manual operation. Non-Patent Document 1 considers that “a cepstrum of a degraded image (inverse Fourier transform of logarithmic spectrum) represents the external shape of the PSF” and proposes a method of determining a PSF by performing pixel search from the cepstrum of a degraded image.

JP-A-1-264372 JP 2000-298300 A

“Image stabilization using cepstrum of degraded images” Yuji Koyamada, Hideo Saito, Koji Otagaki, Mitsuru Eguchi Information Processing Society of Japan Research Report 2008-CVIM-164 (24)

  In Non-Patent Document 1, it can be said that automating of camera shake correction is promoted in that the PSF is obtained from the cepstrum of the deteriorated image. However, since the cepstrum is affected by the logarithm in the frequency domain, not only pixel search for determining the PSF but also correction by a correction value is required. Further, since the cepstrum itself is a blurred image, it is not easy to obtain the PSF by this method.

  The present invention has been conceived in view of the above problems, and if images having similar frequency characteristics are considered to be the same, the cepstrum is considered to be the same regardless of the content of the image, and the same cepstrum as the deteriorated image is generated. A PSF estimation method for obtaining a PSF using a genetic algorithm and a degraded image restoration method for restoring a degraded image from the estimated PSF are provided.

More specifically, the PSF estimation method of the present invention includes:
Obtaining a degraded image cepstrum (Cin) from a degraded image to be restored;
The similarity between the degraded image cepstrum and the cepstrum of the template image obtained by acting the candidate PSF is used as an fitness function,
An estimation step of obtaining an estimated PSF by evolving the candidate PSF according to a genetic algorithm using the candidate PSF as a gene.

More specifically,
The estimation step includes
Obtaining a plurality of pseudo-degraded images by applying a plurality of candidate PSFs to the template image;
Obtaining a pseudo-degraded image cepstrum (Cgen-i) of the plurality of pseudo-degraded images;
A similarity detection step of obtaining a similarity of each of the plurality of pseudo-degraded image cepstrum (Cgen-i) with respect to the degraded image cepstrum (Cin);
A PSF selection step of determining a plurality of candidate PSFs from the similarity;
Selecting the estimated PSF under a predetermined condition from the candidate PSFs surviving in the selection step;
It includes a PSF neonatal step of generating a new candidate PSF from the candidate PSF surviving in the drought step.

In addition, the method for restoring a camera shake deteriorated image of the present invention includes:
A restoration step of obtaining a restored image of the deteriorated image using the estimated PSF estimated by the PSF estimation method is provided.

  The present invention also includes a program that records the PSF estimation method and the degraded image restoration method, a recording medium that records the program, and a computer device that can execute the program.

  Since the algorithm for the degraded image restoration method of the present invention is completely determined, the degree of restoration does not depend on the skill level of the user. That is, stable restoration can be performed. Further, since the PSF is determined by the similarity between the cepstrum of the degraded image and the cepstrum of the template image, the PSF can be estimated with higher accuracy than the PSF is determined by pixel search on the cepstrum. Therefore, it is possible to restore the deteriorated image more accurately.

It is a figure explaining the effect with respect to the image of PSF. It is a figure which shows the whole flow of the process of this invention. It is a figure which shows the processing flow which calculates | requires the cepstrum of a degradation image. It is a figure which illustrates a degradation image and a degradation image cepstrum. It is a figure which shows the processing flow of the process of evolving PSF. It is a figure which illustrates a degradation image and a template image. It is a figure which illustrates candidate PSF. It is a figure which shows the example which makes a candidate PSF act on a template image, and produces | generates a pseudo degradation image. It is a figure which shows a pseudo degradation image and a pseudo degradation image cepstrum. It is a figure which shows the example which judged the similarity degree of a degradation image cepstrum and a pseudo degradation image cepstrum. It is a figure explaining the method of producing a new candidate PSF from the surviving candidate PSF. It is a figure which shows the structure of the computer apparatus which enables execution of the decompression | restoration method of this invention. It is a processing flow which provides another generation method of candidate PSF. It is an image figure explaining the specific process of the processing flow shown in FIG. It is a figure which shows the processing flow which provides the other production | generation method of candidate PSF. It is a figure for demonstrating the example which makes candidate PSF using a brightness | luminance. It is another figure for demonstrating the example which makes candidate PSF using a brightness | luminance.

  Embodiments of the present invention will be described below with reference to the drawings. In addition, this invention is not limited to the following description, In the range which does not deviate from the meaning, it can change.

(Embodiment 1)
First, in describing the restoration method of the present invention (hereinafter also simply referred to as “restoration method”), several items will be described in advance.

  An image that has deteriorated due to camera shake can be considered to be an image that is generated as a result of a movement locus called camera shake being applied to an original image without camera shake. Here, the locus of camera shake can be represented by a point spread function (hereinafter referred to as “PSF”). That is, in order to restore a deteriorated image, it is possible to restore the deteriorated image by obtaining an PSF that has acted on the deteriorated image and adding an operation to cancel the action of the PSF.

  Equation (1) shows the relationship between the original image (g), the deteriorated image (d), and the PSF (p) when the process of generating the deteriorated image is mathematically modeled. Here, the original image is an image that has not deteriorated due to camera shake. “*” Represents a convolution operation. The deteriorated image can be expressed as a convolution operation between the original image and the PSF.

  FIG. 1 illustrates the effect of the above formula (1). FIG. 1A is an image in which dots are scattered on a plane. This is the original image. FIG. 1B shows the PSF. FIG. 1C shows an image (degraded image) obtained by performing a convolution operation on FIG. 1A and FIG. The points scattered on the original image (FIG. 1A) move in the same manner as the movement of the PSF (FIG. 1B), and the movement locus remains on the deteriorated image (FIG. 1C). A degraded image is thus produced.

  Equation (2) shows a mathematical model for restoring the deteriorated image (d). The original image (g) is subjected to PSF (p) deconvolution operation on the degraded image (d). In the formula (2), “/” indicates a deconvolution operation.

  However, the deconvolution operation is difficult to calculate analytically, and the solution cannot be obtained uniquely. Therefore, a method of approximately performing the deconvolution operation is used. This will be described in detail later.

  In the present invention, an image cepstrum is used. The cepstrum is the result of performing Fourier transform on an image, obtaining the logarithm of the result, and further performing Fourier transform.

  FIG. 2 shows an outline of a degraded image restoration method of the present invention (hereinafter also simply referred to as “restoration method”). The restoration method of the present invention includes a step (step S102) of obtaining a cepstrum of a degraded image to be restored (this is referred to as “degraded image cepstrum”), and a camera shake trajectory so that a cepstrum approximated to the degraded image cepstrum can be generated. (PSF) is evolved to estimate the PSF (step S104), and a degraded image is restored based on the estimated PSF (step S106). In addition, it can be said that step S106 estimates an original image by estimated PSF. As a result of the step of estimating the PSF (step S104), an “estimated PSF” obtained by estimating the PSF that caused the deteriorated image is obtained. Each process will be described in detail.

  FIG. 3 shows details of the process (step S102) for obtaining the cepstrum of the deteriorated image. In this step, the degraded image (d) that is to be restored is captured as image data (step S112). The format of the image data is not particularly limited. Next, a cepstrum of the deteriorated image (d) is obtained (step S114). A cepstrum is the spectrum of a spectrum. First, a two-dimensional Fourier transform process is performed on the degraded image (d). Next, logarithmic processing is performed on the processing result. Further, the result of logarithmic processing is Fourier transformed. The cepstrum of the degraded image (d) is called a degraded image cepstrum (Cin).

  These processes are performed by digital signal processing. Therefore, the pitch when sampling the deteriorated image (d) becomes a problem. This sampling pitch affects how finely the image frequency information is acquired. However, in the restoration method of the present invention, since the PSF uses a genetic algorithm including a mutation as described later, it is not necessary to obtain detailed image information at this stage. Therefore, it may be determined as a design item. Also, electronic data that is shaken by a digital camera or the like may be used.

  Further, gradation processing may be performed on the deteriorated image (d) before sampling. In this step, since the cepstrum of the deteriorated image is obtained, the frequency information of the image is important. At this time, because the gradation is low, accurate restoration cannot be performed unless the frequency information of the image can be acquired.

  FIG. 4A illustrates a deteriorated image (d), FIG. 4B illustrates the deteriorated image cepstrum (Cin), and FIG. 4C illustrates a partially enlarged view of the deteriorated image cepstrum. The degraded image cepstrum is a black and white image when viewed as an image. A portion with high luminance is near the center, and a slightly blurred portion exists around it. A locus close to the PSF can be seen slightly in the bright portion of the center where the vicinity of the center (the portion surrounded by the white square) is enlarged. The cepstrum of the degraded image is used in the next step.

  FIG. 5 shows details of the process of estimating the PSF by evolving the PSF (estimation process). In this step, first, a plurality of candidate PSFs are applied to the template image (tp) (step S132), and a plurality of pseudo deteriorated images (pd) are generated (step S134). Next, a cepstrum of the pseudo deteriorated image pd is obtained (step S136). This is called a pseudo-degraded image cepstrum (Cgen-i). The pseudo-degraded image cepstrum (Cgen-i) is generated by the number of candidate PSFs.

  Then, the similarity (SIM) between the individual pseudo-degraded image cepstrum (Cgen-i) and the degraded image cepstrum (Cin) is calculated (step S138). The similarity (SIM) is a numerical value of the degree of similarity between images. This similarity is the fitness of the genetic algorithm. That is, the fitness function can be said to be a means for obtaining the similarity between the pseudo-degraded image cepstrum and the degraded image cepstrum. This process is called a similarity detection process.

  Next, several candidate PSFs are selected from a plurality of candidate PSFs based on the similarity (step S140). That is, the candidate PSF that is the target of the trap is discarded. In other words, it can be said that candidate PSFs to be alive are determined and other candidate PSFs are discarded. This is called a PSF dredge process.

  Then, a new PSF is generated from the surviving candidate PSF (step S142). At this time, crossover and mutation are generated. This is because the possibility of a candidate PSF capable of generating a pseudo-degraded image cepstrum having a higher similarity than the surviving candidate PSF by generating crossover and mutation is investigated. The process of generating a new candidate PSF from the surviving candidate PSF is called a PSF newborn process. Hereinafter, each step will be described in detail.

  In the step of obtaining the pseudo deteriorated image cepstrum (step S136), the candidate PSF is applied to the template image (tp) to generate the pseudo deteriorated image (pd). The template image is an image used for generating a pseudo deteriorated image. As already described, the cepstrum of the image reflects the frequency characteristics of the image and therefore does not depend much on the content of the image itself. FIG. 6A shows a deteriorated image, and FIG. 6B shows a template image.

  As can be seen by comparing (a) and (b) of FIG. 6, the original image and the template image are images that are similar but not similar. However, it has an approximate frequency characteristic on the cepstrum. If the template image is similar to the original image, it is considered that the PSF can be evolved into an accurate PSF with a fast convergence even when the PSF is evolved.

  In other words, even if the content of the image does not match the face of the person, the face of the person, and the scenery of the landscape, the image has the same composition and there is no camera shake or there is not enough camera shake. It is desirable to select as.

  A plurality of candidate PSFs are created for the template image. The PSF is present in a predetermined range in the central portion of the image. This is called a PSF search range. This is because the trail of camera shake does not spread so much. Here, the candidate PSF is a shape in which one of the center point of the image and any two points provided in the PSF search range is a start point and the other is an end point, and the start point, the center point, and the end point are each connected by a straight line of one white pixel. And an image in which the other portions are filled with black pixels.

  Camera shake is an event in time corresponding to the reciprocal of the shutter speed, and the direction is rarely changed frequently. Therefore, it is considered that the PSF can be approximated as a line segment with few inflection points. Of course, this is the most basic PSF generation method. FIG. 7A shows the candidate PSF generated. The start point (AS) and the end point (AT) are determined so as not to overlap the center point (AC) of the PSF search range (A-PSF).

  Here, the candidate PSF is formed by connecting the start point, the center point, and the end point with a straight line, but a curve connecting these three points may be used by a known method. For example, a cubic spline curve or a Bezier curve is preferably used. FIG. 7B shows a shape when three points are connected by a cubic spline curve. Therefore, it can be said that the candidate PSF is formed by connecting the start point, the center point, and the end point provided in the PSF search range in order by a continuous line.

  In the PSF search range (A-PSF), a method of selecting two points of the start point (AS) and the end point (AT) may be random. This is because a candidate PSF having a low fitness is deceived. On the other hand, in the genetic algorithm, it may take a long time to converge depending on the starting gene (candidate PSF). Therefore, it is preferable that a PSF shape having a high possibility of being taken as much as possible is given as an initial value.

  A plurality of candidate PSFs having different shapes are generated. This is because a genetic algorithm is used in the present invention, and therefore it is necessary to be jealous. That is, by preparing a plurality of typical PSFs and giving them as initial values of candidate PSFs, the candidate PSFs can be easily evolved to a shape close to a true PSF.

  An image generated as a result of applying the candidate PSF to the template image is a pseudo-degraded image (pd). The action here is to perform a convolution operation between the template image and the candidate PSF. FIG. 8 shows a state where four candidate PSFs are applied to the template image (tp) to generate four pseudo-degraded images (pd). The four candidate PSFs have different shapes and sizes. The four candidate PSFs are a candidate PSF1, a candidate PSF2, a candidate PSF3, and a candidate PSF4, respectively, and pseudo-degraded images generated from the respective candidates are pd1, pd2, pd3, and pd4.

  Next, a cepstrum of the generated pseudo-degraded image is obtained. The method for obtaining the cepstrum is the same as the method for obtaining the cepstrum of the deteriorated image (step S102). The cepstrum of the pseudo deteriorated image (pd) is referred to as a pseudo deteriorated image cepstrum (Cgen-i). The pseudo-degraded image cepstrum is generated for each candidate PSF. That is, a plurality of pseudo deteriorated image cepstrum are generated. FIG. 9 shows the result of obtaining each pseudo-degraded image cepstrum (Cgen-i) for the four pseudo-degraded images generated in FIG. It should be noted that the four pseudo-degraded image cepstrum are numbered in order, and are designated as Cgen-1, Cgen-2, Cgen-3, and Cgen-4, respectively.

  Next, the degree of similarity between the deteriorated image cepstrum (Cin: see FIG. 4B) and the individual pseudo deteriorated image cepstrum (Cgen-1 to Cgen-4) is obtained (step S138). The method for obtaining the image similarity is not particularly limited. Here, the normalized correlation method expressed by equation (3) was used. In this method, Ri of the same image is 1. That is, if Ri is close to 1, the deteriorated image cepstrum (Cin) and the pseudo deteriorated image cepstrum (Cgen-i) are approximated, and it is determined that an appropriate PSF is selected. The normalized correlation value Ri of Cin and Cgen-i for the total number of pixels M is expressed by the following equation.

  The upper bar of Cin is the average value of the pixel values of Cin, and the upper bar of Cgen-i is the average value of the pixel values of Cgen-i.

  This procedure is performed between all pseudo-degraded image cepstrum (Cgen-i) and degraded image cepstrum (Cin). As a result, the similarity corresponding to the candidate PSF is obtained as a result. The process of obtaining the similarity for the candidate PSF generated in this way is called a similarity detection process. FIG. 10 shows the result of calculating the similarity between the deteriorated image cepstrum (Cin) and the pseudo deteriorated image cepstrum (Cgen-i). In FIG. 10, the similarity of the pseudo-degraded image cepstrum (Cgen-1) is the highest at 0.8, and the similarity of the pseudo-degraded image cepstrum (Cgen-2) is the lowest at 0.2.

  This similarity is used as the fitness of the candidate PSF that generated these pseudo-degraded image cepstrum. That is, from FIG. 8 and FIG. 10, the applicability of candidate PSF1 is 0.8, which is the similarity of Cgen-1. Similarly, the applicability of candidate PSF2, candidate PSF3, and candidate PSF4 are 0.2, 0.3, and 0.7, respectively.

  In the similarity detection step (step S138), the candidate PSF to which the similarity is given is deceived based on the fitness (similarity) (step S140). As a method of deception, a method well known in general with a genetic algorithm can be used. For example, elite selection, roulette and tournament methods can be used. You may determine the number which hesitates among several suitably. A process of selecting at least one candidate PSF from a plurality of candidate PSFs is referred to as a PSF selection process.

  Next, the plurality of surviving candidate PSFs are evolved (step S142). The surviving candidate PSF is a candidate PSF that has generated a pseudo-degraded image with a high degree of similarity. Two of these candidate PSFs are selected to generate crossover and mutation. FIG. 11 shows a method of evolution of candidate PSFs. The selected candidate PSF-a and candidate PSF-b each have a start point (AS1, AS2), a center point, and an end point (AT1, AT2). Therefore, the center points are overlapped, and the respective start points and end points are arranged in one PSF search range. That is, in the PSF search range (A-PSF), there are a total of five points, two start points and two end points, and one central point.

  An intermediate point between the start points (AS1, AS2) and the end points (AT1, AT2) is set as a new start point (NS) and end point (NT). This is crossover. Here, the intermediate point is an intermediate point of the distance between the start points and the end points in the PSF search range. Also, one random point is added as a mutation in the PSF search range. This is the mutation point (MU). As a result of crossover and mutation as described above, seven points other than the center point are generated in the PSF search range (A-PSF).

If the candidate PSF that survived in the PSF selection process is a parent generation, a new candidate PSF of a child generation is generated by two points and a center point arbitrarily selected from these seven points. That is, ₇ C ₂ (= 21) child generations can be created.

  It should be noted that there are a number of ways in which such candidate PSFs can be evolved as a genetic algorithm. In the present invention, if the estimation process includes the elements of the flow shown in FIG. The repeating method may be appropriately modified.

  For example, the following method can be performed. First, prepare and input three parameters: group size (maximum number of candidate PSFs), number of generations (maximum number of wrinkles), number of GA iterations (example: group size = 30, number of generations = 10, GA iterations). Number of times = 5). Here, the number of GA repetitions is the number of candidate PSFs that have finally evolved after the maximum number of wrinkles.

The initial population for the first GA is created at random for all 30 individuals. Next, two are randomly selected from 30 individuals and crossover / mutation is performed. This crossover / mutation can generate ₇ C ₂ = 21 children from the same parents. As an example, assume that two children are generated from two parents. This is repeated [group size ÷ 2 = 15] times to generate 60 candidate PSFs including the parent.

  The same parent may be selected, and the same individual existing in the group may be generated from the same parent. The fitness values of all 60 individuals are examined, and they are drowned to the group size (30 individuals) by dredging work. This part corresponds to step S132 to step S142 in FIG. However, unlike FIG. 5, it is performed before the step of calculating the fitness of candidate PSFs from the parent generation to the child generation (step S138). As described above, the child generation may be created after the user is eclipsed, or the child generation may be eclipsed after the child generation is created.

  This crossover and selection is performed for the number of generations (10 times), and the individual with the highest fitness is finally set as the optimal solution for the first GA.

  Next, after the GAnth time (n ≧ 2), random initial values are created for [2/3 = 20 of the collective size]. Then, the PSF coordinates obtained by shifting the PSF of the optimal solution obtained by GAn-1 for the first time by several pixels (that is, the PSF near the representative point of the optimal solution) are generated for the remaining [1/3 = 10 of the group size]. The initial population is 30 individuals. Crossover and culling are performed in the same way as the first time, and when the number of generations reaches the limit, the nth optimal solution is output.

  By repeating the above operation, the optimum solution of [GA repetition count = 5] is output, and the PSF having the maximum fitness is output. The above method corresponds to repeating each step of FIG. 5 a plurality of times while appropriately changing the order.

  As described above, the estimation step (step S104) includes a step of obtaining a pseudo-degraded image (step S136), a step of generating a pseudo-degraded image cepstrum (step S134), a similarity detection step (step S138), and a PSF. A drought process (step S140) and a PSF newborn process (step S142) are included. By repeating this estimation process, the candidate PSF evolves to resemble a true PSF.

  Actually, the repetition of the estimation process is terminated when at least one of the candidate PSFs exceeds a predetermined similarity, or is terminated by a predetermined number of repetitions. The candidate PSF obtained through the estimation step (most evolved candidate PSF) may be considered as an estimated PSF, and may be referred to as an “estimated PSF”. In addition, in this step, it can be said that a genetic algorithm is applied to PSF estimation using candidate PSFs as genes.

  Next, the degraded image is restored using the candidate PSF (estimated PSF) that has undergone the estimation process (step S106). For this restoration, a deconvolution operation is performed as represented by equation (2). The deconvolution operation is generally considered to have no analytical solution. Therefore, a method for obtaining an approximate solution may be used. Moreover, since it is an approximate solution, several methods can be considered. The deconvolution operation used in the present invention is not particularly limited as long as it is a method that can restore a degraded image using the estimated PSF.

  For example, a deconvolution operation based on Bayesian theory known as an approximate operation of the deconvolution operation is known, and this method can be used. More specifically, the calculation represented by equation (4) is performed.

Here, m represents the number of iterations. Further, gm represents a restored image by the m-th iterative calculation, and h is an estimated PSF. The initial value of the restored image is represented as g ₀ (x, y), and is d (x, y), which is a degraded image.

  The restoration method as described above can be performed by software processing using a computer. That is, it can be realized by a program recording each step shown in FIG. 2 (more specifically, each step shown in FIG. 3 to FIG. 11) and hardware for executing the program. FIG. 12 shows a hardware configuration for implementing the present invention. A computer apparatus equipped with a program for executing the restoration method of the present invention is the restoration apparatus 1 of the present invention.

  The restoration apparatus 1 (computer apparatus) according to the present invention includes an input unit 10, a control unit 12, a main memory 14, and an output unit 16, and an external storage 18 may be added. The input means 10 is connected to the control device 12. The control device 12 is connected to the main memory 14 and the output means 16 and may be connected to the external memory 18. The program describing the restoration method and PSF estimation method of the present invention is recorded in the main memory 14 or the external memory 18. Further, it may be stored in a recording medium removable from the control device 12. Such a program may be in an executable form or in a source state (a state before being compiled).

  The input means 10 transmits the deteriorated image d to the control device 12 as electronic data Dd. The main memory 14 stores a program that codes the above restoration method. In addition, other necessary data is recorded in the external storage 18. The control device 12 outputs the final restored image Dgm to the output unit 16.

  In FIG. 12, a camera is illustrated as the input means 10 in order to convert a deteriorated image d such as a photograph into electronic data Dd. However, when the deteriorated image d is supplied as electronic data Dd from the beginning, the camera may not be a camera. The control device 12 can suitably use a normally used MPU (Micro Processor Unit).

  For example, a display device can be suitably used as the output unit 16. However, the present invention is not limited to this, and the restored image data Dgm may be output as it is. Further, it may be printed out by a printer (not shown).

  As described above, since the restoration method of the present invention uses the genetic algorithm to estimate the PSF, the degraded image can be restored uniformly.

(Embodiment 2)
In the first embodiment, the candidate PSF is generated by generating the start point (AS) and the end point (AT) at random in addition to the center point in the PSF search range. However, if the initial value of a candidate candidate PSF that approximates a true PSF can be given first, the speed of evolution becomes faster, and the calculation time can be shortened or restored with higher accuracy. Therefore, the PSF generated based on the cepstrum of the degraded image is added to the candidate PSF generated at random (step S132 in FIG. 5) of the present invention in addition to the randomly generated candidate PSF.

  A specific method is shown in FIG. This process can be invoked at any time during step S132 of FIG. 5 for generating candidate PSFs (step S200). First, the degraded image cepstrum (Cin) is binarized (step S202). FIG. 14A illustrates a specific example of the degraded image cepstrum. FIG. 14A shows only the PSF search range portion of the degraded image cepstrum.

  FIG. 14B shows a binarized degraded image cepstrum. The binarization method is not particularly limited, and a known method can be used. For example, a threshold value is appropriately determined for the luminance of each pixel (pixel), a pixel having a higher luminance than the threshold is set to white, and a lower pixel is set to black.

  Referring again to FIG. 13, next, the white coordinate point W farthest from the center point of the PSF search range is obtained (step S204). FIG. 14C shows a state where the white coordinate W is designated.

  Referring to FIG. 13 again, next, all the coordinates of the vertical and horizontal k pixels centering on the white coordinate W are acquired as a coordinate group ΣW (step S206). Therefore, the coordinate group ΣW includes (k × k) coordinates. Then, for each coordinate of the coordinate group ΣW, a corresponding coordinate group ΣZ having the center point (AC) as the center of the point object is obtained (step S208). Accordingly, there are (k × k) coordinates in the corresponding coordinate group ΣZ.

  Then, one point is selected from each of the coordinate group ΣW and the corresponding coordinate group ΣZ, and a total of (k × k) candidate PSFs having the start point and the end point are created. Here, the start point and the end point of the created candidate PSF are created so as not to overlap each other. By adding these candidate PSFs to the candidate PSFs in step S132 of FIG. 5, the overall convergence becomes high.

(Embodiment 3)
The PSF can be considered as a movement trajectory of the camera while the camera is exposed. Then, the movement distance per minute unit time can also be considered. Considering this by convolution of the original image and the PSF (Equation (1)), when moving in a minute unit time, the original image shifted by a minute distance corresponding to the minute unit time is calculated. On the other hand, when there is little movement in a minute unit time, the original image is convolved at the same position. This difference should appear as a difference in PSF brightness.

  Therefore, in this embodiment, a method for obtaining an estimated PSF in consideration of luminance is shown. In this method, first, an estimated PSF corresponding to a camera trajectory is obtained using the first or second embodiment. The estimated PSF is an image formed by white pixels in the black PSF search range. Therefore, luminance is given to the white pixel portion to obtain a luminance candidate PSF. The candidate PSF with luminance is evolved using a genetic algorithm to obtain an estimated PSF with luminance.

  FIG. 15 shows an outline of the processing flow. This flow is realized as a subroutine called in step S104 of FIG. 1 (step of evolving PSF (estimating PSF)). Of course, step S104 may be made the method of this embodiment from the beginning.

  Referring to FIG. 15, first, the PSF of the movement trajectory is obtained (step S402). This is the most advanced PSF (estimated PSF) obtained in the first or second embodiment. This estimated PSF is referred to as Ppath. In the Ppath, only the Ppath (estimated PSF) in the PSF search range A-PSF is a white pixel, and the rest is a black pixel.

  Next, the white pixel portion is numbered (step S404). This sequentially assigns numbers to white pixels representing Ppaths without duplication. FIG. 16 shows Ppath binarized in this way and numbered for each pixel. Here, for the sake of illustration, Ppath indicates a case of 11 pixels. In FIG. 16, the start point (AS), the center point (AC), the end point (AT), and each pixel are numbered (1 to 11), but this character is not actually present.

  Next, brightness is given to each white pixel (step S406). This brightness is evolved as a new gene. The PSF to which this luminance is given is called a luminance candidate PSF. The method of giving luminance may be a method of giving a value randomly at first. For example, when the gradation can be made in 256 levels, this gradation is replaced with luminance. Each pixel is given a luminance of 1 to 256 (or 0 to 255). FIG. 17 illustrates a case where three levels of luminance are given to the eleven pixels in FIG. The brightest brightness was given to the four pixels 2, 4, 10, and 11. The brightness of the next brightness was given to the three pixels 7, 8, and 9. The darkest luminance is given to the four pixels 1, 3, 5, and 6.

  Next, the luminance candidate PSF is evolved (step S408). The method of evolution may be the same as in the first embodiment (flow shown in FIG. 5). That is, when the luminance candidate PSF is generated, a pseudo deteriorated image is obtained from the template image and the luminance candidate PSF. Then, the cepstrum of the pseudo-degraded image is obtained, and the candidate PSF with luminance is made to appear while comparing with the degraded image cepstrum.

  In addition, there are various ways to create the child generation of the candidate PSF with brightness that is a gene, and there is no particular limitation. An example is shown below. Two parents are selected from the parent generation luminance candidate PSF, and the intermediate luminance value is set as the luminance value of the child generation for each pixel. It should be noted that a mutation is generated in any pixel with a low probability.

  As another method, one-point crossover often used in a genetic algorithm can be used. In the one-point crossover, after selecting two genes (a, b), one point is determined between gene sequences, and the crossover point is determined at random. Then, the gene is cut into two at the intersection. The gene is divided into four (a1, a2, b1, b2) (numbers 1 and 2 indicate the order of the genes). The divided genes are exchanged and connected to each other (a1, b2 and b1, a2).

  Next, each pixel value of the generated two genes is replaced with another value with a low probability, and this is set as a mutation. The following can be considered as an example of the mutation probability. If the gene length (the number of PSF white pixels) is n, the pixel value is changed to another value with a probability of 1 / n for each pixel, and this is performed n times (n pixels in order, all). That is, it is set so that the value of one gene changes by mutation is about one pixel. By generating a child generation from the parent generation in this way, the candidate PSF with luminance can be evolved.

  Here, when performing a convolution operation on the candidate PSF with brightness and the template image, the weighting in the convolution operation is changed depending on the luminance. For example, when performing a convolution operation at a pixel point with high luminance, a high weight is given to the calculation result at that point, and when performing a convolution operation at a pixel point with low luminance, the calculation result at that point is low. Give weight.

  Therefore, it can be said that the weighted candidate PSF with weight is calculated at a ratio of luminance / total luminance in the image for each point where the convolution calculation is performed.

  The present invention can be used to restore degradation of still images taken by a camera such as a digital camera, and can restore one scene in which a camera shake occurs due to panning of a video shooting camera or the like. It can also be used for applications.

1. Restoration device (computer device)
DESCRIPTION OF SYMBOLS 10 Input means 12 Control apparatus 14 Main memory 16 Output means 18 External memory

Claims (15)

Obtaining a degraded image cepstrum (Cin) from a degraded image to be restored;
The similarity between the degraded image cepstrum and the cepstrum of the template image obtained by acting the candidate PSF is used as an fitness function,
A PSF estimation method including an estimation step of obtaining the estimated PSF by evolving the candidate PSF according to a genetic algorithm using the candidate PSF as a gene. The estimation step includes
Obtaining a plurality of pseudo-degraded images by applying a plurality of candidate PSFs to the template image;
Obtaining a pseudo-degraded image cepstrum (Cgen-i) of the plurality of pseudo-degraded images;
A similarity detection step of obtaining a similarity of each of the plurality of pseudo-degraded image cepstrum (Cgen-i) with respect to the degraded image cepstrum (Cin);
A PSF selection step of determining a plurality of candidate PSFs from the similarity;
Selecting the estimated PSF under a predetermined condition from the candidate PSFs surviving in the selection step;
2. The PSF estimation method according to claim 1, further comprising a PSF newborn step of generating a new candidate PSF from the candidate PSFs that survived in the selection step. The candidate PSF is
A starting point provided within the PSF search range;
A center point of a PSF search range determined in advance from the center of the degraded image;
The PSF estimation method according to claim 2, wherein the PSF search method is created by connecting the end points provided in the PSF search range with continuous lines. The candidate PSF binarizes the degraded image cepstrum (Cin);
Obtaining the white coordinate farthest from the center point of the PSF search range predetermined from the center of the degraded image;
Obtaining a plurality of coordinates of points around the white coordinates as a coordinate group;
Obtaining a corresponding coordinate group having the coordinate group and the center point as a center of a point object;
The PSF estimation method according to claim 2, wherein the PSF estimation method is formed by connecting a start point selected from the coordinate group, the center point, and an end point selected from the corresponding coordinate group by continuous lines. The similarity detection step includes:
5. The PSF estimation method according to claim 2, wherein the PSF estimation method is a step of obtaining a SIM of the deteriorated image cepstrum (Cin) and the pseudo deteriorated image cepstrum (Cgen-i). The PSF dredge process includes
The PSF estimation according to any one of claims 2 to 5, which is a step of surviving at least two candidate PSFs that have generated the pseudo-degraded image cepstrum having the high similarity. Method. The PSF renewal process includes
Two points from seven points of the two starting points and the end points of the surviving candidate PSF, the intermediate point between the start points, the intermediate point between the end points, and the mutation point randomly generated in the PSF search range The PSF estimation method according to any one of claims 2 to 6, characterized in that a new candidate PSF is selected at random with the same probability. Adding a luminance to each white pixel of the estimated PSF obtained by the PSF estimation method according to claim 1 to generate a candidate PSF with luminance;
A PSF estimation method including a step of evolving the luminance candidate PSF.   The program which recorded the PSF estimation method described in Claim 1 thru | or 8.   A recording medium on which the program according to claim 9 is recorded.   A computer apparatus comprising an input unit, a control device, a main memory, and an output unit, and capable of executing the program according to claim 9.   9. A degraded image restoration method comprising: a restoration step of obtaining a restored image of the degraded image using the estimated PSF estimated by the PSF estimation method according to claim 1.   The program which recorded the PSF estimation method described in Claim 12.   A recording medium on which the program according to claim 13 is recorded.   14. A computer apparatus including an input unit, a control device, a main memory, and an output unit and capable of executing the program according to claim 13.