JP4872300B2 - Image restoration apparatus and image restoration method - Google Patents

Description

According to the present invention, when a deteriorated still image is captured due to the effects of camera shake, out-of-focus, atmospheric fluctuation, or the like, or when a moving image is captured, one frame with a moving image is similarly affected and deteriorated still image. The present invention relates to an image restoration apparatus and an image restoration method for restoring an image with respect to the deteriorated image.
In particular, the present invention relates to an imaging device of a camera mounted on a digital still camera, a digital video camera, a mobile phone, or the like, but is not limited thereto.

Camera shake that occurs when an imaging device is held in the hand and camera shake that occurs when a subject is photographed from inside a shaken vehicle is deteriorated because the imaging device is shaken and the photographed image flows in a certain direction. In addition, when there is a panning shot of a moving subject or a part of the subject that moves at high speed, a part of the image is taken still or a part of the image flows and is captured. In the captured image that has undergone such deterioration, if a deteriorated portion in the captured image is detected and the deteriorated image can be mathematically modeled, the deterioration can be repaired by applying an image restoration algorithm.

Except in special cases, the mathematical image model of the deteriorated image can be modeled as a two-dimensional linear model, and information on the deterioration is expressed by an impulse response function. That is, the image restoration is to apply an inverse filter to the deteriorated image by an inverse function of the impulse response function in the two-dimensional linear model. (For example, Reference 1)

H. C. Andrews and B.M. R. Hunt, “Digital Image Restoration”, Prentice Hall, 1977.

However, in the method according to Reference 1, there is a modeling error between an image actually captured due to deterioration and a mathematical image model in the vicinity of the boundary of the image, and a highly accurate repaired image cannot be obtained. Alternatively, the modeling error is amplified by the inverse filter, resulting in a further deteriorated image. Further, in the method using a discrete Fourier transform (DFT) matrix for processing, an approximation occurs near the boundary of a defined area, and a modeled error due to the approximation causes a repaired image to be greatly deteriorated.
As a similar approach to the method using DFT, there is a method using a discrete cosine transform (DCT) matrix or a discrete sine transform (DST). Similarly, since a modeling error due to approximation occurs, the repaired image is greatly deteriorated.

An image restoration device according to the present invention is an image restoration device that restores a deteriorated image input from the outside where the image has deteriorated by estimating an original image without deterioration based on a mathematical model, enter beauty degradation image correction parameter Oyo boundary value correction degraded image is the input image of the image restoration process modified to the boundary matching the mathematical model of the imaging region outputs a new degraded image correction parameter a parameter estimation unit that, the original image estimating unit for outputting a new repair image enter the date of the boundary value correction degraded image and the latest of the degraded image correction parameter, the degraded image and the latest input from the outside enter the degraded image correction parameter, or degraded image input from the outside and enter the date of the repair image and the latest of the degraded image correction parameter And a boundary value correcting unit that outputs the Do boundary value correction deteriorated image,
In a first process of repairing a deteriorated image input from the outside, the boundary value correction unit inputs the deteriorated image input from the outside and an initial value of the deteriorated image correction parameter, and is input from the outside. After filtering the boundary portion between the imaging region and the outer region of the imaging region after padding the outer boundary of the imaging region of the degraded image with the initial value of the degraded image correction parameter, the degradation image input from the outside A first process of merging and outputting a new boundary value corrected degraded image to the parameter estimating unit is performed, and the parameter estimating unit outputs the new boundary value corrected degraded image output in the first process. And an initial value of the deteriorated image correction parameter is input, and a second process of outputting a new deteriorated image correction parameter is performed,
In the process of repeatedly performing the parameter estimation process performed after the first process, the boundary value correction unit inputs the latest deteriorated image correction parameter and the deteriorated image input from the outside, and is input from the outside. After filtering the boundary area between the imaging area and the outer area of the imaging area after padding the outside of the imaging area boundary of the degraded image, the degraded image input from the outside after filtering with the latest degradation image correction parameter input And a third process of outputting a new boundary value corrected degraded image to the parameter estimating unit, and the parameter estimating unit outputs the new boundary value corrected degraded image output in the third process. And the latest process for inputting the deteriorated image correction parameter and outputting a new deteriorated image correction parameter are performed, and the third process is performed. Management and the fourth process is repeated a predetermined number of times,
In the process of repeatedly performing original image estimation performed after the process of repeatedly performing the parameter estimation process, the boundary value correction unit inputs the latest deteriorated image correction parameter and the deteriorated image, and the deterioration input from the outside After filtering the boundary portion between the imaging region and the outer region of the imaging region after padding the outer boundary of the imaging region of the image with the input latest degraded image correction parameter, the degraded image input from the outside And performing a fifth process of outputting a new boundary value correction deteriorated image to the original image estimation unit, the original image estimation unit including the latest boundary value correction deteriorated image and the latest deteriorated image correction parameter. The boundary value correcting unit performs the sixth process of outputting a new repaired image, and the boundary value correcting unit outputs the new repaired image output in the sixth process. An area outside the imaging area and the imaging area after the image, the latest degraded image correction parameter, and the degraded image input from the outside are input and padded outside the boundary of the imaging area of the input latest restored image The boundary part is filtered with the inputted latest deteriorated image correction parameter, merged with the deteriorated image inputted from the outside, and a new boundary value corrected deteriorated image is output to the original image estimating part. 7 is performed, the sixth process and the seventh process are repeated a predetermined number of times, and then a new repaired image output in the sixth process is output .

The image restoration apparatus according to the present invention is configured as described above, so that an image captured by being deteriorated due to camera shake or blurring is located near the boundary of a defined region so as to conform to a mathematical image model. Since the pixel value is corrected and the image restoration algorithm is applied, stable and highly accurate image restoration can be performed without depending on the deteriorated image.
Further, it can be incorporated into blind image restoration, and the present invention improves the accuracy of the restored image and improves the estimation accuracy of parameters included in the mathematical image model.

  Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below is mainly suitable for use as a method for restoring an image that has been deteriorated due to camera shake or blurring in imaging using a digital still camera. It is a technology that can also be applied to. Further, it is a technique that can capture and process captured images in a computer.

Embodiment 1 FIG.
FIG. 1 is a diagram showing processing of the padding unit 11 included in the boundary value correcting unit 1 of FIG. 4 in the present invention, and FIG. 2 is a diagram of the filtering unit 12 included in the boundary value correcting unit 1 of FIG. FIG. 3 is a diagram showing processing, and FIG. 3 is a diagram showing processing of the merge unit 13 included in the boundary value correction unit 1 of FIG. 4 in the present invention. 4 is a block diagram showing an image restoration algorithm in the present invention, and FIG. 5 is a block diagram of the boundary value correction unit 1 of FIG.

  The present invention relates to an image restoration algorithm and a blind image restoration algorithm in digital signal processing, and the image restoration algorithm is an impulse response function related to the degradation when the obtained image is degraded due to camera shake or blurring. And an additional noise property, and the original image is estimated (image restoration, image restoration). Further, the blind image restoration algorithm uses only the obtained degraded image and estimates the impulse response function related to degradation and the nature of additional noise in addition to the estimation of the original image. This is the same as estimating the parameters included in the mathematical image model.

Further, the original image is also modeled with a mathematical image model, and a blind image restoration algorithm may be applied. In this case, parameters included in the mathematical image model indicating the original image are also estimated. At this time, it is considered that the parameter estimation accuracy and the original image estimation accuracy largely depend on the algorithm. But the algorithm is not a big problem. The modeling error included in the mathematical image model is a major factor, and the accuracy of parameter and original image estimation is greatly reduced.
In the following, a mathematical image model of a deteriorated image is described, the modeling error and the processing of the present invention are clarified, and then the present invention is described in detail using a block diagram and an image schematic diagram.

An image restoration apparatus according to the present invention is an image restoration apparatus that restores a deteriorated original image based on a mathematical model, and after the original image is input and padded outside the boundary of the imaging region of the original image of the boundary between the outer region and the imaging region of the imaging region, and the boundary value correction unit for filtering by the initial value of the degraded image correction parameter, based on the corrected image boundary portion by the pre-Symbol boundary value correction unit a parameter estimation section for estimating the degraded image correction parameter, based on the previous SL corrected image boundary by the boundary value correction section, and the front Symbol parameters estimated degraded image corrected by estimator parameter estimates original image , and an original image estimation unit for outputting a restoration image, before Symbol boundary value correction unit further receives the restoration image estimated by the original image estimating unit, imaging territory of the repair image After padding bordering Kaige side, in which the boundary between the outer region and the imaging region of the imaging region, again filtered by pre-Symbol parameter estimator degraded image correction parameter estimated by.

In Expression (1), i and j represent pixel positions, and i = 1,..., M1, j if the image has a size of M1 pixels in the vertical direction of the screen and N1 pixels in the horizontal direction of the screen. = 1,..., N1, which is an integer value. At this time, the image is defined by an area of M1 × N1. The defined area is the same as the area to be imaged. u _{i, j} is an original image without degradation, z _{i, j} is a degraded image captured due to degradation during imaging such as camera shake and blurring, and η _{i, j} is additional noise included in the captured image. , D _{k, l} are impulse response functions indicating camera shake and blur, (2K + 1) is calculated from K (2K + 1) is the order of the impulse response function in the screen vertical direction, and (2L + 1) is calculated from L (2L + 1) is the screen horizontal direction of the impulse response function Represents the order of. However, the average value of the original image and the deteriorated image is assumed to be zero. This can be calculated by subtracting the respective average values from the pixel signals of the respective captured images. At this time, as a statistical property of the original image and the deteriorated image, assuming that the expected value of these images is equal to the average value (ergodic property), E [•] _{i, j} ] = 0 and E [z _{i, j} ] = 0. That is, here, the expected value is the same as the average value. Further, it is assumed that the additional noise η _{i, j} has a property of E [u _{i, j} η _{ia, ja} ] = 0 (where ia and ja are arbitrary). This assumption is derived from the fact that the additional noise is almost independent of the original image. Among the image restoration algorithms, E [η _{i, j} ] = 0, E [η _{i, j} η _{ia, ja} ] = σ ^2δ _{i-ia, j-ja} (δ: Kronecker's property) Some of them assume a delta function), some assume a sum of squares of additional noise included in a degraded image, and others can ignore noise. However, the present invention is not limited by the assumption of noise.

An enormous number of image restoration algorithms or blind image restoration algorithms based on the formula (1) have been proposed, and there is no limit to enumeration. However, in these algorithms, it is assumed that the original image is zero for a region outside the defined region, or a discrete Fourier transform matrix (DFT) is used in the algorithm assuming that a cyclic signal continues. Or using a discrete cosine transform (DCT) matrix or a discrete sine transform matrix (DST) on the assumption that the original image continues in a mirror image relationship with the region outside the defined region. The original image that exists outside the region is assumed to fit the algorithm. However, since the original image existing outside the region does not take a zero value, and the captured image existing within the region does not exist outside the region, the modeling error can be obtained by applying the above approach. As a result, a good repaired image cannot be obtained.
Further, in the degraded image in the area, the original image outside the area is reflected in the area at the time of degradation, but the value of the original image outside the area is defined in Equation (1). However, since it is treated as a value that takes 0 value, a modeling error still occurs.

In the numerical experiments in the image processing algorithms that have been proposed so far, the modeling error is treated as being included in the additional noise η _{i, j} in the equation (1), which causes an error in the properties of the additional noise. Yes. Further, since the image restoration algorithm is inverse filtering using an inverse function of the impulse response function, there is a disadvantage that additional noise is amplified during image restoration. In other words, additional noise is amplified by the modeling error at the time of image restoration, resulting in a large noise included in the restored image.
Almost no research has been done to analyze the handling of the vicinity of the above-mentioned boundary and the accuracy of the repaired image, and the image restoration algorithm and blind image restoration algorithm that have been published so far do not evaluate the performance of the algorithm correctly and depend on the degraded image. The result is only obtained. Therefore, in the present invention, in order to suppress the influence of the modeling error, a boundary value correction process for a deteriorated image is incorporated.
In the following, the boundary value correction processing for a deteriorated image incorporated to suppress modeling errors will be described in detail using schematic diagrams.

First, in FIG. 1, an area indicated by an inner frame is an area to be imaged, and is also an area where a mathematical image model is defined. When the captured image is deteriorated, the filtered result due to camera shake or blurring is captured, so that the original image up to the region indicated by the outer frame is reflected to the region where the image is captured. At this time, the hatched portion in FIG. 2 shows a region that may be reflected.
However, in the mathematical image model, the original image is defined only in the defined area, and the outside of the area is assumed to take a zero value or an average value for convenience. Therefore, the image area has a different value from the actually captured image. This is a factor of modeling error. In order to suppress modeling errors, it is necessary to make the description near the boundary of the mathematical image model accurate. However, it is accurately indicated that only the defined area is reflected from outside the defined area. It is impossible to describe. Therefore, it is impossible to suppress this error if it is assumed that a zero value or the like is simply taken outside the defined region for convenience. Therefore, the defined area is expanded, and boundary value correction processing is performed so that the captured image conforms to the mathematical image model.

In the mathematical image model, it is assumed that the original image outside the defined area is assumed to have zero value or average value, so that the boundary value of the degraded image is corrected as much as possible to meet this assumption. . If the estimated value of the original image is not obtained (initial stage of the algorithm), the degraded image is the initial estimated value of the original image (this is the initial value when the iterative algorithm is executed, and the initial estimated value of the original image is Will be called.) Considering that the size of the filtering area is (2K + 1) × (2L + 1), the upper end and lower end, the left end and the right end of the estimated value of the original image are expanded to the area indicated by the outer frame shown in FIG. In this case, 2K rows of zero values are filled in at the upper and lower ends of the degraded image, and 2L columns of zero values are filled in at the left and right ends. Hereinafter, the description will be expanded by filling in the zero value, but even if a constant is determined from the obtained information such as the average value of the deteriorated image and the value is filled, the process does not change. The reason why the extended area is set to 2K rows and 2L columns is to satisfy the requirement from the mathematical image model that the value at the outer boundary of the filtering area must be zero.

At this time, if the estimated value of the original image and the estimated value of the parameter are true values, a correction value that eliminates the modeling error can be obtained by filtering the inside of the defined region with an impulse response function and calculating the deteriorated image. be able to.
However, since there is no guarantee that the estimated value of the original image and the estimated value of the parameter are true values, only the hatched portion in FIG. 4 is filtered in order to create a deteriorated image without a modeling error in the extended region. The degraded image is buried in the shaded area. As a result, it is possible to obtain a boundary value corrected and degraded image in which modeling errors are eliminated as much as possible.

Through the above processing, it is possible to create a deteriorated image under the assumption that zero values exist outside the defined area. By incorporating this processing into the blind image restoration algorithm, the problem with respect to the boundary value in the image restoration algorithm is solved, and the accuracy of the estimated value of the parameter and the estimated value of the original image can be improved.
Hereinafter, the present invention will be described in detail with reference to the block diagram of FIG.

FIG. 6 is a block diagram showing a configuration of the image restoration apparatus 101 according to Embodiment 1 for carrying out the present invention, and FIG. 7 is a block diagram showing a configuration of the boundary value correction unit 1 of the image restoration apparatus.

First, in the image restoration apparatus 101 in FIG. 6, a degraded image is input from the input terminal 111, an initial estimated value for estimating an unknown parameter included in the mathematical image model is input from the input terminal 112, and the original value is input from the output terminal 113. The estimated value of the image (restored image) is output from the output terminal 114 as the estimated value of the unknown parameter included in the mathematical image model.

The image restoration apparatus 101 includes a parameter estimation unit 3 that estimates unknown parameters included in a mathematical image model, and an original image estimation unit 4 that estimates an original image. Digital signal processing is used for these estimations. Use a blind image restoration algorithm. In the proposed algorithms, degraded images are input to both parameter estimation and original image estimation. One of the main purposes of the present invention is to model a mathematical image model in an image restoration device. Since the error correction process is incorporated, the boundary value correction unit 1 corrects the value near the boundary of the deteriorated image and inputs the corrected value to the parameter estimation unit 3 or the original image estimation unit 4. The switch 2 switches between the case of estimating the parameter and the case of estimating the original image.

The operation of the switch 2 varies depending on the employed blind image restoration algorithm. For example, when the sequential prediction error method is used to estimate the parameters and the Kalman / fixed interval smoother is used to estimate the original image, the left side of the switch 2 is turned on first, and the parameter estimation unit 3 estimates the parameters, and then the right side is turned on. Then, the original image estimation unit 4 performs image restoration using the estimated parameters. The parameter estimation method using the successive prediction error method and the original image estimation method using the Kalman / fixed interval smoother will be described in detail later.
The parameter estimation algorithm and the original image estimation algorithm have been studied enormously. However, the present invention using the boundary value correction unit 1 is not limited to these algorithms.

  Further, as an algorithm different from the above, the right side of the switch 2 is turned ON first, the original image estimation unit 4 estimates the original image, the left side of the switch 2 is turned ON, and the parameter estimation unit 3 sets the parameter value. There is an algorithm that performs the estimation. In this case, the estimation of the original image and the estimation of the parameters are alternately repeated until both converge. That is, the switch 2 is turned on alternately left and right. As one of such solutions, there is a maximum likelihood estimation method called EM algorithm (Expectation-Maximization Algorithm), but there are many other signal processing algorithms and system identification methods that perform parameter estimation.

In addition, the image restoration algorithms employed in the original image estimation unit 4 are numerous. Most of these methods are based on the equation (1), and are an image restoration algorithm based on a method using the equation (1) and a method derived by modifying the equation (1). Need to estimate the impulse response function in equation (1). A typical image restoration algorithm under the assumption of the impulse response function is described below.

  First, there is an inverse filtering method using an inverse function of an impulse response function. In addition, there is an inverse filtering method for obtaining an inverse function called a pseudo inverse or a general inverse when an inverse function is not available and cannot be obtained. Further, there are a modification of the feature extraction operator in the least square method with constraint condition used in the first embodiment and an algorithm derived as a least square method with weight constraint condition. These techniques are generally performed in the spatial frequency domain using orthonormal transformation. For the orthonormal transformation, DFT, DCT, and DST are often used (for example, when DCT is used, it is often called a DCT spatial frequency domain). All of these algorithms can also be executed as an iterative process in the spatial domain.

Next, there is an algorithm that assumes a statistical property of the original image of the captured image and restores the image based on the property. A typical example is the Wiener filter, which can be executed in either the spatial domain or the spatial frequency domain. Further, there is an image restoration algorithm using a Kalman filter as a Wiener filter successive estimation method. This method can derive an algorithm performed in the spatial domain, the frequency domain, and the spatial frequency domain under the equation (1). Further, since the Kalman filter is a sequential estimation method, it may depend on its initial value, and there is an algorithm that is performed after execution of the Kalman filter called a fixed interval smoother in order to estimate the initial value. As a result, an optimal initial value is estimated, the Kalman filter is executed, and the fixed section smoother is executed again to obtain the best restored image. Further, even if a fixed lag smoother or a fixed point smoother is used instead of the fixed section smoother, image restoration substantially equivalent to that of the fixed section smoother can be performed. These algorithms can also be considered as one of image restoration methods using maximal posterior probability (MAP) estimation. For the MAP estimation, a method using a probability relaxation method using a simulated annealing method is used.
In addition to the above, there are many methods such as an algorithm centered on a projection filter and an algorithm based on an entropy maximization method. In the present invention, the various methods described above can be used as the image restoration algorithm, and an optimal method can be selected according to the application purpose, the performance of the electronic circuit to be used, the amount of memory, and the like.

Furthermore, in the image restoration apparatus 101 of FIG. 6 according to the present invention, the parameter control unit 5 is used, thereby making it possible to estimate a parameter incorporating a more complicated iterative process and to estimate an original image. The parameter control unit 5 controls the timing of parameter transfer to the boundary value correction unit 1, the parameter estimation unit 3, and the original image estimation unit 4, which will be described below.

When an algorithm that requires an initial parameter estimated value (initial value) is adopted as the parameter estimation algorithm, the parameter estimated value estimated by the parameter estimating unit 3 is input to the parameter control unit 5. By immediately returning to the parameter estimation unit 3, this value can be used as the initial parameter estimation value, and parameter estimation can be performed again. If this process is performed a certain number of times, the parameter estimation accuracy is improved. The method of giving the initial estimated value of the parameter will be shown at the same time because an algorithm in the case of using the sequential prediction error method as the parameter estimating method will be described later in detail.

Moreover, if the estimated value of the parameter estimated by the parameter estimation unit 3 is input to the parameter control unit 5 and output from the parameter control unit 5 to the boundary value correction unit 1, the boundary value is obtained using the new parameter with higher accuracy. A corrected degraded image can be calculated. Therefore, if the parameter estimation unit 3 performs parameter estimation again, the parameter estimation accuracy is improved. If this process is performed a certain number of times, the parameter estimation accuracy is improved.

In addition, when the original image estimation unit 4 executes image restoration, the parameters of the mathematical image model are required, and therefore the parameter estimation value with higher accuracy than the parameter control unit 5 is sent to the original image estimation unit 4. Is output. The original image estimation unit 4 not only outputs the repair image to the output terminal 113 but also outputs it to the boundary value correction unit 1. The boundary value correction unit 1 uses the degraded image input from the input terminal 111, the restored image input from the original image estimation unit 4, and the estimated value of the parameter input from the parameter control unit 5. Output an image.

As described above, when the boundary value correction degradation image is calculated by the boundary value correction unit 1 every time the estimated value of the parameter or the estimated value of the original image is obtained, the accuracy of the estimated value of the parameter or the estimated value of the original image is further increased. Will improve. The number of times of feedback processing is not particularly determined, and may be determined in advance. It is also possible to incorporate in the algorithm monitoring of the parameter accuracy in the parameter control unit 5, monitoring the accuracy of the repaired image output from the original image estimation unit 4, and the like. Various procedures are possible, all of which are included in the present invention.
Next, correction of the vicinity of the boundary of the deteriorated image in the boundary value correction unit 1 will be described in detail with reference to FIG.

In the boundary value correction unit 1 shown in FIG. 7, a degraded image input from the input terminal 111 of the image restoration apparatus 101 is input from the input terminal 21, and a repaired image output from the original image estimation unit 4 is input from the input terminal 22. An estimated value of a parameter output from the parameter control unit 5 is input from the terminal 23. Then, the boundary value corrected degraded image is output from the output terminal 24.
When calculating the boundary value corrected deteriorated image, the switch 14 selects either the repaired image input from the input terminal 22 or the deteriorated image input from the input terminal 21 (initial estimated value of the original image). The image selected by the padding unit 11 is zero-filled in the area indicated by the oblique lines in FIG. 3 (described above, but the average value of the deteriorated image and other constant values are filled instead of the zero value). Is also good). The zero-filled image is output to the filtering unit 12, and the filtering unit 12 performs filtering with an impulse response function in the estimated value of the parameter input from the input terminal 23 only in the region indicated by the oblique lines in FIG. 4. The image output from the filtering unit 12 is input to the merge unit 13, and the degraded image input from the input terminal 21 is embedded in the area indicated by the oblique lines in FIG. 5 and output to the output terminal 24. Finally, a boundary value corrected degraded image is output from the output terminal 24.

Here, the switch 14 may be considered to perform processing for turning on the lower side during the execution of the algorithm by monitoring the accuracy of the repaired image. Here, the accuracy of the restored image is obtained by degrading the estimated value x _{i, j} of the original image by the convolution operation using the estimated parameters h _{k, l} , for example, to the center (M−2K) × (N−2L). On the other hand, the sum of squares of errors with the deteriorated image is calculated, and this error can be determined from the magnitude of the sum of squares of errors calculated when the initial estimated value of the original image is used. However, the present invention is not limited to the processing method of the switch 14.
Hereinafter, the invention will be described in detail with reference to FIGS. 3 to 7 from the signal flow when the parameter estimation based on the sequential prediction error method is used. The algorithm can include an iterative process, and the number of iterations can be set freely. For the sake of explanation, the number of iterations of the parameter estimation unit 3 with the parameter control unit 5 is p1, the number of iterations of the parameter estimation unit 3 with the parameter control unit 5 and the boundary value correction unit 1 is p2 times, and the original image estimation unit. 4, the number of iterations of the original image is q1 times, the number of iterations of the original image estimation unit 4 and the boundary value correction unit 1 is q2 times, the original image estimation unit 4 including parameter estimation and the boundary value correction unit 1 And r iterations. The order of repetition will be described in detail below.

  First, in the image restoration apparatus 101 shown in FIG. 6, a deteriorated image is input from the input terminal 111 and input to the boundary value correction unit 1. An initial estimated value (initial value) of the parameter is input from the input terminal 112, and the parameter control unit 5 outputs the initial estimated value of the parameter to the boundary value correcting unit 1.

Since the boundary image correction unit 1 shown in FIG. 6 cannot obtain a repaired image from the input terminal 22 at this stage, the switch 14 is turned on at the lower side, and the degraded image input from the input terminal 21 is used as the initial estimation of the original image. A value is input to the padding unit 11. The padding unit 11 fills the hatched portion shown in FIG. 3 with a zero value and outputs it to the filtering unit 12. The filtering unit 12 receives an estimated value of the parameter input from the input terminal 23 (initial estimated value at this stage), and is an estimated value of the impulse response function included in the estimated value of the parameter, as shown in FIG. Filter only the shaded area. Further, the merging unit 13 embeds a deteriorated image input from the input terminal 21 in the shaded portion shown in FIG. 5 and outputs a boundary value corrected image from the output terminal 24.

  Next, since the sequential prediction error method is executed, the left side of the switch 2 is turned ON, and the boundary value corrected degraded image is input to the parameter estimation unit 3. The parameter estimation unit 3 executes the sequential prediction error method and outputs an estimated value of the parameter to the parameter control unit 5. At this time, the first parameter estimation value is obtained. The parameter control unit 5 outputs the first parameter estimation value to the parameter estimation unit, and the parameter estimation unit 3 executes the sequential prediction error method using the first parameter estimation value as the initial estimation value to perform parameter estimation. . The p1 parameter estimation value obtained after performing this iterative process p1 times is output from the parameter control unit 5 to the boundary value correction unit 1 and the parameter estimation unit 3.

  The boundary value correction unit 1 performs the above-described processing using the estimated value of the parameter for the first time, and outputs a boundary value correction deteriorated image. Since the parameter estimation including the boundary value correction unit 1 is repeated p2 times, the parameter control unit 5 has the latest parameter estimation value of p1 × p2. The estimated value is output to the boundary value correcting unit 1 and the original image estimating unit 4, and the right side of the switch 2 is turned on to perform image restoration from the boundary value corrected degraded image output from the boundary value correcting unit 1.

The original image estimation unit 4 inputs the boundary value corrected degraded image via the switch 2 and performs image restoration using the above-described image restoration algorithm using the p1 × p2 parameter estimation values. Some image restoration algorithms can be used for iterative estimation. Then, it is assumed that q1 iteration estimation is performed. The image restored for the first time q is output to the boundary value correction unit 1 and the output terminal 113. In the boundary value correction unit 1, the first restored image is input from the input terminal 22 of FIG. 7, and the boundary value correction deteriorated image is output from the output terminal 24 in the above-described procedure by turning on the upper side of the switch 14. . The output boundary value corrected deteriorated image is input to the original image estimation unit 4 via the switch 2 and image restoration is performed. Such image restoration is repeated q2 times. When the image restoration is repeated q1 × q2 times, the parameter estimation is repeated p1 × p2 times again, and the image restoration is performed q1 × q2 times again using the parameter estimation values obtained in this iteration. The q1 × q2 × r restored image obtained by repeating these processes r times is used as an estimated value of the original image finally obtained, and is output from the output terminal 113. At the same time, the estimated value of the parameter of p1 × p2 × r is output from the output terminal 114 as the final estimated value of the parameter.
The blind image restoration algorithm using the above iterative prediction error method will be described in detail with reference to FIG. 8 as a flowchart.

  First, initial estimated values of parameters included in the degraded image and the mathematical image model are input (S1, S2). A boundary value correction process (S3) is performed using the degraded image (initial estimated value of the original image) and the initial estimated value of the parameter. Parameter estimation (S4) is performed by the sequential prediction error method using the boundary value corrected degraded image. Using the obtained parameter estimation value as an initial value, parameter estimation (S4) is again performed by the successive prediction error method. Such parameter iterative estimation is performed p1 times.

  Next, boundary value correction processing (S3) is performed using the estimated value of the parameter for the first time. Since the boundary value corrected degraded image is calculated, the parameter estimation (S4) is repeated p1 times, and the boundary value correcting process (S3) is performed again. Such boundary value correction processing is performed p2 times.

Boundary value correction processing (S5) is performed using the parameters estimated at the p1 × p2th time, a boundary value corrected degraded image is input, and an original image is estimated (S6). The estimated original image is again used as an initial value, and the original image is estimated (S6). Such repetitive estimation of the original image is performed q1 times. q The boundary value correction process (S5) is performed using the estimated value of the original image obtained for the first time, and the original image is estimated (S6). The iterative estimation of the original image including the boundary value correction process is performed q2 times.

Boundary value correction processing (S3) is performed using the estimated value of the q1 × q2 original image obtained in this way, and parameter estimation is performed again.
By performing the above iteration, the finally obtained estimated value of the p1 × p2 × r parameter and the estimated value of the q1 × q2 × r original image are output, and a restored image is obtained.
Next, a parameter estimation method using the successive prediction error method and an original image estimation method using a Kalman / fixed interval smoother will be described in detail.

The sequential prediction error method and the Kalman / fixed interval smoother need to introduce a mathematical image model called a statistical mathematical model for the original image u _{i, j} in Equation (1). Now, since the image size has been expanded by the boundary value correction unit 1, when the vertical size is expressed as M and the horizontal size as N, and a semi-causal probability model is introduced into the statistical mathematical model showing the original image, The image u _{i, j} is

（式２）
ただし，ξ_i,jはシステム雑音と呼ばれ、平均

(Formula 2)
Where ξ _{i, j} is called system noise and is average

および分散

And distributed

の性質を仮定している。このとき、式（２）および式（１）をベクトル・行列表現すれば、

Is assumed. At this time, if Expression (2) and Expression (1) are expressed as a vector / matrix,

で表現できる。ただし、

Can be expressed as However,

でそれぞれ定義する。
これらの行列はサイズが大きいことから計算機上で演算することは困難であるが、巡回型の行列として近似できれば離散フーリエ変換行列で対角化でき、画像をｉ行ごとに独立して扱うことができる。上式の巡回行列は、それぞれ

Respectively.
These matrices are difficult to calculate on a computer due to their large size, but if they can be approximated as a cyclic matrix, they can be diagonalized with a discrete Fourier transform matrix, and images can be handled independently for each i row. it can. The cyclic matrix of the above equation is

として表現でき、さらにこれらを離散フーリエ変換行列で対角化すれば、式（２）および式（１）は、

If these are further diagonalized by a discrete Fourier transform matrix, equations (2) and (1) can be expressed as

（式３）
で記述することができる．ただし、離散フーリエ変換行列を

(Formula 3)
Can be described as However, the discrete Fourier transform matrix

で定義し、

Defined in

を定義した。右肩に示す添え字（ｉ）は、行を示しており、

Defined. The subscript (i) on the right shoulder indicates a line,

（式４）

(Formula 4)

(式５)

(Formula 5)

(式６)
として表現される。

(Formula 6)
Is expressed as

Here, the unknown parameters necessary for estimating the original image are included in the equations (4) to (6).

と

When

を推定することになる。
以下に、この逐次予測誤差法について詳述する。

Will be estimated.
Hereinafter, this sequential prediction error method will be described in detail.

を定義し、ｉ行ごと画像モデルをＡＲＭＡ（Ａｕｔｏ−Ｒｅｇｒｅｓｓｉｖｅ Ｍｏｖｉｎｇ Ａｖｅｒａｇｅ）モデルによる表現をすると、

And the image model for each i row is expressed by an ARMA (Auto-Regressive Moving Average) model.

に対して、

Against

と表現できる。このとき、本来の未知パラメータ

Can be expressed as At this time, the original unknown parameter

を直接求めるのではなく、

Instead of directly seeking

を推定した後に、最適化問題を定式化し、解くことにより求める。
（６）式において、行を示す変数の右肩の添え字（ｉ）を省略し、表記の便宜性から、

Is obtained by formulating and solving the optimization problem.
In equation (6), the subscript (i) on the right shoulder of the variable indicating the line is omitted, and for convenience of notation,

を定義すると、

Define

（式７）
を得る。ここで，z^-1は遅延演算子を表すとし、

(Formula 7)
Get. Where z ^-1 represents the delay operator,

を定義すると、

Define

の表現を得ることができる。また、次数

Can be obtained. Also the order

と２Ｌのうち大きい方を

And the larger of 2L

と定義すると、観測画像y_jは次数をＲとし、平均が

If the observation image y _j is defined as

、分散が

The variance is

である正規性白色雑音e_jを用いて

Using regular white noise e _j which is

として表現できる。e_jはy_jの一段予測誤差（Ｉｎｎｏｖａｔｉｏｎ過程）と呼ばれる。

Can be expressed as e _j is called y _j one-step prediction error (innovation process).

は安定な多項式であり、

Is a stable polynomial,

を定義することにより、

By defining

として表すこともできる。さらに、y_jのスペクトル密度の関係から、多項式Ｃ(z^-1)は、

It can also be expressed as Furthermore, from the relationship of the spectral density of y _j , the polynomial C (z ⁻¹ ) is

により一意に決定される。このとき、（７）式の対数尤度関数は、未知パラメータを

Uniquely determined by At this time, the log likelihood function of equation (7)

として、

As

（式８）
で表現することができ、θに対する最大化を行うことで、パラメータ推定を行うことができる。ここで、次のような変形を用いて未知パラメータ数を減らし、パラメータの推定を簡潔に行う。
（８）式の両辺を

(Formula 8)
The parameter can be estimated by maximizing θ. Here, the number of unknown parameters is reduced by using the following modification, and parameter estimation is simply performed.
(8)

で割り、

Divide by

を定義すると、

Define

（式９）
を得ることができる。ここで、未知パラメータを、

(Formula 9)
Can be obtained. Where the unknown parameter is

と再定義すれば、予測誤差規範（Ｐｒｅｄｉｃｔｉｏｎ Ｅｒｒｏｒ Ｃｒｉｔｅｒｉｏｎ）

If we redefine, Prediction Error Criterion (Prediction Error Criterion)

の最小化により推定値

Estimated by minimizing

が得られ、

Is obtained,

により一段予測誤差の分散が求まる。さらにこれらが求まれば、（９）式にスペクトル因子分解法を適用して一段予測誤差係数

Thus, the variance of the one-step prediction error is obtained. If these are obtained, the spectral error factorization method is applied to equation (9) to obtain a one-step prediction error coefficient.

および

and

を求め、状態空間モデルのシステム雑音と付加的雑音それぞれの分散を

And calculate the variances of the system noise and additional noise in the state space model.

として求めることができる。スペクトル因子分解法については後述する。
次に、未知パラメータの推定値

Can be obtained as The spectral factor decomposition method will be described later.
Next, an estimate of the unknown parameter

の推定誤差共分散を

The estimated error covariance of

と定義し、初期値を

And the initial value is

、

,

として与えることにより、未知パラメータは逐次的に、

The unknown parameters are sequentially given as

として求めることができる。ただし、

Can be obtained as However,

で定義されており、逐次予測誤差法を実行するには、e_j、Ψ_jを推定することが必要となる。これについて以下に述べる。
e_jは多項式{C_r}が得られていれば、

In order to execute the sequential prediction error method, it is necessary to estimate e _j and Ψ _j . This is described below.
If e _j is the polynomial {C _r },

を用いて得ることができる。この両辺をθで微分すると、

Can be used. Differentiating both sides by θ,

となるので、

So,

を定義すると、

Define

で表現することができる。すなわちΨ_jは、

Can be expressed as That is, Ψ _j is

（式１０）
として表現できる。ただし、

(Formula 10)
Can be expressed as However,

を定義した。
次に、

Defined.
next,

は、

Is

を定義し、

Define

と表現すると、

And

（式１１）
なる関係があるので、

(Formula 11)
Because there is a relationship

を定義して、

Define

を求める。いま、

Ask for. Now

の係数を

Coefficient of

として表すこととし、

And represent it as

を定義すると、

Define

で表すことができる。ここに、

Can be expressed as here,

なる関係があることから

Because there is a relationship

を得ることができ、

Can get the

と

When

の定義から

From the definition of

を得ることができる。また、

Can be obtained. Also,

として、

As

、

,

、

,

と同様な形で

In the same way as

、

,

、

,

、

,

を定義するとκは、

Κ is defined as

として表現することができ、

Can be expressed as

は、

Is

で表される。したがって、

It is represented by Therefore,

（式１２）
が得られる。また、（６）式中における、

(Formula 12)
Is obtained. Further, in the formula (6),

、

,

の具体的な形は、

The specific form of

、

,

、

,

の定義により決定される。本論文では、

Determined by the definition of In this paper,

（式１３）

(Formula 13)

（式１４）
で与えられる。また、

(Formula 14)
Given in. Also,

であるので、最終的にΨ_jは式（１０）、（１１）、（１２）、（１３）、（１４）式を用いて算出することができる。
ここで、逐次予測誤差法におけるパラメータ推定時の初期推定値について述べる。

Therefore, finally, Ψ _j can be calculated using the expressions (10), (11), (12), (13), and (14).
Here, initial estimated values at the time of parameter estimation in the successive prediction error method will be described.

での

In

、

,

、

,

、

,

、ｙ_j（の平均値と同じ値で良い）を与え、初期値Ｐ_Ｌは対角要素に１〜１００程度の値を入れ、初期値

, Y _j (may be the same value as the average value thereof), and the initial value P _L is set to a value of about 1 to 100 in the diagonal element,

においては、インパルス応答関数にはｈ_0,0のみ１を入れそれ以外の要素については０を入れたものから

In the impulse response function, only h _0,0 is set to 1 and other elements are set to 0

を算出し、半因果的確率モデルに含まれる予測係数は経験的に

And the prediction coefficient included in the semi-causal probability model is empirically

を入れたものから

From what put

を算出する。また、

Is calculated. Also,

は

Is

に１０の−３〜−４乗の値を入れ計算したものを初期値とし、

The initial value is calculated by putting 10 to the power of -3 to -4.

については

about

に１０の−４〜−５乗の値を入れ計算したものを初期値とすれば良い。
以上により求まった未知パラメータ

A value calculated by putting a value of 10 −4 to −5 to the initial value may be used as the initial value.
Unknown parameter obtained by the above

を用い、最適化問題を定式化することによって

By formulating the optimization problem using

を推定する手法について述べる。
上述したアルゴリズムによって推定値として得られるパラメータは、

We describe a method for estimating.
The parameter obtained as an estimated value by the algorithm described above is

、

,

、

,

、

,

であり、これらから

And from these

、

,

、

,

、

,

を求めるために、以下の評価規範を最小化する。

To minimize the following evaluation criteria:

は、

Is

の関数であり、評価関数

Function and evaluation function

を、

The

として定義し、パラメータ

Define as parameter

について最小化を行う。最小化のための初期値は、逐次予測誤差法で用いた初期値と同じで良い。
次に、得られた

Minimize for. The initial value for minimization may be the same as the initial value used in the successive prediction error method.
Then obtained

を用いて、

Using,

を計算し、

Calculate

の関係より擬似逆行列を求めてシステム雑音の分散である

The pseudo-inverse is obtained from the relationship of

を求める。
また、インパルス応答関数の推定は、

Ask for.
The impulse response function is estimated as

なる関係式を用い、擬似逆行列により推定する。
最後に、観測雑音の分散は

Is estimated by a pseudo inverse matrix.
Finally, the variance of the observed noise is

である

Is

についての算術平均として計算する。
次に、スペクトル因子分解について述べる。式で、右辺が得られている下で左辺を求めるために本手法を用いる。右辺の

Calculate as the arithmetic mean for.
Next, spectral factor decomposition will be described. This method is used to find the left side under the right side in the equation. Right side

の係数が

Coefficient of

として得られているとする。このとき、（９）式の

Assuming that At this time, the equation (9)

の係数の関係をベクトル・行列表現すれば、

If the relationship between the coefficients of

が成り立つ。ベクトル関数

Holds. Vector function

を定義し、両辺を

Define both sides

で微分すると、

Differentiated by

であるので、Ｎｅｗｔｏｎ−Ｒａｐｈｓｏｎ法により

Because of Newton-Raphson method

を求める反復式は、ｎ回目の推定値を

The iterative formula for calculating the nth estimate is

と表すと、初期値

Represents the initial value.

を与えることにより、

By giving

として求めることができる。初期値は

Can be obtained as The initial value is

の第一要素を比較的大きな値（１００等）とし、他の要素を０で与えることで解を得ることができる。

The solution can be obtained by setting the first element of the above to a relatively large value (100, etc.) and giving the other elements 0.

The algorithm described so far is summarized below.
Step1. Degraded image data

をＤＦＴ行列

DFT matrix

により変換し、

Converted by

を計算して、初期設定として

As the default setting

での

In

、

,

、

,

、

,

を与え，初期値

, Initial value

、

,

、

,

、

,

を与える。さらに、

give. further,

を求め、スペクトル因子分解法により、

And by spectral factorization method,

、

,

を求める。
ステップ２．

Ask for.
Step 2.

とし、

age,

、

,

を用いて一段予測誤差

One-stage prediction error using

を

The

により求める。
Ｓｔｅｐ３．逐次予測誤差法

Ask for.
Step3. Sequential prediction error method

を実行する。
Ｓｔｅｐ４．推定値

Execute.
Step4. Estimated value

から

From

を算出し、スペクトル因子分解法により、

Is calculated by spectral factorization method,

、

,

を求める。
Ｓｔｅｐ５．新しく求まった

Ask for.
Step5. Newly found

、

,

を用いて、再び一段予測誤差

Again, one-stage prediction error

を

The

により求める。
Ｓｔｅｐ６．予測誤差分散、システム雑音、観測雑音を

Ask for.
Step 6. Predictive error variance, system noise, observation noise

により求める。

Ask for.

を計算し

Calculate

を求め、

Seeking

により

By

を求めてStep２へ戻る。
Ｓｔｅｐ７．

To return to Step 2.
Step7.

まで到達したときに収束していなければ、最終推定値

If it does not converge when it reaches

を初期値

The initial value

に代入し、再びStep １から繰り返す。

And repeat from Step 1.

Next, a Kalman / fixed interval smoother is derived from the state space representation. Since there is a delay term by L in equation (3),

を

The

のように定義する。ただし、

Define as follows. However,

を定義した。いま、

Defined. Now

と２Ｌの最大値を

And the maximum value of 2L

として定義し、状態変数

Defined as a state variable

をＲ次に拡大化した新たな状態変数

A new state variable that expands to R next

を

The

で定義する。このとき状態空間モデルを記述し直すと、

Define in. At this time, if the state space model is rewritten,

に対して

Against

（式１５）

(Formula 15)

（式１６）
を得ることができる。ここで、Ｇは、

(Formula 16)
Can be obtained. Where G is

（式１７）
であり、

(Formula 17)
And

であれば、

If,

（式１８）

(Formula 18)

（式１９）
で定義される。もし、

(Formula 19)
Defined by if,

であれば、

If,

の第1行目が

The first line of

となる（２Ｌ＋１）×（２Ｌ＋１）サイズの行列として定義され、

Is defined as a matrix of (2L + 1) × (2L + 1) size,

であれば

If

が

But

となる１×Ｑの行列として定義される。
１５式乃至１９式中の変数右肩にある行を示す添え字（ｉ）を省略し、条件付き期待値と誤差共分散行列を、

Is defined as a 1 × Q matrix.
The subscript (i) indicating the line on the right side of the variable in Equations 15 to 19 is omitted, and the conditional expected value and the error covariance matrix are

で定義すると、以下のカルマン・固定区間スムーザアルゴリズムを与えることができる。初期分布の平均と分散を

The following Kalman-fixed interval smoother algorithm can be given: The mean and variance of the initial distribution

（通常は

(Normally

をデータの平均値、

The average value of the data,

を定常分散値）として与え、

As the steady dispersion value)

に対してカルマンフィルタは

Whereas the Kalman filter is

（式２０）

(Formula 20)

の逐次的計算で実行される。また、カルマンフィルタで求まった

It is executed by sequential calculation. Also found by Kalman filter

、

,

を初期値として、

Is the initial value,

に対して固定区間スムーザは

On the other hand, the fixed interval smoother

の逆逐次計算によって実行される。

It is executed by the inverse sequential calculation.

The following is a summary of the Kalman-fixed interval smoother algorithm.
Step1. Degraded image

をＤＦＴし、

DFT

を計算する。
Ｓｔｅｐ２． （２０）式以下の カルマン・固定区間スムーザを実行する。
Ｓｔｅｐ３． Ｓｔｅｐ２を適切な回数繰り返す。
Ｓｔｅｐ４． 状態推定値から

Calculate
Step2. (20) The following Kalman / fixed section smoother is executed.
Step3. Repeat Step 2 an appropriate number of times.
Step4. From state estimates

を求める。
Ｓｔｅｐ５． 最終的に得られた

Ask for.
Step5. Finally obtained

を逆ＤＦＴすることにより、修復画像

Image by reverse DFT

を得る。

Get.

Embodiment 2. FIG.
Referring to FIG. 8, the second preferred embodiment of the present invention. Is described in detail.

  In FIG. 8, the deteriorated image is input from the input terminal 111 in the image restoration apparatus 101. In addition, the deterioration detection unit 51 estimates an impulse response function included in the mathematical image model based on the camera displacement using the angular velocity sensor output, and outputs the impulse response function as an impulse response function. At the input terminal 112 of the image restoration apparatus 101, the impulse response function estimated by the deterioration detection unit 51 is input, and the deterioration image and the impulse response function are input to the boundary value correction unit 1. In addition, the restored image is input from the original image estimation unit to the boundary value correction unit 1, and the boundary value correction unit 1 calculates a boundary value correction deteriorated image and outputs it to the original image estimation unit 3. The original image estimation unit 3 estimates an original image using an image restoration algorithm and outputs a restored image from the output terminal 113 of the image restoration apparatus 101. Further, the repaired image is also output to the boundary value correcting unit 1. The impulse response function input from the input terminal 112 of the image restoration apparatus 101 is output from the output terminal 114 as the final estimated value of the parameter.

  By using the sensor output as described above, a better restoration image can be estimated.

It is the figure which showed the area | region which may be image | photographed by the imaging area and camera shake etc. in this invention. It is a figure which shows the area | region which receives deterioration from the area | region which may be reflected by the imaging area in this invention, camera shake, etc., and the said area | region which may be reflected. It is a figure which shows the area | region by which 0 is filled in the boundary value correction | amendment part of the image restoration apparatus in this invention. It is a figure which shows the area | region filtered in the boundary value correction | amendment part of the image restoration apparatus in this invention. It is a figure which shows the area | region which fills in a degraded image in the boundary value correction | amendment part of the image restoration apparatus in this invention. It is a block diagram which shows the structure of the image restoration apparatus in this invention. It is a block diagram which shows the structure of the boundary value correction | amendment part in this invention. It is a flowchart of the image restoration in this invention. It is a block diagram which shows the structure of the image restoration apparatus in Embodiment 2 of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Boundary value correction | amendment part 3 Parameter estimation part 4 Original image estimation part 5 Parameter control part

Claims (1)

An image restoration device that restores a deteriorated image input from the outside of the image by estimating an original image without deterioration based on a mathematical model,
Enter beauty degradation image correction parameter Oyo boundary value correction degraded image is the input image of the image restoration process modified to the boundary matching the mathematical model of the imaging region outputs a new degraded image correction parameter A parameter estimation unit to perform,
Enter the date of the boundary value correction degraded image and the latest of the degraded image correction parameter and the original image estimation unit for outputting a new repair image,
Input the externally input deteriorated image and the latest deteriorated image correction parameter, or input the externally input deteriorated image, the latest repaired image and the latest deteriorated image correction parameter, and enter a new boundary. A boundary value correction unit for outputting a value correction deteriorated image ;
With
In the first process of repairing degraded images input from the outside,
The boundary value correction unit inputs an initial value of the deteriorated image input from the outside and the deteriorated image correction parameter, and the imaging region after padding outside the boundary of the imaging region of the deteriorated image input from the outside After filtering the boundary portion between the outer region and the imaging region with the initial value of the deteriorated image correction parameter, it is merged with the deteriorated image input from the outside, and a new boundary value correction deterioration is performed in the parameter estimating portion. Perform the first process to output the image,
The parameter estimation unit receives the new boundary value corrected deteriorated image output in the first process and the initial value of the deteriorated image correction parameter, and outputs a new deteriorated image correction parameter. And
In the process of repeatedly performing the parameter estimation process performed after the first process,
The boundary value correction unit inputs the latest deteriorated image correction parameter and the deteriorated image input from the outside, and the imaging area after padding outside the boundary of the image pickup area of the deteriorated image input from the outside After filtering the boundary portion between the outer region and the imaging region with the input latest deteriorated image correction parameter, the boundary portion is merged with the deteriorated image input from the outside, and a new boundary value correction deterioration is performed in the parameter estimating unit. Perform a third process to output the image,
The parameter estimation unit performs a fourth process of inputting the new boundary value corrected deteriorated image output in the third process and the latest deteriorated image correction parameter, and outputting a new deteriorated image correction parameter. Done
The third process and the fourth process are repeated a predetermined number of times,
In the process of repeatedly performing the original image estimation performed after the process of repeatedly performing the parameter estimation process,
The boundary value correction unit inputs the latest deteriorated image correction parameter and the deteriorated image input from the outside, and the imaging area after padding outside the boundary of the image pickup area of the deteriorated image input from the outside After filtering the boundary portion between the outer region and the imaging region with the inputted latest deteriorated image correction parameter, the boundary portion is merged with the deteriorated image input from the outside, and a new boundary value correction is performed in the original image estimating portion. Perform a fifth process to output a degraded image,
The original image estimation unit performs a sixth process of inputting the latest boundary value correction deteriorated image and the latest deteriorated image correction parameter, and outputting a new repaired image,
The boundary value correction unit inputs the new restored image output in the sixth process, the latest degraded image correction parameter, and the degraded image input from the outside, and the inputted latest restored image. After filtering the boundary area between the imaging area and the outer area of the imaging area after padding the outer boundary of the imaging area with the latest degradation image correction parameter input, the merged with the degradation image input from the outside Then, a seventh process of outputting a new boundary value corrected degraded image to the original image estimation unit is performed,
The image restoration device that repeats the sixth process and the seventh process a predetermined number of times and then outputs a new repaired image output in the sixth process .

Images