JP5493191B2 - Image restoration apparatus and image restoration method - Google Patents

Description

  The present invention relates to an image restoration apparatus and an image restoration method.

  In recent years, in the field of image engineering, much research and development of techniques for restoring original images from degraded images has been performed. That is, unnecessary information (blur and noise) is removed from a deteriorated image (received information) in which unnecessary information (blur and noise) is mixed in the original image (desired information and clear image), and the original image (desired information) Extracting only the image is an indispensable technique in the field of image restoration, and research and development have been actively conducted in recent years. For example, images taken with a digital camera (a generic name for digital still cameras and digital video cameras) and mobile phones are “blurred” due to camera shake and defocus, and Gaussian due to dark current and thermal noise. Or, due to the influence of impulsive “noise”, it is inevitable that the image quality deteriorates compared to the actual product. “Image restoration” is to restore an image as close to the original image as possible from the image thus deteriorated.

  Many of the image restoration techniques that are currently popular in the market are mainly preventive techniques that reduce the effects of blur and noise in advance using, for example, camera shake correction, face recognition, color correction, and various filters. It has become. As a result, in recent years, especially in the field of digital cameras, clear images can be easily obtained due to the increase in functionality and performance of digital cameras.

  However, these preventive technologies do not pose a problem in situations where you can retake the image many times, but images that have already deteriorated, such as old documents, and images that change instantaneously in fields such as sports and medical care, etc. Restoring images that cannot be retaken is still a difficult problem. Here, the images that change instantaneously in the fields of sports and medicine are, for example, the instantaneous movements of athletes and the instantaneous appearance of organs such as the lungs and heart. Therefore, the importance of image restoration is increasing today, particularly in situations where re-taking is not permitted.

  An image restoration technique using a Wiener filter is widely known as a conventional image restoration technique in a situation where re-shooting is not permitted (Non-patent Documents 1 and 2). This method is a filter that minimizes the mean square error between the restored image obtained through the filter and the original image, and is also called a least square filter. Since this method performs processing in the frequency domain, it is an image restoration method on the premise of the continuity of a stochastic process and the size of a semi-infinite image.

  As another image restoration technique, an image restoration method using a projection filter is also known (Non-Patent Document 3 and Non-Patent Document 4). The projection filter evaluates the closeness of the original image and the restored image directly in the space of the original image, and the noise image component of the restored image is the best approximation of each original image, that is, the orthogonal projection of each original image. Among these, the mean square of the noise component of the restored image is minimized. From this property, the projection filter is a technique for restoring the best approximate image regardless of the appearance frequency.

  As another image restoration technique, an image restoration method using a Kalman filter is also known (Non-Patent Documents 5 and 6). In this method, first, in step 1, parameters of the AR (Auto Regressive) system (hereinafter referred to as “AR coefficient”) of the original image are estimated using the degraded image, and then estimated in step 1 in step 2. A state space model (state equation and observation equation) is formed using the AR coefficient, and Kalman filter theory (Kalman filter algorithm) is applied to the state space model, thereby realizing high-performance image restoration. Furthermore, an image restoration method in which the Kalman filter that performs such two-stage processing is expanded to a two-dimensional Kalman filter is also known (Non-Patent Document 7 and Non-Patent Document 8). This method realizes a Kalman filter that restores a deteriorated image using a state space model expanded in two dimensions. In order to distinguish from the two-dimensional Kalman filters shown in Non-Patent Document 7 and Non-Patent Document 8, the Kalman filters shown in Non-Patent Document 5 and Non-Patent Document 6 are appropriately referred to as “one-dimensional Kalman filters”.

Ryohei Nishinomiya, 1 other person, “Image Restoration by Using Multiple Wiener Filters”, IEICE Transactions, A, Vol. J83-A, no. 7, pp. 892-902, July 2000 Nobumoto Yamane and three others, “Optimal Noise Removal Using Adaptive Wiener Filter Based on Locally Stationary Gaussian Mixture Distribution Model for Images”, IEICE Transactions, A, Vol. J85-A, no. 9, pp. 993-1004, September 2002 Hidemitsu Ogawa, 1 other, “Properties of Partial Projection Filter”, IEICE Transactions, A, Vol. J71-A, no. 2, pp. 527-534, February 1988 Yuji Koide and two others, “A Unified Theory of the Family of Projection Filters for Signal and Image Estimation”, IEICE Transactions, D-II Vol. J77-D-II, no. 7, pp. 1293-1301, July 1994 Takashi Shiro and two others, “Image Modeling and Parameter Identification for Image Restoration Using a Kalman Filter”, IEICE Transactions, D-II, Vol. J80-D-II, no. 11, pp. 2912-2919, November 1997 Satoshi Matsumura and two others, “A Kalman Filter Using Adaptive Image Modeling for Noise Reduction”, IEICE Transactions, D-II, Vol. J86-D-II, no. 2, pp. 212-222, February 2003 Mahmood R Azimi-Sadjadi and Ping Wong, "Two-Dimensional Block Kalman Filtering for Image Restoration," IEEE Trans. On Signal Process, vol.ASSP-35, no.12, Dec. 1987 Stuart Citrin and Mahmood R. Azimi-Sadjadi, "A Full-Plane Block Kalman Filter for Image Restoration," IEEE Trans. On Image Processing, vol.1, no.4, Oct. 1992

  However, the image restoration method using the Wiener filter has the advantage that restoration is possible regardless of the degradation state of the image, but on the other hand, it is a natural image (processed) that has a strong non-stationarity (change in image dispersion). However, there is a drawback in that the restoration accuracy is low for an image that has not been shot).

  In other words, the image restoration method using the Wiener filter performs processing in the frequency domain as described above, and therefore assumes the stationary nature of the stochastic process and the size of the semi-infinite image, but in reality, Since it is difficult to satisfy this precondition in a real environment, there are cases where it cannot be properly restored (the problem of non-stationarity of natural images). Further, since this method is a batch process using the least square error as an evaluation amount, there is a possibility that the restored image remains blurred (evaluation amount problem).

  Specifically, in the image restoration method using the Wiener filter, when the appearance frequency of the image is lowered, the restoration accuracy is lowered, that is, in the image, there is a problem that the restoration of the edge portion having the low appearance frequency is affected. . In other words, for example, an image photographed with a camera has a problem that it cannot be optimally restored especially at the edge portion because it has many edge portions and strong non-stationarity. However, since the tone, color, or the like does not change, that is, the monotonous background portion in which the dispersion of the image does not change has high steadiness, usually, good restoration accuracy can be obtained.

  The image restoration method using the Kalman filter is a method for solving the problems of the image restoration method using the Wiener filter (the problem of unsteadiness of natural images and the problem of the evaluation amount). While having the advantage of being able to solve the problems of the used image restoration technique, there is a problem that the restoration accuracy is low when the processing target image (observed image / degraded image) is deteriorated due to blur.

  That is, in the image restoration method using the Kalman filter, in step 1, the AR coefficient of the original image is estimated with respect to the target pixel and the surrounding pixels as preprocessing so as to consider the correlation between the target pixel and the surrounding pixels. After that, in step 2, a state space model (state equation and observation equation) is constructed using the AR coefficient estimated in step 1, and specifically, the state equation is obtained from the AR coefficient estimated in step 1 and the original image. The image is reconstructed using the Kalman filter theory (Kalman filter algorithm) by configuring the observation equation from the original image, the blur function, and the noise. Therefore, the image restoration method using the Kalman filter is a process only in the time domain that does not assume stationarity, and is a sequential process using the variance of the estimation error as an evaluation amount. The problem of the method can be solved. Note that the two-step process is necessary in the same way using the one-dimensional Kalman filter and the two-dimensional Kalman filter.

  However, on the other hand, the method using the Kalman filter, whether it is a one-dimensional Kalman filter or a two-dimensional Kalman filter, executes the Kalman filter algorithm in Step 2 using the AR coefficient estimated in Step 1, and therefore, the AR coefficient in Step 1 is determined. There is a problem that the restoration accuracy of the degraded image greatly depends on the estimation accuracy (problem of the AR system). For example, in the case of a digital camera, if the image to be processed includes deterioration based on blurring (such as defocus), it is difficult to determine the AR order and estimate the AR coefficient in step 1, so that the Kalman filter in step 2 is used. The accuracy of image restoration will be affected.

  In this respect, generally, it is difficult to accurately estimate the AR coefficient. This is because accurate estimation of the AR coefficient depends on a clear image (original image) in the case of image restoration, for example. This means that the original image must be known, so real-time processing becomes difficult. Also, even if it becomes possible to accurately estimate the AR coefficient in real time by any method, the problem of computational complexity is inevitable because the processing increases. In the first place, the estimation of the AR coefficient is performed after determining the order of the AR coefficient. However, it is very difficult to determine the order of the AR coefficient, and it is difficult to accurately estimate the AR coefficient from this point. It can be said.

  Therefore, a simple and practical high-performance image restoration method that can solve both the problems of the image restoration method using the Wiener filter and the image restoration method using the Kalman filter, that is, a simple configuration today. There is a need for an image restoration technique that has high image restoration performance and that can be used in a real environment and has high restoration performance (that is, image restoration capability) for degraded images. In particular, the degraded image restoration performance (image restoration capability) requires high performance (capability) that can remove blur and noise at the same time for degraded images that contain blur and noise in a clear image. Yes.

  An object of the present invention is to provide a simple and practical image restoration apparatus and image restoration method capable of realizing a high image restoration capability capable of simultaneously removing blur and noise.

  The image restoration device of the present invention is an image restoration device that estimates the original image information only from the deteriorated image information in which unnecessary information is mixed in the original image information. a correlation calculation unit that calculates a correlation value of an estimation error when the state quantity of the system at time n + 1 including the original image information is estimated from information up to n or time n + 1, and deteriorated image information only at time n , Using the correlation value calculated by the correlation calculation unit, the optimal estimated value of the state quantity at time n + 1 based on the information up to time n + 1 and the optimal estimation of the state quantity at time n + 1 based on the information up to time n A weighting factor calculating unit that calculates a weighting factor for defining the relationship between the value and the estimation error of the observation amount including the deteriorated image information, and the weighting factor calculating unit for the deteriorated image information only at time n In Using a weighting factor calculated, employs a configuration having a best estimate calculation unit for calculating an optimal estimate of the state quantity at time n + 1 by the information until time n or time n + 1.

  The image restoration method of the present invention is an image restoration method for estimating the original image information only from the deteriorated image information in which unnecessary information is mixed in the original image information. a correlation calculation step of calculating a correlation value of an estimation error when the state quantity of the system at time n + 1 including the original image information is estimated from information up to n or time n + 1, and deterioration image information only at time n , Using the correlation value calculated in the correlation calculation step, the optimal estimated value of the state quantity at time n + 1 based on the information up to time n + 1 and the optimal estimated value of the state quantity at time n + 1 based on the information up to time n And a weighting factor calculating step for calculating a weighting factor for defining the relationship between the estimated error of the observation amount including the deteriorated image information, and the weighting factor calculating step for the deteriorated image information only at time n. Using the weight coefficients out, and to have, and the optimum estimated value calculation step of calculating the optimum estimated values of the state quantity at time n + 1 by the information until time n or time n + 1.

  According to the present invention, it is possible to obtain a simple and practical image restoration apparatus and image restoration method capable of realizing a high image restoration capability capable of simultaneously removing blur and noise.

1 is a block diagram showing a configuration of an image restoration apparatus according to Embodiment 1 of the present invention. (A) is a block diagram showing the configuration of the image restoration processing unit in FIG. 1, and (B) is a block diagram showing the configuration of the first restoration processing unit in FIG. 2 (A). Diagram for explaining the process of image degradation Block diagram showing the system configuration of the state space model of Invention Method 1 It is a figure for demonstrating the specific example of formulation of the state equation of invention method 1, especially (A) is a figure which shows the state process of a state space model, (B) is a process target block and its time change The figure which shows an example, (C) is a figure which shows the specific example of a state equation The figure for demonstrating the structure of the state equation of invention method 1 It is a figure for demonstrating the specific example of formulation of the observation equation of invention method 1, and especially (A) is a figure which shows the observation process of a state space model, (B) shows the specific example of an observation equation. Figure It is a figure for demonstrating the structure (general example) of a conventional general observation equation, (A) is a figure which shows the definition of the conventional observation equation, (B) is a state quantity which affects an observation amount. Figure visually showing the range of It is a figure for demonstrating the structure of the observation equation of invention method 1, and (A) is a figure which shows the definition of the observation equation of invention method 1, and (B) is the range of the state quantity which affects an observation amount. Visually showing the Diagram for explaining how to assign the coefficients that make up the observed transition matrix The figure which shows the range of the state quantity which influences each observation amount based on the observation transition matrix shown to FIG. 9 (A) with the allocated coefficient The figure which shows an example of the algorithm of invention method 1 The flowchart which shows the process sequence which performs the algorithm of FIG. Explanatory diagram summarizing invention method 1 visually Diagram for explaining simulation conditions Illustration for explaining the original image "Cameraman" The figure which shows the simulation result (visual evaluation) of the invention method 1 with respect to the original image "cameraman" in contrast with the conventional method The figure which expanded the peripheral area of the circle of the broken line in FIG. The figure which shows the simulation result (visual evaluation) similar to FIG. The figure which expanded the peripheral region of the circle of the broken line in FIG. The figure which shows the simulation result (visual evaluation) of the invention method 1 with respect to the original image "Lena" in contrast with the conventional method The figure which expanded the peripheral region of the circle of the broken line in FIG. The figure which shows the simulation result (objective evaluation) of the invention method 1 with respect to an original image in contrast with the conventional method The figure which shows the simulation result (subjective evaluation) of the invention method 1 with respect to an original image in contrast with the conventional method The flowchart which shows an example of operation | movement of the image restoration apparatus of FIG. The figure for demonstrating an example of the detection method of step S2300 of FIG. The figure for explaining an example of implementation of restoration mode Diagram for explaining another example of implementation of the restoration mode Diagram for visually explaining the degraded image model The figure for demonstrating the state space model of invention method 1 visually The figure for demonstrating the state process of invention method 1 visually The figure for demonstrating the observation process of invention method 1 visually Explanatory drawing showing the state equation and observation equation of Invention Method 1 together The figure for demonstrating the algorithm of invention method 1 The flowchart which shows the process sequence which performs the algorithm of FIG. The figure for demonstrating the subject of the invention method 1 visually It is explanatory drawing which shows the structure of the state equation of the invention method 2 visually, Especially, (A) is a figure which shows the specific example of the state equation of the invention method 1, and (B) is a process target accompanying the change of time. Diagram showing block changes Explanatory drawing visually showing the structure of the observation equation of Invention Method 2 Other explanatory drawing which shows visually the composition of the observation equation of invention method 2 Explanatory drawing showing the state equation and observation equation of Invention Method 2 together The figure for demonstrating the algorithm of the invention method 2 The flowchart which shows the process sequence which performs the algorithm of FIG. Explanatory diagram comparing algorithms of Invention Method 1 and Invention Method 2 The figure for demonstrating the computational complexity evaluation of the invention method 2 The figure for demonstrating the restoration processing speed evaluation of the method 2 of invention Diagram for explaining simulation conditions The figure which shows the simulation result (visual evaluation) of invention method 2 with respect to the original image "cameraman" in contrast with the comparison object method FIG. 47 is an enlarged view of the peripheral area of the broken-line circle in FIG. The figure which shows the simulation result (visual evaluation) of the invention method 2 with respect to the original image "Lena" in contrast with the comparison object method The figure which expanded the peripheral area of the broken-line circle in FIG. The figure which shows the simulation result (objective evaluation) of the invention method 2 with respect to an original image in contrast with a comparison object method The figure for demonstrating the subject of the invention method 2 visually Explanatory drawing visually showing the structure of the observation equation of Invention Method 3 The figure for demonstrating the state space model of invention method 3 visually Explanatory drawing which shows the state equation and observation equation of Invention Method 3 together The figure for demonstrating the algorithm of the invention method 3 The flowchart which shows the process sequence which performs the algorithm of FIG. The figure for demonstrating the calculation amount evaluation of the invention method 3 The figure for demonstrating the restoration processing speed evaluation of the invention method 3 The figure which shows the simulation result (visual evaluation) of the invention method 3 with respect to the original image "cameraman" in contrast with the method for comparison The figure which expanded the peripheral area of the broken-line circle in FIG. The figure which shows the simulation result (visual evaluation) of the invention method 3 with respect to the original image "Lena" in contrast with the comparison object method The figure which expanded the peripheral area of the circle of the broken line in FIG. The figure which shows the simulation result (objective evaluation) of the invention method 3 with respect to an original image in contrast with a comparison object method The figure for explaining the feature of the one-dimensional image restoration technique The figure for demonstrating the characteristic of the two-dimensional image restoration method The figure for demonstrating the state space model of invention method 4 visually The figure for demonstrating the state process of invention method 4 visually Explanatory drawing which shows the structure of the state equation of invention method 4 visually Diagram for visually explaining the observation process of Invention Method 4 Explanatory drawing visually showing the structure of the observation equation of Invention Method 4 The figure which shows the specific example of the observation transition matrix of invention method 4. Explanatory drawing visually showing the structure of the observed transition matrix in FIG. The figure which shows the specific example of the allocation method of the coefficient which comprises the observation transition matrix of FIG. FIG. 72 is another diagram showing a specific example of a method for assigning coefficients constituting the observation transition matrix of FIG. Still another diagram showing a specific example of a method for assigning coefficients constituting the observation transition matrix of FIG. Explanatory drawing which shows the state equation and observation equation of Invention Method 4 together The figure for demonstrating the algorithm of the invention method 4 The flowchart which shows the process sequence which performs the algorithm of FIG. The figure for demonstrating the calculation amount evaluation of the invention method 4 The figure for demonstrating the restoration processing speed evaluation of the invention method 4 The figure which shows the simulation result (visual evaluation) of the invention method 4 with respect to the original image "cameraman" in contrast with the comparison object method The figure which expanded the peripheral area of the broken-line circle in FIG. The figure which shows the simulation result (visual evaluation) of the invention method 4 with respect to the original image "Lena" in contrast with the comparison object method The figure which expanded the peripheral region of the circle of the broken line in FIG. The figure which shows the simulation result (objective evaluation) of the invention method 4 with respect to an original image in contrast with a comparison object method The figure for demonstrating the subject of the invention method 4 visually It is explanatory drawing which shows the structure of the state equation of the invention method 5 visually, Especially, (A) is a figure which shows the specific example of the state equation of the invention method 4, and (B) is a process target accompanying the change of time. Diagram showing block changes Explanatory drawing visually showing the structure of the observation equation of Invention Method 5 Other explanatory drawing which shows visually the composition of the observation equation of invention method 5 Explanatory drawing which shows the state equation and observation equation of Invention Method 5 together The figure for demonstrating the algorithm of the invention method 5 92 is a flowchart showing a processing procedure for executing the algorithm of FIG. Explanatory diagram comparing algorithms of Invention Method 4 and Invention Method 5 The figure for demonstrating the calculation amount evaluation of the invention method 5 The figure for demonstrating the restoration processing speed evaluation of the invention method 5 The figure which shows the simulation result (visual evaluation) of the invention method 5 with respect to the original image "cameraman" in contrast with the comparison object method Fig. 97 is an enlarged view of the area around the dashed circle in Fig. 97 The figure which shows the simulation result (visual evaluation) of the invention method 5 with respect to the original image "Lena" in contrast with the comparison object method The figure which expanded the peripheral region of the circle of the broken line in FIG. The figure which shows the simulation result (objective evaluation) of the invention method 5 with respect to an original image in contrast with a comparison object method The figure for demonstrating the subject of the invention method 5 visually Explanatory drawing which shows visually the composition of the observation equation of invention method 6 The figure for demonstrating the state space model of invention method 6 visually Explanatory drawing which shows the state equation and observation equation of Invention Method 6 together The figure for demonstrating the algorithm of the invention method 6 106 is a flowchart showing a processing procedure for executing the algorithm of FIG. The figure for demonstrating the computational complexity evaluation of the invention method 6 The figure for demonstrating restoration processing speed evaluation of the invention method 6 The figure which shows the simulation result (visual evaluation) of the invention method 6 with respect to the original image "cameraman" in contrast with the comparison object method The figure which expanded the peripheral region of the circle of the broken line in FIG. The figure which shows the simulation result (visual evaluation) of the invention method 6 with respect to the original image "Lena" in contrast with the comparison object method The figure which expanded the peripheral region of the circle of the broken line in FIG. The figure which shows the simulation result (objective evaluation) of the invention method 6 with respect to an original image in contrast with a comparison object method Block diagram showing the configuration of an image restoration apparatus according to Embodiment 2 of the present invention The figure which shows the specific example of the restoration mode which can be designated by the restoration mode designation | designated part of FIG. FIG. 115 is a flowchart showing an example of the operation of the image restoration apparatus.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an image restoration apparatus according to Embodiment 1 of the present invention. Here, a suitable image restoration apparatus to which the image restoration method of the present invention is applied will be described by taking a general-purpose image restoration apparatus that can handle various applications as an example.

  The image restoration apparatus 100 shown in FIG. 1 is configured by a computer, and is roughly divided into an image input apparatus 110, an input interface unit 120, an operation unit 130, an internal interface unit 140, a storage unit 150, an image restoration processing unit 160, and an output. An interface unit 170 and an image output device 180 are included.

  The image input device 110 is an input device for inputting image data (degraded image) to be restored to a computer as digital data. The input image may be a still image or a moving image. As the image input device 110, for example, a camera 112, a scanner 114, a recording medium 116, a modem 118, or the like can be used. The camera 112 means all devices having an imaging function. For example, in addition to a digital camera (digital still camera and digital video camera), a mobile phone equipped with a camera function, a security camera (surveillance camera), an image diagnosis Medical equipment (endoscope, X-ray, echo, CT, MRI, etc.), etc. The scanner 114 is one of typical image input devices, and includes a film scanner that exclusively reads from files such as negatives and positives. The recording medium 116 broadly means a recording medium capable of recording image data. For example, a magnetic disk (HDD, FD, etc.), an optical disk (CD, DVD, BD, etc.), a magneto-optical disk (MO), a flash memory ( Memory card, USB memory, etc.). The modem 118 is a device for connecting to an external communication network (for example, a telephone line, a LAN, the Internet, etc.). Note that the type of the image input device 110 may be appropriately selected according to the application of the image restoration device 100.

  The input interface unit 120 performs input processing such as converting image data provided from the image input device 110 into a data format that can be processed by a computer. Although not shown, the input interface unit 120 is provided separately and independently according to the type of the image input device 110. For example, the input interface unit 120 of the recording medium 116 is also called a drive, and various types of drives can be used depending on the type of the recording medium 116. The drive is a device that reads and writes recording media. As far as the recording media are concerned, the input interface unit 120 and the output interface unit 170 are usually integrated. In general, the modem 118 can function as both the image input device 110 and the image output device 180. Therefore, the input interface unit 120 and the output interface unit 170 are usually integrated with respect to the modem 118 as well. The input interface unit 120 may be a built-in card (board) stored inside the computer main body, or may be an externally installed device connected via the internal interface unit 140.

  When the image input device 110 outputs image information as analog data, the corresponding input interface unit 120 includes a sampling unit and an A / D conversion unit (both not shown). The sampling unit samples the input analog signal at a predetermined sampling frequency and outputs the sampled analog signal to the A / D conversion unit. The sampling frequency can be changed according to the type of restoration process target (information source). The A / D converter performs A / D conversion processing on the amplitude value of the sampled signal with a predetermined resolution.

  The operation unit 130 is typically a keyboard, a mouse, a touch panel, or the like, but a voice recognition device or the like may be used. The user can operate the computer using the operation unit 130 while confirming on a display 182 to be described later, for example. Further, the operation unit 130 includes a parameter setting unit 132, an area specifying unit 134, and a restoration mode specifying unit 136. The parameter setting unit 132 sets various parameter values necessary for image restoration processing according to the present embodiment, which will be described in detail later, by a user input operation. The area designating unit 134 designates an area (a specific range of an image) to be subjected to image restoration processing for the input image by a user input operation. The restoration mode designating unit 136 designates a restoration mode to be described later by a user input operation.

  The internal interface unit 140 is inside the computer main body and has a function of connecting the input interface unit 120, the operation unit 130, the storage unit 150, the image restoration processing unit 160, and the output interface unit 170 to each other. Various signals are exchanged inside the computer via the internal interface unit 140.

  The storage unit 150 includes a main storage device 152 and an auxiliary storage device 154. The main storage device 152 is an element constituting the main body of the computer, and mainly stores programs and data. The auxiliary storage device 154 is a storage device that compensates for the lack of capacity of the main storage device 152. The auxiliary storage device 154 is typically a hard disk (HD), but may be a portable device such as a CD-ROM, DVD, SSD (Solid State Drive), flash memory, or the like. A combination of these may be used. The program (image restoration algorithm) for executing the image restoration process in the present embodiment may be stored in advance in the storage unit 150 (the main storage device 152 or the auxiliary storage device 154), or both the interface units from the recording medium 116. It may be installed in the storage unit 150 via 120 and 140, or may be downloaded to the storage unit 150 from the outside via the modem 118 and both interface units 120 and 140.

  In order to execute a series of processes in which image data is captured from the image input device 110, image restoration processing is performed on the captured image data, and the restored image data is retrieved from the image output device 180. A storage area for storing image data, a storage area temporarily necessary for data processing (called a work area or a work memory), and a storage area for storing image data to be output is necessary. These storage areas are allocated on the main storage device 152 or the auxiliary storage device 154. Here, for convenience of explanation, it is assumed that image data after restoration processing is output to a display 182 to be described later. The display memory 156 is shown separately on the drawing.

  The image restoration processing unit 160 is a characteristic component of the present invention, and executes a built-in image restoration algorithm described later. For example, in a conventional image restoration method using a Kalman filter, a two-step process, that is, an AR order is determined, an AR coefficient is estimated, and then a state space model (state equation and observation equation) is used using the estimated AR coefficient. The image restoration is realized by executing the Kalman filter by configuring the image, whereas the image restoration method of the present invention (hereinafter referred to as “invention method”) is a new prediction method composed of the state equation and the observation equation. The image restoration is realized. Specifically, in the inventive method, by constructing a new state space model (state equation and observation equation) that does not require the concept of the AR system, more specifically, the state equation is clear image information (original image). Information), and using a new state space model consisting of degraded image information, clear image information (original image information), blur information, and noise in the observation equation, image restoration is realized. Yes. The image to be restored may be a still image or a moving image. The image restoration method of the present invention will be described later in detail.

  The image output device 180 is an output device for taking out image data (restored image) restored by the computer (image restoration processing unit 160) to the outside in a predetermined output form. As the image output device 180, for example, a display 182, a printer 184, a recording medium 186, a modem 188, or the like can be used. The recording medium 186 and the modem 188 may be shared with the recording medium 116 and the modem 118 as the image input device 110, respectively. Note that the type of the image output device 180 may be appropriately selected according to the application of the image restoration device 100.

  The output interface unit 170 performs output processing such as converting image data (restored image) restored by the computer (image restoration processing unit 160) into a data format that can be output to the image output device 180. Although not shown, the output interface unit 170 is provided separately and independently according to the type of the image output device 180. As described above, regarding the recording medium and the modem, the input interface unit 120 and the output interface unit 170 are usually integrated. Similarly to the input interface unit 120, the output interface unit 170 may be a built-in card (board) stored inside the computer main body, or an externally installed device connected via the internal interface unit 140. May be.

  2A is a block diagram showing the configuration of the image restoration processing unit 160 in FIG. 1, and FIG. 2B is a block diagram showing the configuration of the first restoration processing unit 160a in FIG. 2A. is there.

  In the present embodiment, as shown in FIG. 2A, the image restoration processing unit 160 includes a first restoration processing unit 160a and a second restoration processing unit 160b. The first restoration processing unit 160a performs the image restoration method of the present invention. The second restoration processing unit 160b performs an image restoration method other than the image restoration method of the present invention, for example, a conventional image restoration method such as an image restoration method using a Wiener filter or an image restoration method using a Kalman filter. As a result, image restoration processing using the first restoration processing unit 160a and the second restoration processing unit 160b can be performed. Hereinafter, as a conventional image restoration method performed by the second restoration processing unit 160b, for example, an image restoration method using a Wiener filter (hereinafter referred to as “conventional method 1”) will be described as an example.

  As shown in FIG. 2B, the first restoration processing unit 160a includes an initial setting unit 162, a correlation calculation unit 164, a weight coefficient calculation unit 166, and an optimum estimated value calculation unit 168. The first restoration processing unit 160a executes a built-in image restoration algorithm (invention technique) in cooperation with each of the units 162 to 168, so that a clear image (original image) is obtained only from the captured image data (degraded image). ). At this time, the initial setting unit 162 performs initial setting of the image restoration algorithm of the inventive method, and the correlation calculation unit 164 performs correlation calculation of the estimation error of the original image (desired information, clear image), and the weight coefficient calculation unit 166 calculates a weighting coefficient necessary for calculating the optimum estimated value of the original image (desired information), and the optimum estimated value calculating unit 168 calculates the optimum estimated value of the original image (desired information). The specific processing contents of the units 162 to 168 will be described later in detail.

  Hereinafter, the image restoration processing operation performed by the first restoration processing unit 160a will be described in detail. Here, first, a degraded image model serving as a theory of image restoration will be described.

  In general, a deteriorated image is obtained by a model obtained by adding noise to a convolution of an original image and a blurred point spread function (PSF: Point Spread Function). That is, a degraded image g (x, y) obtained by an imaging system (an apparatus that captures an object, a system that generates an image, etc.) is f (n, m) representing the original image representing the object, and the point spread of the blur is increased. When the function is represented by h (x, y) and the noise is represented by n (x, y), it is represented by the following equation (1). Note that the blurred point spread function h (x, y) represents the characteristics of the imaging system including imaging conditions.

  FIG. 3 is a diagram for explaining the process of image degradation.

  For example, when the original image shown at the left end of FIG. 3 is blurred, the image shown at the center of FIG. 3 is obtained, and when noise is further added to this image, the deteriorated image shown at the right end of FIG. Blur is caused by a pixel affecting its surrounding pixels, but noise occurs regardless of the pixel. As described above, in the case of an image taken with a camera or the like, blur is caused by camera shake or defocus, and noise is unavoidably caused by dark current or thermal noise. As shown in FIG. 3, restoration of a degraded image is nothing but removing blur and noise from the degraded image.

  In the inventive method, as described above, in order not to use the concept of the AR system, a new state space model (state equation and observation equation) is configured. That is, the state equation is configured using only clear image information (original image information), and the observation equation is configured using degraded image information, clear image information (original image information), blur information, and noise. . In the present embodiment, six specific techniques are presented as inventive techniques. Here, for convenience, the first to sixth invention methods will be referred to as invention method 1, invention method 2, invention method 3, invention method 4, invention method 5, and invention method 6, respectively, and will be described sequentially.

  Invention methods 1 to 6 are all image restoration methods that do not use an AR coefficient, and are simple, practical, and high-performance image restoration methods that can simultaneously remove blur and noise. However, the inventive methods 1 to 6 are different from each other in the calculation amount, the restoration processing speed, the image restoration capability, and the like depending on the algorithm. Here, before explaining each invention method 1-6 in detail, the characteristics, advantages, and disadvantages of each invention method 1-6 will be summarized.

  Invention methods 1 to 3 are one-dimensional image restoration methods, and invention methods 4 to 6 are two-dimensional image restoration methods obtained by extending invention methods 1 to 2 to two dimensions. Invention method 1 is a basic type of the image restoration method according to the present invention. Invention method 1 is an image restoration method that does not use an AR coefficient, and constitutes a state space model having a colored drive source. Inventive method 1 has the advantage that high-performance degraded image restoration is possible compared to the prior art, but has the disadvantage that the amount of calculation is larger than other inventive methods 2 and 3. Invention method 2 is a method of reconstructing a state space model that takes into account only the pixel region of interest in comparison with method 1, and uses only the pixel region of interest to significantly reduce the amount of computation. Invention method 2 has the advantage that it is possible to restore a deteriorated image at high speed (low computational complexity) (image restoration capability is almost equivalent to that of invention method 1). Invention method 3 is a method in which a state space model is reconstructed in consideration of the influence of a colored drive source, compared to invention methods 1 and 2. Invention Method 3 has advantages such as reduction in the amount of computation, improvement in restoration processing speed, application to a colored drive source, and improvement in image restoration capability, and high speed (low computation amount) while improving image restoration capability. This is a technique that can restore a deteriorated image.

  As described above, Invention Method 1 is a one-dimensional image restoration method that does not use an AR coefficient, and has a feature that a high-performance degraded image restoration is possible compared to the prior art. Invention methods 4 to 6 are two-dimensional image restoration methods, and have a feature that the processing efficiency can be improved more flexibly than invention method 1. Invention method 4 is an image restoration method obtained by extending invention method 1 two-dimensionally, and has the advantage that the processing efficiency can be improved by arbitrarily taking the pixel region of interest and the peripheral pixel region, while other methods are available. Compared with the inventive methods 1 to 3, 5 and 6, there is a disadvantage that the amount of calculation is large. Invention method 5 is a method of reconstructing a state space model that takes into account only the pixel region of interest in comparison with method 4, and uses only the pixel region of interest to significantly increase the amount of computation compared to method 4. It is mitigating. Inventive method 5 has the advantage of being able to restore a degraded image at high speed (low computational complexity), but has the disadvantage that the restoration accuracy is slightly lower than in inventive method 4. Invention method 6 is a method of reconstructing a state space model that takes into account the influence of the colored drive source as compared with invention methods 4 and 5, and is an improvement of the disadvantage of invention method 5. The invention method 6 has advantages such as good image restoration capability, application to a colored drive source, reduction of calculation amount, and restoration processing suitable for the pixel to be restored, and practical use considering processing efficiency. This is a degraded image restoration technique. In particular, the inventive methods 4 to 6 in which the inventive methods 1 to 3 are expanded two-dimensionally can arbitrarily select the target pixel region and the peripheral pixel region, so that the state of the deteriorated image and the user's request (processing speed) It is also possible to adapt to the one-dimensional invention methods 1 to 3 according to the processing quality), or to the two-dimensional invention methods 4 to 6 by a slight change. Has characteristics.

Specifically, in the inventive method, a new state space model (state equation and observation equation) is constructed, and this new state space model is expressed by the following equation (2). However, in the state equation, the vector x _p1 (n) is a state vector (original image information), the matrix Φ _p1 is a state transition matrix, and the vector δ _p1 (n) is a drive source vector. In the observation equation, the vector y _p1 (n) is an observation vector (observation image information and degraded image information), the matrix M _p1 is an observation transition matrix, and the vector ε _p1 (n) is an observation noise vector. “N” is the time n of the apparatus. The time n represents the processing order (nth) of the processing target block including the target pixel (or target pixel region) and a plurality of peripheral pixels (or peripheral pixel regions). Here, for convenience of explanation, for example, Invention Method 1 which is a basic type is presented as an invention method. In the following description, the subscript “p1” indicates that the invention method 1 is concerned. In addition, here, the “pixel region” in the target pixel region and the peripheral pixel region means both a case where only one pixel is included and a case where a plurality of pixels are included. That is, as a restoration processing target when restoring a degraded image including blur and noise, one pixel is referred to as a “target pixel”, and a region including at least one or more pixels is referred to as a “target pixel region”. Further, a pixel adjacent to the target pixel is referred to as a “peripheral pixel”, and a region including the pixel adjacent to the target pixel, or a region adjacent to the target pixel and a region that is not adjacent to the target pixel but includes the surrounding pixels, This is referred to as “peripheral pixel region”.

  FIG. 4 is a block diagram showing the system configuration in this state space model.

As shown in FIG. 4, this state space model is composed of a state process and an observation process. A state process is described by a state equation, and an observation process is described by an observation equation. In FIG. 4, “201” is a state vector x _p1 (n) at time n, “202” is a state vector x _p1 (n + 1) at time n + 1, “203” is an observation vector y _p1 (n) at time n, “ 204 ”is the observed noise vector ε _p1 (n) at time n,“ 205 ”is the drive source vector δ _p1 (n + 1) at time n + 1,“ 206 ”is the state transition matrix Φ _p1 , and“ 207 ”is the observed transition matrix M _p1. It is. The state equation in equation (2) describes the system to be observed with a state space model, and is an internal state, that is, a state variable (here, state vectors x _p1 (n), x _p1 (n + 1)) with respect to time. It shows the generation process. The observation equation in equation (2) describes the process of observation through some kind of observation device, and the observation result (here, the observation vector y _p1 (n)) is the observed quantity, that is, the input (here, It shows how it is generated depending on the state vector x _p1 (n)).

  FIG. 5 is a diagram for explaining a specific example of the formulation of the state equation according to the method 1 of the invention. In particular, FIG. 5A shows a state process of the state space model, and FIG. FIG. 5C is a diagram showing an example of the processing target block and its time change, and FIG. 5C is a diagram showing a specific example of the state equation.

  In Invention Method 1, in the degraded image restoration process, not only the target pixel to be restored is used, but also the surrounding pixels are processed. That is, an area of K × K (where J> K) around a pixel of interest in an image of J × J size is set as a processing target block, and pixel information in the processing target block is used. To restore the image. Thus, the processing target block means a range in which processing is performed using K × K pixels in image restoration.

For example, when J = 256 and K = 3, as shown in FIG. 5, the pixel area of 3 × 3 around the pixel of interest in the 256 × 256 size image is the processing target block. Become. In the drawing, the block to be processed is shaded. The image restoration process is performed using pixel information in the 3 × 3 processing target block. For convenience of explanation, as shown in FIG. 5B, assuming that some pixels of an image having a size of 256 × 256 are numbered 1 to 36, the state vector x _p1 (n) at time n The processing target block corresponding to is composed of 9 pixels of 3 × 3 with numbers 1, 2, 3, 7, 8, 9, 13, 14, 15, and the state vector x _p1 at the next time n + 1. The processing target block corresponding to (n + 1) includes 9 pixels of 3 × 3 with numbers 7, 8, 9, 13, 14, 15, 19, 20, and 21 as constituent elements.

At this time, the state equation in the equation (2) is defined by the following equation (3) also shown in FIG. Here, the state vector x _p1 (n) is nine pieces of pixel information x ₁ (n), x ₂ (n), and x ₃ (x ₃ (n) included in a 3 × 3 processing target block as a state quantity including original image information. n), x ₄ (n), x ₅ (n), x ₆ (n), x ₇ (n), x ₈ (n), x ₉ (n), a 9 × first-order vector . The state transition matrix Φ _p1 is a 9 × 9 order matrix defined by the equation (3), and the drive source vector δ _p1 (n) is a 9 × 1 order matrix similarly defined by the equation (3). Is a vector.

  FIG. 6 is a diagram for explaining the configuration of the state equation represented by Expression (3).

The feature of the state equation represented by the equation (3) is that a part of the state transition matrix Φ _p1 is set to “1” and the remaining elements are all set to “0”, and the drive source vector A part of δ _p1 (n) is a point represented by a state quantity x _i (n) (i = 7, 8, 9) which is a colored signal. As shown in FIG. 6, the current state quantity (pixel information of the original image) x ₁ (n + 1), in the portion where the processing target block at the current time n + 1 and the processing target block at the past time n overlap. x ₂ (n + 1), x ₃ (n + 1), x ₄ (n + 1), x ₅ (n + 1), x ₆ (n + 1) and past state quantities (pixel information of the original image) x ₄ (n), x ₅ ( This is because n), x ₆ (n), x ₇ (n), x ₈ (n), and x ₉ (n) are associated with each other. As a result, the state equation representing the relationship between x _p1 (n) and x _p1 (n + 1) includes a state vector x _p1 (n) composed of a clear image, a state transition matrix Φ _p1 composed of 0 and 1, and a colored signal. Since it is composed of a drive source vector δ _p1 (n) consisting of a clear image, it is composed only of a desired state quantity (original pixel information), that is, clear image information (original image information). It will be.

  FIG. 7 is a diagram for explaining a specific example of the formulation of the observation equation of the invention method 1. In particular, FIG. 7A is a diagram showing the observation process of the state space model, and FIG. It is a figure which shows the specific example of an observation equation.

Corresponding to the examples of FIGS. 5 and 6, the observation equation in the equation (2) is defined by the following equation (4) also shown in FIG. Here, the observation vector y _p1 (n) is nine pieces of pixel information y ₁ (n), y ₂ (n), y ₃ (y) included in the 3 × 3 processing target block as an observation amount including degraded image information. n), y ₄ (n), y ₅ (n), y ₆ (n), y ₇ (n), y ₈ (n), y ₉ (n), a 9 × 1 order vector . The observed transition matrix M _p1 is a 9 × 9 order matrix defined by the equation (4), and corresponds to the blur point spread function (PSF) in the image degradation model. Each element h _{r, s} (r, s is the coordinate of h, r, s = -1, 0, 1) constituting the observation transition matrix M _p1 is known and is appropriately set in advance as data. Has been. The observation noise vector ε _p1 (n) is the observation noise ε ₁ (n), ε ₂ (n), ε ₃ (n), ε corresponding to nine pixels included in the 3 × 3 processing target block. ₄ (n), ε ₅ (n), ε ₆ (n), ε ₇ (n), ε ₈ (n), and ε ₉ (n) are 9 × 1 order vectors.

FIG. 8 is a diagram for explaining the configuration (general example) of a conventional general observation equation, and FIG. 9 is a diagram for explaining the configuration of the observation equation of Invention Method 1 represented by Expression (4). FIG. In particular, FIG. 8A is a diagram showing the definition of a conventional general observation equation, and FIG. 8B is a diagram visually showing the range of state quantities that affect the observed quantity. FIG. 9A is a diagram showing the definition of the observation equation of Invention Method 1, and FIG. 9B is a diagram visually showing the range of state quantities that affect the observed quantity. In the description of FIGS. 8 and 9, the observation noise vector ε _p1 (n) is omitted for convenience.

Deterioration due to blur in an image means that a certain pixel deteriorates under the influence of surrounding pixels as described above. Therefore, conventionally, an observation equation is generally defined as shown in FIG. In this case, the blur is caused by being affected by all surrounding pixels. That is, each pixel in the processing target block is affected by all the pixels in the processing target block. In other words, each observation amount y _i (i = 1, 2,..., 9) is affected by all the state amounts x _i (i = 1, 2,..., 9). When this is shown visually, for example, it is as shown in FIG. That is, as shown in FIG. 8B, for example, both the observation amount y ₁ and y ₂ are affected by all 3 × 3 (= 9) state quantities x _{1 to} x _9. It will be.

On the other hand, the characteristic of the observation equation of the invention method 1 represented by the equation (4) is that some elements of the observation transition matrix M _p1 are expressed by 3 × 3 (= 9) coefficients h _{r, s} (coordinates). r, s = −1, 0, 1), and the remaining elements are all set to “0”. For example, FIG. 9A shows a part of expressions (expressions relating to y ₁ (n), y ₂ (n)) obtained by expanding the observation equation. The method for assigning the coefficients h _{r, s} to the observed transition matrix M _p1 is as follows. That is, nine coefficients h _{r, s} (r, s = −1, 0, 1) are arranged in a matrix as shown in FIG. 10 (hereinafter referred to as “coefficient matrix”), and the center of the coefficient matrix (that is, When the position of the coefficient h _0,0 ) is matched with the position of the target pixel i of the observation amount y _i (i = 1, 2,..., 9), a part of the observed transition matrix M _p1 Factors h _{r, s} (r, s = -1, 0, 1) are assigned to the elements. This is shown visually in FIG. 9B, for example. That is, as shown in FIG. 9 (B), for example, if the observed amount of _{y 1, 3 × 3 (=} 9) of the number of state quantities _x 1 ~x _9, 5 pieces of _x 1 ~x ₅ When the observation amount is y ₂ , it is affected by _six x ₁ to x ₆ .

FIG. 11 shows the range of state quantities x _i that affect each observed quantity y _i based on the observed transition matrix M _p1 shown in FIG. 9 (A), and assigned coefficients h _{r, s} (r, s = −1). , 0, 1). Here, for convenience of explanation, in each of FIGS. 11 (A) to 11 (I), the left processing target block shows only the observation amount y _{i of} interest, and the right processing target block shows , Only h _{r, s} to be multiplied by this state quantity x _i is shown at the location of each state quantity x _i that has an influence.

In the observation equation of the formula (4), the observed quantity _{y 1,} as shown in FIG. 11 (A), influenced by the five state quantities _x 1 ~x _5, observation quantity _{y 2,} as shown in FIG. As shown in FIG. 11 (B), the observation quantity y ₃ is affected by the _six state quantities x _{1 to} x ₆ , and the observed quantity y ₃ is changed into _seven state quantities x _{1 to} x 7 as shown in FIG. Affected by. Further, as shown in FIG. 11D, the observation quantity y ₄ is affected by _eight state quantities x _{1 to} x ₈ , and the observation quantity y ₅ is 9 as shown in FIG. 11E. affected all state quantity _x 1 ~x ₉ pieces, observation quantity _{y 6,} as shown in FIG. 11 (F), influenced by the eight state quantity _x 2 ~x _9. Further, as shown in FIG. 11 (G), the observation amount y ₇ is affected by seven state quantities x _{3 to} x ₉ , and the observation amount y ₈ is 6 as shown in FIG. 11 (H). affected by number of state quantities _x 4 ~x _9, observation quantity _{y 9,} as shown in FIG. 11 (I), influenced by the five state quantities _x 5 ~x _9.

  FIG. 12 is a diagram showing an example of the algorithm of Invention Method 1. This algorithm can be applied not only to still images but also to moving images, regardless of the type of image.

  As shown in FIG. 12, the algorithm of the invention method 1 is roughly divided into an initialization process and an iteration process, and the iterative process is a new state space model (state equation and observation equation). It is composed based on. In the iteration process, steps 1 to 6 are repeated sequentially.

  FIG. 13 is a flowchart showing a processing procedure for executing the algorithm of FIG.

First, the initial setting unit 162 performs initial setting (S1000). Specifically, in the initial setting unit 162, an initial value x _{p1 of} an optimum estimated value of a state vector, that is, a desired signal (original image signal) vector that is a state quantity (hereinafter referred to as an “optimum estimated value vector of a desired signal”). (0 | 0), the initial value P _p1 (0 | 0) of the correlation matrix of the estimation error of the desired signal vector (hereinafter referred to as “estimation error vector of the desired signal”), and the covariance value R _δp1 (n ) [I, j] and the covariance value R _εp1 (n) [i, j] of the observed noise vector are set as shown in the following equation (5). Although not shown in FIG. 12, the state transition matrix Φ _p1 and the observed transition matrix M _p1 are set as shown in Expression (3) and Expression (4), respectively. Here, the initial value of the counter at time n is set to “0”. In addition, when a vector and a matrix element are indicated, the i-th element of the vector a (n) is expressed as a (n) [i], and the i-th row and j-th column element of the matrix A (n) is A (n). It shall be written as [i, j]. In FIG. 12, the optimum estimated value vector x _p1 of the desired signal is expressed as a hat x _p1 .

However, the vector 0 _K is a K-dimensional zero vector, and the matrix I _K is a K-th order unit matrix. K is the dimension of the processing target block (the target pixel region and the peripheral pixel region) having a size of K × K. In the case of K = 3, K ² −K = 6, and “i, j> K ² −K” corresponds to “i, j = 7, 8, 9”. E [δ _p1 (n), δ _p1 ^T (n)] is an autocorrelation matrix of the drive source vector δ _p1 (n). E [ε _p1 (n) ε _p1 ^T (n)] [i, j] is an autocorrelation matrix of the observed noise vector ε _p1 , here it is equal to σ ² _ε [i, j] and is assumed to be known. ing. “Known” here means obtained and given by another arbitrary method (algorithm). If the observed noise vector ε _p1 (n) is white noise and is zero average, σ ² _ε is given by a variance of a predetermined number of samples.

Next, the correlation calculation unit 164 calculates an error covariance matrix of n → (n + 1), that is, a correlation value (matrix) (hereinafter also referred to as “correlation matrix”) of an estimation error of n → (n + 1) (S1100). ). Specifically, the correlation calculation unit 164 calculates a correlation matrix P _p1 (n + 1 | n) of an error (estimated error vector of a desired signal) when a desired signal vector at time n + 1 is estimated from information up to time n. To do. This calculation is performed by setting the value of the state transition matrix Φ _p1 and the covariance R _δp1 (n + 1) [i, j] of the state transition matrix Φ _p1 and the driving source vector set in Step S1000, and the value set in Step S1000 (when n = 0) or Using the correlation matrix P _p1 (n | n) of the estimation error vector of the desired signal calculated in the previous step S1600 (when n ≧ 1), the following equation (6) is used. This step S1100 corresponds to procedure 1 of the iterative process of FIG.

Next, the weighting factor calculation unit 166 calculates a weighting factor (matrix) (S1200). Specifically, the weighting factor calculation unit 166 multiplies the estimation error of the observation signal vector that is an observation amount (hereinafter referred to as “estimation error vector of the observation signal”) by a weighting factor (matrix), and uses the information up to time n. The addition of the optimum estimated value vector x _p1 (n + 1 | n) of the desired signal at time n + 1 becomes the optimum estimated value vector x _p1 (n + 1 | n + 1) of the desired signal at time n + 1 based on the information up to time n + 1. The weight coefficient matrix K _p1 (n + 1) is calculated. This calculation is performed by calculating the correlation matrix P _p1 (n + 1 | n) of the estimation error vector of the desired signal calculated in step S1100, and the covariance R _εp1 (n + 1) of the observation transition matrix M _p1 and the observation noise vector set in step S1000, respectively. ) Using the value of [i, j], the following equation (7) is used. This step S1200 corresponds to procedure 2 of the iterative process of FIG.

Next, the optimum estimated value calculation unit 168 calculates the optimum estimated value (vector) of the state quantity (desired signal) of n → (n + 1) (S1300). Specifically, the optimum estimated value calculation unit 168 calculates an optimum estimated value vector x _p1 (n + 1 | n) of the desired signal at time n + 1 based on information up to time n. This calculation is performed by the following equation (8) using the state transition matrix Φ _p1 set in step S1000 and the optimum estimated value vector x _p1 (n | n) of the desired signal calculated in the previous step S1400. . This step S1300 corresponds to procedure 3 of the iterative process of FIG.

Next, the optimum estimated value calculation unit 168 calculates the optimum estimated value (vector) of the state quantity (desired signal) from (n + 1) → (n + 1) (S1400). Specifically, the optimum estimated value calculation unit 168 calculates an optimum estimated value vector x _p1 (n + 1 | n + 1) of a desired signal at time n + 1 based on information up to time n + 1. This calculation includes an optimal estimated value vector x _p1 (n + 1 | n) of the desired signal calculated in step S1300, a weight coefficient matrix K _p1 (n + 1) calculated in step S1200, an observation transition matrix M _p1 set in step S1000, and Using the observation signal vector y _p1 (n + 1) at time n + 1, the following equation (9) is used. This step S1400 corresponds to procedure 4 of the iterative process of FIG.

  Next, it is determined whether or not to end the process (S1500). This determination is made, for example, by determining whether the time n has reached a predetermined number N of samples. As a result of this determination, when the time n has not reached the predetermined number of samples N (S1500: NO), the process proceeds to step S1600, and when the time n has reached the predetermined number of samples N (S1500: YES), The process proceeds to step S1800. Note that the criterion for determination is not limited to the above example. For example, when processing is performed in real time, the processing may be terminated when there are no more samples even if the time n has not reached the predetermined number of samples N.

In step S1600, the correlation calculation unit 164 calculates an error covariance matrix of (n + 1) → (n + 1), that is, a correlation value (matrix) of an estimation error of (n + 1) → (n + 1). Specifically, the correlation calculation unit 164 calculates a correlation matrix P _p1 (n + 1 | n + 1) of an error (estimated error vector of the desired signal) when the desired signal vector at time n + 1 is estimated from information up to time n + 1. To do. This calculation is performed by calculating the correlation matrix P _p1 (n + 1 | n) of the weighting coefficient matrix K _p1 (n + 1) calculated in step S1200, the observed transition matrix M _p1 set in step S1000, and the estimated error vector of the desired signal calculated in step S1100. ) Using the following equation (10). This step S1600 corresponds to procedure 5 of the iterative process of FIG.

  In step S1700, the counter at time n is incremented by 1 (n = n + 1), and the process returns to step S1100.

On the other hand, in step S1800, the calculation result of this algorithm is temporarily stored as an output value. Specifically, the optimum estimated value vector x _p1 (n + 1 | n + 1) of the desired signal calculated in step S1400 is temporarily stored in the image restoration processing unit 160 or the storage unit 150 as an output value of this algorithm.

  FIG. 14 is an explanatory diagram in which Invention Method 1 is visually summarized. Thus, in Invention Method 1, since a new state space model (state equation and observation equation) is configured, it is possible to perform image restoration processing in a one-step process. This is one of the major features of the present invention, including other inventive methods 2 to 6.

  Here, in contrast with the conventional image restoration method using the Kalman filter, the features and effects of the inventive method 1 will be described together.

  In the conventional image restoration method using the Kalman filter, as described above, the two-stage process (after determining the AR order and estimating the AR coefficient in Step 1, using the estimated AR coefficient in Step 2, the state is determined. Image restoration is realized by a spatial model (a state equation and an observation equation are formed and a Kalman filter is executed). Therefore, it is easily expected that the image restoration capability by the Kalman filter in step 2 is greatly influenced by the determination of the AR order in step 1 and the estimation accuracy of the AR coefficient. On the other hand, in the method 1 of the invention, a new state space model (state equation and observation equation) that does not require the concept of the AR system is constructed, and a high-performance image is obtained by a new one-step prediction method using the model. Restore is realized. In Invention Method 1, since the number of processing steps can be reduced by one step, the amount of calculation can be reduced, and consequently the circuit scale can be reduced and the memory capacity can be reduced.

  In addition, in the conventional image restoration method using the Kalman filter, the determination of the order of the AR coefficient that is required when the AR coefficient is estimated in Step 1 becomes a serious problem. Generally, since the order of the AR coefficient depends on the state quantity, it is difficult to theoretically accurately determine the order of the AR coefficient unless the state quantity is known. This means that the state quantity must be known, and real-time processing becomes difficult. Moreover, since the order of the AR coefficient that is not accurate is used, it is difficult to estimate the AR coefficient accurately. Therefore, this is a major factor that degrades the image restoration capability of the conventional image restoration method using the Kalman filter. Even if the AR order and the AR coefficient can be estimated in real time by some technique, an increase in the calculation amount is unavoidable due to an increase in the number of processing steps. In contrast, Invention Method 1 does not require an AR system concept, and thus does not cause such a problem.

  In the conventional image restoration method using the Kalman filter, the state quantity is modeled by expressing it using the AR system. This means that the conventional image restoration method using the Kalman filter can be applied only to state quantities that can be modeled by the AR system. That is, the conventional image restoration method using the Kalman filter cannot be applied to state quantities that are difficult to model in the AR system. On the other hand, Invention Method 1 does not require the concept of the AR system, and thus there is no such restriction regarding the application target.

  In the conventional image restoration method using the Kalman filter, the Kalman filter theory is applied on the assumption that the drive source of the state equation is a white signal and the state quantity and the observation noise are uncorrelated. On the other hand, in Invention Method 1, although the driving source is a colored signal (clear image), the algorithm of Invention Method 1 can be executed by a special configuration of the state equation and the observation equation. This means that Invention Method 1 can be executed without considering general application conditions of Kalman filter theory. That is, it can be said that Invention Method 1 has a wider application range than Kalman filter theory.

  Therefore, Invention Method 1 that does not require the concept of the AR system can restore scenes where re-taking of images is not permitted, for example, instantaneous restoration of images of the heart or lungs in the medical field, restoration of old documents that have deteriorated due to dirt, etc. And technology that can contribute to character / object recognition.

  In addition, the present inventor conducted a simulation to verify the effect of the present invention (effectiveness of Invention Method 1). Specifically, in order to evaluate the image restoration ability of the inventive method 1, (1) visual evaluation, (2) objective evaluation, and (3) subjective evaluation were performed. The visual evaluation is an evaluation in which an original image and a restored image are compared visually. Objective evaluation is numerical evaluation. Subjective evaluation is an interview survey. Hereinafter, these will be described in order.

  First, simulation conditions will be described.

  FIG. 15 is a diagram for explaining the simulation conditions.

  In this simulation, two images shown in FIG. 15, ie, (a) “Cameraman” and (b) “Lenna” were used as original images. Further, a two-dimensional Gaussian function shown in FIG. 15 was used as a model of a point spread function (PSF) corresponding to the blur added to the original image, and additive white Gaussian noise was used as noise. The signal-to-noise ratio SNR was 20 dB. In addition, for comparison, simulations are performed under the same simulation conditions as in the conventional method 1 (image restoration method using a Wiener filter), the conventional method 2 (image restoration method using a projection filter), and the conventional method. 3 (image restoration method using Kalman filter) and Invention method 1. For example, as shown in FIG. 16, the image “cameraman” has a strong steadiness because the dispersion of the image is not changed in the sky portion, and is non-stationary because the dispersion of the image is changed in the head portion of the person. It can be said that the nature is strong.

(1) Visual Evaluation FIG. 17 is a diagram showing a simulation result (visual evaluation) for the original image “cameraman”, and FIG. 18 is an enlarged view of a peripheral area of a broken-line circle in FIG.

  As well shown in FIG. 18, the conventional method 1 has almost no blur. In the conventional method 2, the blur is removed more than in the conventional method 1, but the blur still remains as compared with the original image. On the other hand, the conventional method 3 is a restored image different from the original image, although the blur is removed and the image is clearer than the conventional methods 1 and 2. In particular, the conventional method 3 is darker as a whole than the original image, and the sky portion is deteriorated more than the deteriorated image.

  In contrast, it can be confirmed that the inventive method 1 faithfully restores the original image as compared with the conventional methods 1, 2, and 3, as shown in FIG. That is, the effectiveness of the inventive method can also be confirmed in the enlarged image of FIG.

  FIG. 19 is a diagram showing a simulation result (visual evaluation) for the original image “cameraman” similar to FIG. 17, and FIG. 20 is an enlarged view of a peripheral area of a broken-line circle in FIG. 19.

  As shown well in FIG. 20, when attention is paid to the tripod portion of the camera, the conventional method 1 does not remove any blur. In the conventional method 2, the blur is removed from the conventional method 1, but the original image is not restored. On the other hand, the conventional method 3 seems to have removed the blur as compared with the conventional methods 1 and 2, but the entire image is rough, and it can be confirmed that it is a restored image far from the original image.

  On the other hand, it can be confirmed that the inventive method 1 restores the original image more faithfully than the conventional methods 1, 2, and 3, as well shown in FIG. That is, the effectiveness of Invention Method 1 can be confirmed also in the enlarged image of FIG.

  FIG. 21 is a diagram showing a simulation result (visual evaluation) for the original image “Lena”, and FIG. 22 is an enlarged view of a peripheral region of a broken-line circle in FIG.

  As well shown in FIG. 22, in the conventional method 1, the blur is hardly removed, and the restored image is generally whiter than the original image (the luminance is increased). In contrast, the conventional method 2 has the blur removed as compared with the conventional method 1, but the blur has not been removed up to the original image. On the other hand, the conventional method 3 is clearer than the conventional methods 1 and 2 and appears to be clear, but has deteriorated more than the deteriorated image, and is an image far from the original image. . This point is particularly noticeable when paying attention to the skin area.

  On the other hand, it can be confirmed that the inventive method 1 faithfully restores the original image as compared with the conventional methods 1, 2, and 3, as shown in FIG. That is, the effectiveness of Invention Method 1 can also be confirmed in the enlarged image of FIG.

(2) Objective evaluation (numerical evaluation)
FIG. 23 is a diagram illustrating a simulation result (objective evaluation) for an original image.

  Here, in order to evaluate the image restoration capability of the conventional methods 1 to 3 and the invention method 1 by numerical values, the image restoration capability is evaluated using PSNR [dB] expressed by the following equation (11) shown in FIG. did. PSNR (Peak Signal-to-Noise Ratio) is the ratio of noise to image signal, and the larger the value, the less degradation (blur, noise, etc.), and the better the image.

In this case, as shown in FIG. 23, it can be confirmed that the invention method 1 has a larger value of SNR _OUT than the conventional methods 1, 2, and 3 for both the “cameraman” and “Lena” images. Thus, it can be seen that the inventive method 1 has higher image restoration capability than the conventional methods 1, 2, and 3 in terms of objective evaluation.

(3) Subjective evaluation (interview survey)
FIG. 24 is a diagram illustrating a simulation result (subjective evaluation) for an original image.

  Here, in order to evaluate the image restoration ability of the conventional methods 1 to 3 and the inventive method 1, a subjective evaluation was performed by an interview survey. The performance evaluation of the image restoration was performed by an interview survey using a 5-step MOS (average opinion value) based on ACR (absolute category evaluation). The MOS evaluation criteria are as shown in FIG. Fifty listeners evaluated images obtained by image restoration (see FIGS. 17 to 22). Each listener determines an evaluation value “1” to “5”. The evaluation value “5” is the best.

  As shown in FIG. 24, it can be confirmed that the MOS method has a higher evaluation than the conventional methods 1, 2, and 3 for the “cameraman” and “Lena” images. Thus, it can be seen that Invention Method 1 has higher image restoration capability than Subject Methods 1, 2, and 3 in terms of subjective evaluation.

  From the above experimental results, it can be seen that the image restoration method (invention technique 1) of the present invention exhibits higher image restoration ability than the conventional techniques 1 to 3. In particular, as is apparent from the visual evaluations of FIGS. 17 to 22, it can be seen that the image restoration method of the present invention has significantly higher restoration accuracy than the conventional methods 1 to 3 at edge portions having strong non-stationarity. .

  Next, the operation of the image restoration apparatus 100 having the above configuration will be described using the flowchart shown in FIG. The flowchart shown in FIG. 25 is stored as a control program in the storage unit 150 (the main storage device 152 or the auxiliary storage device 154), and is executed by a CPU (not shown).

  First, in step S2000, image data (degraded image) to be restored is captured from the image input device 110 and stored in a predetermined storage area of the storage unit 150 (the main storage device 152 or the auxiliary storage device 154).

  In step S2050, the image data captured in step S2000 is written into the display memory 156 and displayed on the display 182.

  In step S2100, the area designation unit 134 of the operation unit 130 performs area designation processing. Specifically, when an area to be subjected to image restoration processing (a specific range of an image) is specified in the image displayed on the display 182 in step S2050 by the user's input operation, the specified area is displayed. Generate data. The area is specified by, for example, a pointer on the screen. If the area is not designated by the user, the entire displayed image is treated as designated.

  In step S2150, it is determined whether or not enlargement processing is performed. This determination is made based on the designated area data generated in step S2100, for example. Specifically, when the designated area is smaller than the entire displayed image, it is determined that the enlargement process is performed. As a result of this determination, when the enlargement process is performed (S2150: YES), the process proceeds to step S2200, and when the enlargement process is not performed (S2150: NO), the process immediately proceeds to step S2250.

  In step S2200, the area specified in step S2100 is enlarged. Specifically, for example, the enlargement process is performed so that the designated area has a size corresponding to the entire displayed image. The result of the enlargement process is written in the working memory of the storage unit 150 (the main storage device 152 or the auxiliary storage device 154). Even when the enlargement process is not performed, the image data is written in the working memory.

  In step S2250, a processing target block at time n (for example, a size of 3 × 3) is selected.

  In step S2300, it is determined whether an abrupt change region of the image is detected for the processing target block selected in step S2250. The sudden change region of the image corresponds to, for example, an edge portion of the image. Specifically, for example, with respect to image data written in the work memory (regardless of whether or not enlargement processing is performed), pixel data is processed on the processing target block (3 × 3 size) selected in step S2250. By sequentially scanning, it is detected whether there is a sudden change point of pixel data. As a result of this determination, if an abrupt change region of the image is detected (S2300: YES), the process proceeds to step S2350. If an abrupt change region of the image is not detected (S2300: NO), step S2400 is performed. Proceed to

  An example of a specific detection method is as follows. The absolute value of the difference between an average value y ′ (n−1) of a certain amount of past observation amounts and the current observation amount y (n), that is, | y ′ (n−1) −y (n) | Is compared with a threshold value α. If the value is greater than or equal to the threshold value α, that is, if | y ′ (n−1) −y (n) | ≧ α, it is determined that there is a sudden change point of pixel data, and the value is Is less than the threshold value α, that is, if | y ′ (n−1) −y (n) | <α, it is determined that there is no abrupt pixel data change point. For example, in FIG. 26, the average value y ′ (n−1) of the observation amount of the predetermined portion 302 in the processing target block 301 at time n−1 and the observation amount of the pixel portion 304 of interest in the processing target block 303 at time n. Compare y (n).

  In step S2350, image restoration processing by the inventive method is performed on the processing target block selected in step S2250. As described above, the image restoration process according to the invention technique can be performed with high accuracy even when there is a sudden change point of pixel data, that is, when an edge portion or the like is included. An example of the detailed procedure of the image restoration process according to the inventive method is as described with reference to the flowchart of FIG.

  On the other hand, in step S2400, image restoration processing by an image restoration method other than the inventive method is performed on the processing target block selected in step S2250. As another image restoration method, any image restoration method including an image restoration method using a Wiener filter, an image restoration method using a projection filter, an image restoration method using a Kalman filter, and the like can be used. This is because when there is no abrupt change point of pixel data, that is, when an edge portion or the like is not included, restoration processing can be performed with high accuracy by other image restoration methods.

  In step S2450, the restoration processing result in step S2350 or the restoration processing result in step S2400 is sequentially stored in the working memory of the storage unit 150 (the main storage device 152 or the auxiliary storage device 154).

  In step S2500, the counter value at time n is incremented by one.

  In step S2550, it is determined whether the image restoration process for one page is completed. This determination is made based on the value of the counter at time n. As a result of this determination, if the image restoration process for one page is completed (S2550: YES), the process proceeds to step S2600, and if the image restoration process for one page is not completed (S2550: NO), step The process returns to S2250.

  In step S2600, since the image restoration process for one page is completed, the display memory 156 is updated. In other words, the display memory 156 is updated when the image restoration process for one page is completed.

  In step S2650, it is determined whether image restoration processing has been completed for all pages of the image data captured in step S2000. As a result of this determination, if the image restoration process for all pages has not been completed (S2650: NO), the process returns to step S2250, and if the image restoration process for all pages has been completed (S2650: YES), A series of processing ends.

Next, a specific example of setting and implementation of the observation transition matrix M _p1 corresponding to the point spread function (PSF) that is blur information will be described with reference to FIGS. 27 and 28. FIG.

  FIG. 27 is a diagram for explaining an example of implementation of the restoration mode.

As described above, the observation transition matrix M _p1 is a 9 × 9 order matrix defined by the equation (4), and corresponds to the blur point spread function (PSF) in the image degradation model. Each element h _{r, s} (r, s = −1, 0, 1) constituting the transition matrix M _p1 is known, and is appropriately set in advance as data. That is, in the present embodiment, the degree of blur varies depending on the image, so that several combinations of values of each element h _{r, s} of the observation transition matrix M _p1 that is so-called blur information are set in advance as a restoration mode and used. The user can arbitrarily specify the restoration mode via the restoration mode designation unit 136 of the operation unit 130. For example, in the example of FIG. 27, an appropriate observation transition matrix M _p1 suitable for each shooting mode is set in advance for each shooting mode such as a night view mode or a sport mode that is often set for a digital camera or the like. . Therefore, the user can use the optimum restoration mode by switching the photographing mode. Note that the restoration mode (values of the elements h _{r and s} of the observed transition matrix M _p1 ) itself can be readjusted automatically or manually.

  FIG. 28 is a diagram for explaining another example of implementation of the restoration mode.

In the example of FIG. 28, since the degree of blur varies depending on the image, first, as shown in FIG. 28A, after selecting a deteriorated image to be processed on the display 182, as shown in FIG. Several restored images obtained by changing the observed transition matrix M _p1 (values of the elements h _{r and s} ) are displayed on the display 182. At this time, for example, when the user selects an optimal restored image from a plurality of restored images displayed on the display 182 via the operation unit 130 (such as a touch panel), the restored image is determined. In addition, it is possible to automatically select a restoration mode (values of the elements h _{r and s} of the observation transition matrix M _p1 ) according to the user's preference by accumulating the user's selection history as data. Become.

  As described above, according to the present embodiment, a new state space model (state equation and observation equation) that does not require the concept of the AR system is constructed, and image restoration is performed by a new prediction method of one-stage processing. Thus, practical and high-performance image restoration can be realized. In other words, it has a simple configuration that does not require the steps of determining the AR order and estimating the AR coefficient, and has a utility that can effectively perform restoration processing even for a natural image with strong non-stationarity, In addition, the image restoration capability can be improved as compared with the conventional methods 1 to 3.

  Further, in the present embodiment, as described above, since the concept of the AR system is not required, the present invention can be widely applied to scenes in which retaking of images is not permitted. Of course, here, as described above, the type of image is not limited and may be a still image or a moving image.

  For example, scratches and dirt on old documents are regarded as blurs and noises, and they are removed using Invention Method 1 and are effective in decoding and restoring old documents and creating a digital database by using character / object recognition technology. Expected to demonstrate.

  Also in the crime prevention field, the installation of security cameras (surveillance cameras) has been spreading in recent years, and many criminals are often taken by security cameras. May be a clue. However, these images generally have poor image quality, and the images of moving objects are generally blurred. Further, when these images are enlarged, the images are usually further deteriorated. Therefore, it is expected that by applying Invention Method 1 to these deteriorated images including enlarged images, the blur and noise of the deteriorated images are removed and a clear image is provided, thereby leading to early detection of criminals. The In this way, by applying Invention Method 1 to a deteriorated image including an enlarged image, it is not limited to the crime prevention field to provide a clear image by removing the blur and noise of the deteriorated image, such as a surveillance camera. The present invention can also be applied to the case where accident determination, instrument failure diagnosis, and the like are performed based on the images taken in step (1).

  In recent years, mobile phones with a camera function, digital cameras, and the like have spread rapidly and have become one-person era. However, the image restoration technology installed in these products is a technology for preventing blur and noise such as face recognition and filters, and does not assume a scene where re-shooting is not allowed. Therefore, by applying Invention Method 1 to these products, instantaneous image restoration that does not allow re-taking is possible.

  In the medical field, as one of the most effective means for investigating health, there is image diagnosis such as endoscope, X-ray, echo, CT, MRI and the like. For example, in the image diagnosis of the lungs and the heart, the health diagnosis is performed from information including blur and noise by the operation of the lungs and the heart. By removing the blur and noise by applying the inventive method to these image diagnoses, it is possible to quickly provide a clear image to a specialist, and it is expected to help early detection of diseases.

  In recent years, since car navigation systems have become widespread, the number of cars equipped with front and rear cameras is increasing. The invention method 1 is also effective in the case of such a front / rear camera of an automobile that requires real-time image restoration processing.

  As described above, the specific technique (invention technique) of the image restoration method according to the present invention is not limited to Invention technique 1. Below, as another invention method, five invention methods 2-6 are demonstrated one by one. However, in order to facilitate comparison between the inventive methods 1 to 6, the above-described inventive method 1 will also be briefly described. In addition, the description of common parts is omitted as appropriate.

<Invention Method 1>
First, as a problem setting common to each of the inventive methods 1 to 6, an image deterioration model as a theory of image restoration will be briefly described again. FIG. 29 is a diagram for visually explaining the deteriorated image model (see FIG. 3, equation (1)).

  As shown in the upper side of FIG. 29, the image is deteriorated by adding blur and noise to the original image. Blur is caused by a pixel affecting its surrounding pixels, but noise occurs regardless of the pixel. In the case of an image taken with a camera or the like, blur is caused by camera shake or defocus, and noise is unavoidably caused by dark current or thermal noise. The degraded image is generally obtained by a model obtained by adding noise to a convolution of the original image and the blur point spread function (PSF). Specifically, the degraded image g (x, y) is represented by f (m, n) for the original image, h (x, y) for the point spread function of blur, and n (x, y) for noise. , Represented by the degraded image generation formula shown in FIG. In this deteriorated image model, the noise n (x, y) and the original image f (m, n) are uncorrelated, and the noise n (x, y) is an additive white Gaussian noise. ing. Restoration of a degraded image is nothing but removing blur and noise from the degraded image.

  30 is a diagram for visually explaining the state space model of the inventive method 1, FIG. 31 is a diagram for visually explaining the state process of the inventive method 1, and FIG. 32 is an observation of the inventive method 1. It is a figure for demonstrating a process visually (refer above, FIGS. 4-11, Formula (2)-Formula (4)).

In Invention Method 1, as shown collectively in FIG. 30, a new state space model (state equation and observation equation) is constructed and formulated. This state space model is composed of a state process and an observation process. The state process is described by a state equation as shown in FIG. 31, and the observation process is described by an observation equation as shown in FIG. Here, in the state equation, the vector x _p1 (n) is a state vector (original image information), the matrix Φ _p1 is a state transition matrix, and the vector δ _p1 (n) is a drive source vector. In the observation equation, the vector y _p1 (n) is an observation vector (observation image information and degraded image information), the matrix M _p1 is an observation transition matrix, and the vector ε _p1 (n) is an observation noise vector. Further, “n” as a variable of the vectors x _p1 (n), δ _p1 (n), y _p1 (n), and ε _p1 (n) is the time n of the apparatus. The time n represents the processing order (nth) of the processing target block including the target pixel (or target pixel region) and a plurality of peripheral pixels (or peripheral pixel regions). The state equation describes the system to be observed with a state space model, and represents the generation process of the internal state, that is, the state variables (here, state vectors x _p1 (n), x _p1 (n + 1)) with respect to time. Yes. The observation equation describes the process of observation through some kind of observation device, and the observation result (here, observation vector y _p1 (n)) is the observed quantity, ie, the input (here, state vector x _p1 ( n)).

FIG. 33 is an explanatory diagram showing the state equation and the observation equation of Invention Method 1 together, and FIG. 34 is a diagram for explaining the algorithm of Invention Method 1 (FIGS. 12, 14, and Equation (5)). -See formula (10)). As a notation of the expression, generally, the lower case letter “α _γ ” of the alphabet represents a vector having a size of γ × 1, and the upper case letter “の_γ ” of the alphabet represents a matrix having a size of γ × γ. ing.

In the state equation and the observation equation shown in FIG. 33 and the algorithm shown in FIG. 34, K ₁ is the number of pixels included in the peripheral pixel region (strictly, the processing target block), and m and n are respectively the vertical lengths of the peripheral pixel region. In the case of the length of the axis and the horizontal axis, K ₁ = m × n. That is, in Invention Method 1, in the degraded image restoration process, not only the target pixel to be restored is used, but also the surrounding pixels are processed. Specifically, for example, an area of m × n (where J> m, n) around a pixel of interest in an image with a size of J × J is a processing target block, and this processing target block Image restoration processing is performed using the pixel information in. The processing target block means a range in which processing is performed using m × n pixels in image restoration. The lower right part of FIGS. 33 and 34 exemplifies a processing target block in the case of K ₁ = 9 (= 3 × 3).

  FIG. 35 is a flowchart showing a processing procedure for executing the algorithm of FIG. 34 (see FIG. 13).

Here, the sizes of the matrix and the vector are specified in correspondence with the notation of the algorithm of FIG. Specifically, the correlation value (matrix) (S1100) of the estimation error of n → (n + 1) is a matrix of size K ₁ × K ₁ , and the weight coefficient (matrix) (S1200) is of size K ₁ × K. ₁ of the matrix, n → (n + 1) state of optimal estimate of (vector) (S1300), the size vector of _{K 1 × 1, (n +} 1) → (n + 1) state of optimal estimate of (vector) (S1400) is a vector of size K ₁ × 1, and the correlation value (matrix) (S1600) of the estimation error of (n + 1) → (n + 1) is a matrix of size K ₁ × K ₁ .

<Invention Method 2>
FIG. 36 is a diagram for visually explaining the problem of the inventive method 1.

In Invention Method 1, as shown in FIG. 36, the change at time n is expressed using the transition of the pixel region (processing target block). At time n + 1, for example, past one time before the 9 (= 3 × 3) pieces of the state quantity by using the x 1 ~x ₉ (pixel information of the original _image), 9 of the current process target block (= 3 × 3) amount number of states (by estimating the pixel _information) x 4 _{~x 12} of the original image, to restore the state amount _{x 8} of the target pixel region. As described above, the invention method 1 estimates all the state quantities (elements) in the processing target block in order to restore the state quantities (elements) of one target pixel region. However, in the first place, in order to restore an element of one target pixel area, only the element of the target pixel area needs to be obtained, and estimation of other elements is unnecessary. Therefore, in Invention Method 2, a state space model is reconstructed considering only the target pixel region. Specifically, in Invention Method 2, the observation equation of Invention Method 1 is reconfigured to extract only the elements of the pixel region of interest. As a result, the invention method 2 enables a large amount of computation to be reduced compared to the invention method 1, and realizes high-speed (low computation amount) degraded image restoration.

  FIG. 37 is an explanatory diagram visually showing the configuration of the state equation of Invention Method 2, in particular, (A) is a diagram showing a specific example of the state equation of Invention Method 1, and (B) is a change in time. It is a figure which shows the change of the process target block accompanying with. As shown in FIG. 37, the state equation of the inventive method 2 is exactly the same as the state equation of the inventive method 1, and the description thereof is omitted.

  FIG. 38 is an explanatory diagram visually showing the configuration of the observation equation of Invention Method 2, and FIG. 39 is another explanatory diagram visually showing the configuration of the observation equation of Invention Method 2.

In Invention Method 2, as shown in FIG. 38, only the element of the pixel region of interest (for example, pixel information y ₅ (n)) is extracted from the observation equation of Invention Method 1, and the observation equation is reconstructed. As a result, in Invention Method 2, as shown in FIG. 39, the observation vector y _p1 (n), the observation transition matrix M _p1 , and the observation noise vector ε _p1 (n) in Invention Method 1 are _scalarized and vectorized, respectively. Are converted into an observed value y _p2 (n), an observed transition vector m ^T _p2 , and an observed noise ε _p2 (n). As a result, the size of the inventive method 2 is reduced compared to the inventive method 1, so that the amount of calculation is greatly reduced. The calculation amount reduction effect will be described in detail later.

FIG. 40 is an explanatory diagram collectively showing the state equation and the observation equation of Invention Method 2, and FIG. 41 is a diagram for explaining the algorithm of Invention Method 2. As a notation of the expression, generally, the lower case letter “α _γ ” of the alphabet represents a vector having a size of γ × 1, and the upper case letter “の_γ ” of the alphabet represents a matrix having a size of γ × γ. ing.

State equation and an observation equation shown in FIG. 40, as well as in the algorithm shown in FIG. 41, K ₂ is the number of pixels included in the peripheral pixel area (strictly, the target block), m, vertical n each peripheral pixel region When it is set as the length of an axis | shaft and a horizontal axis, it represents with K < ₂ > = mxn. As shown in FIG. 42, the algorithm of the invention method 2 is roughly divided into an initialization process and an iteration process, and the iteration process is more than the case of the invention method 1 (see FIG. 34). It is configured based on a new state space model (state equation and observation equation) shown in FIG. In the iteration process, steps 1 to 6 are repeated sequentially.

  FIG. 42 is a flowchart showing a processing procedure for executing the algorithm of FIG.

As described above, in Invention Method 2, as shown in FIG. 39, the observation vector y _p1 (n), the observation transition matrix M _p1 , and the observation noise vector ε _p1 (n) in Invention Method 1 are _scalarized , respectively. Vectorized and scalarized into an observed value y _p2 (n), an observed transition vector m ^T _p2 , and an observed noise ε _p2 (n). Thus, in the flowchart of FIG. 42, the correlation value (S3100) of the estimation error of n → (n + 1) is a matrix of size K ₂ × K ₂ , and the weighting coefficient (S3200) is a vector of size K ₂ × 1 , N → (n + 1) state quantity optimum estimate (S3300) is a vector of size K ₂ × 1, and (n + 1) → (n + 1) state quantity optimum estimate (S3400) is size K _2. The correlation value (S3600) of the estimation error of (x + 1) vector (n + 1) → (n + 1) is a matrix of size K ₂ × K ₂ .

  FIG. 43 is an explanatory diagram comparing the algorithms of Invention Method 1 and Invention Method 2. FIG.

  As shown in FIG. 43, the algorithm of the inventive method 1 requires calculation of an inverse matrix (refer to iterative process, that is, procedure 2 of the update formula). In general, the calculation of an inverse matrix is complicated and requires a large amount of computation. On the other hand, in the invention method 2, the inverse matrix calculation in the invention method 1 is the scalar reciprocal calculation due to the above-described reduction in size (refer to the iterative process, that is, the update formula procedure 2). Calculation of the reciprocal of a scalar is simple and requires only one calculation. Therefore, the amount of calculation of Invention Method 2 is significantly reduced as compared with Invention Method 1.

  FIG. 44 is a diagram for explaining the calculation amount evaluation of the inventive method 2. Here, Conventional Method 3 is an image restoration method using a Kalman filter (for example, see Non-Patent Document 6). However, since the method described in Non-Patent Document 6 can only deal with a degraded image in which noise is added to the original image, the present inventor seems to be able to cope with blur and noise in the method described in Non-Patent Document 6. The following various comparisons (including simulation) were performed.

  In FIG. 44, the calculation amount is represented by the number of multiplications. The comparison result of the number of operations per update for the conventional method 3, the inventive method 1, and the inventive method 2 is shown on the upper side of FIG. Based on this, when the calculation amount (number of multiplications) per update is obtained, a bar graph shown on the left side of the center of FIG. 44 is obtained. As is apparent from this bar graph, the calculation amount of Invention Method 2 is significantly smaller than that of Invention Method 1. When viewed in terms of the effect of reducing the amount of computation, the magnitude relationship is Conventional Method 3 <Invention Method 1 <Invention Method 2. Therefore, Invention Method 2 achieves a significant reduction in the amount of calculation.

  FIG. 45 is a diagram for explaining the restoration processing speed evaluation of the method 2 of the invention.

  Here, in order to compare the processing speed of each method, the processing time of two types of images having different sizes was measured based on the specification table shown in FIG. As a result, both the inventive method 1 and the inventive method 2 are faster in processing speed than the conventional method 3 because the calculation amount of AR coefficient estimation (derivation) is unnecessary. In addition, since the invention method 2 has a much smaller amount of calculation than the invention method 1 as described above, the processing can be performed at a higher speed than the invention method 1. When viewed in terms of processing speed, as shown in FIG. 45, the magnitude relationship is Conventional Method 3 <Invention Method 1 <Invention Method 2. Therefore, Invention Method 2 can perform high-speed image restoration processing.

  In addition, the present inventor performed a simulation to verify the effectiveness of Invention Method 2. Specifically, visual evaluation and objective evaluation were performed in order to evaluate the image restoration ability of Invention Method 2. The visual evaluation is an evaluation in which the original image and the restored image are visually compared, and the objective evaluation is an evaluation by numerical values.

  FIG. 46 is a diagram for explaining simulation conditions.

  In this simulation, as in the simulation conditions of FIG. 15, “Cameraman” and “Lenna” were used as the original images (both image sizes are 256 × 256). Further, a two-dimensional Gaussian function shown in FIG. 15 was used as a model of a point spread function (PSF) corresponding to the blur added to the original image, and additive white Gaussian noise was used as noise. The signal-to-noise ratio SNR was 20 dB.

  47 is a diagram showing a simulation result (visual evaluation) of Invention Method 2 for the original image “cameraman” in comparison with the comparison target method. FIG. 48 is an enlarged view of the peripheral area of the broken-line circle in FIG. FIG. 49 is a diagram showing a simulation result (visual evaluation) of Invention Method 2 for the original image “Lena” in comparison with the comparison target method, and FIG. 50 shows the peripheral area of the broken-line circle in FIG. FIG. Here, the comparison target techniques are the conventional technique 3 and the invention technique 1.

  As shown in FIGS. 47 to 50, it can be seen that Invention Method 1 and Invention Method 2 clearly restores the original image as compared with Conventional Method 3. However, when the inventive method 1 and the inventive method 2 are compared, it can be said that the image restoration capability is almost the same level.

  FIG. 51 is a diagram showing a simulation result (objective evaluation) of Invention Method 2 for the original image in comparison with the comparison target method.

  Here, PSNR (Peak Signal-to-Noise Ratio) (see Expression (11)) shown in FIG. 51 is used to evaluate the image restoration capability of Conventional Method 3, Invention Method 1, and Invention Method 2 by numerical values. Image restoration ability was evaluated. The PSNR is a value indicating the degree of similarity to the image, and it can be said that the larger the value, the less the deterioration (blur, noise, etc.) and the better the image.

  In this case, as shown in FIG. 51, it can be confirmed that the invention method 1 and the invention method 2 have larger PSNR values than the conventional method 3 for both the “cameraman” and “Lena” images. Thus, it can be seen that Invention Method 1 and Invention Method 2 have a higher image restoration capability than Object Method 3 also in terms of objective evaluation. On the other hand, when the inventive method 1 and the inventive method 2 are compared, the inventive method 2 has a slightly higher objective evaluation (evaluation by numerical values) than the inventive method 1. As a result, as shown in FIG. 51, the objective evaluation magnitude (quality / non-defective) relationship becomes Conventional Method 3 <Invention Method 1 <Invention Method 2. However, this slight difference is almost non-existent, as can be seen from the comparison of FIGS.

  From the above, it can be seen that Invention Method 2 achieves high-speed image restoration processing by substantially reducing the amount of computation while substantially maintaining the image restoration capability compared to Invention Method 1.

<Invention technique 3>
FIG. 52 is a diagram for visually explaining the problem of the inventive method 2.

  As described above, the invention method 2 can realize a high-speed image restoration process by significantly reducing the amount of calculation compared to the invention method 1, but the image restoration ability is slightly lowered. As shown in FIG. 52, this is considered to be because the observation equation of Invention Method 2 does not include a region where the influence of the colored drive source is large. Therefore, in Invention Method 3, a state space model is reconstructed in consideration of not only the target pixel region but also the colored drive source. That is, in Invention Method 3, the state space model is reconfigured so as to maximize the influence of the colored drive source.

  FIG. 53 is an explanatory diagram visually showing the configuration of the observation equation of the inventive method 3, and FIG. 54 is a diagram for visually explaining the state space model of the inventive method 3. The state equation of the invention method 3 is exactly the same as the state equation of the invention method 2 (and thus the invention method 1), and thus the description thereof is omitted.

In Invention Method 3, as shown in FIG. 53, in the observation equation of Invention Method 1, in addition to the element of the pixel region of interest (for example, pixel information y ₅ (n)), the region affected by the chromatic drive source Elements (eg, pixel information y ₇ (n), y ₈ (n), y ₉ (n)) are also extracted to reconstruct the observation equation. That is, the invention method 3 incorporates pixel information y ₇ (n), y ₈ (n), and y ₉ (n) as compared with the invention method 2 in order to consider the influence of the colored drive source. Thus, in Invention Method 3, the observation vector y _p1 (n), the observation transition matrix M _p1 , and the observation noise vector ε _p1 (n) in Invention Method 1 are reduced in size, and the observation vector y _p3 (n ), Observation transition matrix M _p3 , and observation noise vector ε _p3 (n). In this way, in Invention Method 3, as shown in FIG. 54, in order to consider the influence of the chromaticity driving source in the state equation to the maximum, the observation equation is calculated from the target pixel area and the area affected by the chromaticity driving source. Reconfigure. Thus, although the size of the invention method 3 is smaller than that of the method 2 of the invention, the size is reduced compared to the method 1 of the invention, so that the calculation amount is greatly reduced.

FIG. 55 is an explanatory diagram collectively showing the state equation and the observation equation of Invention Method 3, and FIG. 56 is a diagram for explaining the algorithm of Invention Method 3. As a notation of the expression, generally, the lower case letter “α _γ ” of the alphabet represents a vector having a size of γ × 1, and the upper case letter “の_γ ” of the alphabet represents a matrix having a size of γ × γ. The capital letter “C _{(γ × ζ)} ” of the alphabet represents a matrix of size γ × ζ.

State equation and an observation equation shown in FIG. 55, as well as in the algorithm shown in FIG. 56, K ₃ is the number of pixels included in the peripheral pixel area (strictly, the target block), m, vertical n each peripheral pixel region If the length of the axis and the horizontal axis is represented by K 3 _{= m} × n. L ₃ is the number of pixels included in the region including the target pixel region and the region including the colored drive source, and is represented by L ₃ = 1 + m (L ₃ <K ₃ ). As shown in FIG. 56, the algorithm of the invention technique 3 is roughly divided into an initialization process and an iteration process, and the iteration process is more than that of the invention technique 2 (see FIG. 41). Although the calculation amount is slightly increased, the calculation amount is reduced as compared with the case of Invention Method 1 (see FIG. 34), so that the calculation amount is reduced based on the new state space model (state equation and observation equation) shown in FIG. . In the iteration process, steps 1 to 6 are repeated sequentially.

  FIG. 57 is a flowchart showing a processing procedure for executing the algorithm of FIG.

As described above, in Invention Method 3, as shown in FIG. 53, the size of observation vector y _p1 (n), observation transition matrix M _p1 , and observation noise vector ε _p1 (n) in Invention Method 1 is reduced. (That is, L ₃ <K ₃ ), observation vector y _p3 (n), observation transition matrix M _p3 , and observation noise vector ε _p3 (n). Accordingly, in the flowchart of FIG. 57, the correlation value (S4100) of the estimation error of n → (n + 1) is a matrix of size K ₃ × K ₃ , and the weight coefficient (S4200) is of size K ₃ × L ₃ . The optimal estimate (S4300) of the state quantity of the matrix n → (n + 1) is a vector of size K ₃ × 1, and the optimal estimate (S4400) of the quantity of state (n + 1) → (n + 1) is size K The correlation value (S4600) of the estimation error of ₃ × 1 vector, (n + 1) → (n + 1) is a matrix of size K ₃ × K ₃ .

  FIG. 58 is a diagram for explaining the calculation amount evaluation of the inventive method 3.

  The upper side of FIG. 58 shows the number of operations (number of multiplications) per update for Invention Method 3. Based on this, the amount of calculation per update (the number of multiplications) is obtained, and when the obtained result is added to the bar graph shown in FIG. 44, the bar graph shown in the center left side of FIG. 58 is obtained. As is apparent from this bar graph, Invention Method 3 has a significantly smaller amount of computation than Invention Method 1. When viewed in terms of the effect of reducing the amount of calculation, the magnitude relationship is the conventional method 3 <invention method 1 <invention method 3 <invention method 2. Therefore, Invention Method 3 realizes a good reduction in the calculation amount.

  FIG. 59 is a diagram for explaining the restoration processing speed evaluation of Invention Method 3.

  Here, the measurement result of the inventive method 3 is added to the result of FIG. As a result, the inventive method 3 also has a higher processing speed than the conventional method 3 because it does not require the amount of calculation for AR coefficient estimation (derivation). In addition, since the method 3 of the invention has a much smaller calculation amount than the method 1 of the invention as described above, the processing can be performed at a higher speed than the method 1 of the invention. Invention method 3 is slightly slower than Invention method 2 in processing speed. When viewed in terms of processing speed, as shown in FIG. 59, the magnitude relationship is Conventional Method 3 <Invention Method 1 <Invention Method 3 <Invention Method 2. Therefore, the invention method 3 can also perform high-speed image restoration processing.

  FIG. 60 is a diagram showing a simulation result (visual evaluation) of Invention Method 3 for the original image “cameraman” in comparison with the comparison target method, and FIG. 61 is an enlarged view of the peripheral area of the broken-line circle in FIG. FIG. 62 is a diagram showing a simulation result (visual evaluation) of Invention Method 3 for the original image “Lena” in comparison with the comparison target method, and FIG. 63 shows a peripheral region of a broken-line circle in FIG. FIG. The simulation conditions are as shown in FIG. The comparison target methods here are the conventional method 3 and the invention method 1.

  As shown in FIGS. 60 to 63, it can be seen that Invention Method 3 clearly restores the original image faithfully compared to Conventional Method 3. Moreover, it can be seen that the image restoration capability of the invention method 3 is higher than that of the method 1 of the invention.

  FIG. 64 is a diagram showing a simulation result (objective evaluation) of Invention Method 3 for the original image in comparison with the comparison target method.

  In this case, as shown in FIG. 64, the invention method 3 has a PSNR higher than that of any other method (the conventional method 3 and the invention method 1 and the invention method 2) for both “cameraman” and “Lena” images. It can be confirmed that the numerical value is large. Thus, it can be seen that Invention Method 3 has higher image restoration capability than other methods (Conventional Method 3, Invention Method 1, Invention Method 2) in terms of objective evaluation. As a result, as shown in FIG. 64, the magnitude relationship of the objective evaluation (evaluation by numerical values) is as follows: Conventional method 3 <Invention method 1 <Invention method 2 <Invention method 3.

  As described above, Invention Method 3 realizes high-speed image restoration processing that can exhibit better image restoration ability than Invention Method 1 and Invention Method 2 while slightly increasing the amount of calculation compared to Invention Method 2. You can see that

<Invention Method 4>
FIG. 65 is a diagram for explaining the characteristics of the one-dimensional image restoration technique.

  Invention methods 1 to 3 are one-dimensional image restoration methods. In Invention Methods 1 to 3, which are one-dimensional image restoration methods, as shown in FIG. 65, the target pixel region (only one pixel) is restored using the peripheral pixel region. At this time, the image restoration method can be applied according to the use or situation. For example, as described above, Invention Method 2 is a faster (low calculation amount) image restoration method than Invention Method 1, and Invention Method 3 has a slightly larger calculation amount than Invention Method 2, but the invention Since this method improves the image restoration capability over Method 1 and Invention Method 2, Invention Method 2 is applied when priority is given to processing speed, and Invention Method 3 is applied when priority is given to image quality. It can be used.

  FIG. 66 is a diagram for explaining the characteristics of the two-dimensional image restoration method.

  The invention method 4 proposes a two-dimensional image restoration method obtained by extending the one-dimensional invention method 1 to two dimensions. In Invention Method 4, by taking a pixel area of interest larger than that in Invention Method 1, it is possible to reduce the number of updates and restore the processing efficiency. Therefore, in Invention Method 4, the pixel region (only one pixel) in Invention Method 1 is expanded to a pixel region having arbitrary vertical and horizontal lengths p and q (hereinafter referred to as “two-dimensional pixel region”). Say). FIG. 66 shows an example where one pixel area is p = q = 2 and the processing target block (consisting of the target pixel area and the peripheral pixel area) is 3pq × 3pq. When the pixel area is two-dimensionalized, the size of the pixel area and the size of the processing target block can be freely selected. By making the pixel area two-dimensional, the number of pixels in the target pixel area increases, so that the processing efficiency can be improved. In addition, when p = q = 1, the target pixel region is equal to the one-dimensional image restoration method (Invention methods 1 to 3), so the invention method 4 which is a two-dimensional image restoration method is a flexible method. It can be said. As described above, a two-dimensional Kalman filter obtained by extending a one-dimensional Kalman filter to two dimensions is also known. However, since the two-dimensional Kalman filter uses an AR coefficient as in the one-dimensional Kalman filter, the estimation accuracy of the AR coefficient is improved. The resulting reduction in image restoration capability is inevitable. In order to solve this, a new prediction method that does not require an AR coefficient may be devised and a state space model of a two-dimensional Kalman filter may be reconstructed. Invention method 4 is exactly a new method for this purpose, and is an image restoration method based on a new prediction method obtained by extending invention method 1 to two dimensions. Hereinafter, Invention Method 4 will be described in more detail.

  FIG. 67 is a diagram for visually explaining the state space model of Invention Method 4. FIG. 68 is a diagram for visually explaining the state process of the inventive method 4, and FIG. 69 is an explanatory diagram visually showing the configuration of the state equation of the inventive method 4.

With respect to the state process, as shown in FIG. 68, the invention method 4 designates an area having a size of p × q as a pixel area, and arranges pixel information included in the designated pixel area to form a pq × first order pixel area vector. Is generated. For example, as shown in FIG. 68, in the case of an area of p = q = 2 and 3pq × 3pq, the pixel area vector x ₁ (n) at the upper left is four pieces of pixel information x ₁ (n), x ₂ (n), x ₃ (n), x ₄ (n) vector elements, and the lower right pixel region vector x ₉ (n) is This is a 4 × first-order vector having four pieces of pixel information x ₃₃ (n), x ₃₄ (n), x ₃₅ (n), and x ₃₆ (n) as elements. Then, the state vector x (n) is expressed by arranging pixel region vectors x _k (n) (k = 1, 2,..., 9) corresponding to the nine pixel regions. That is, in Invention Method 4, as shown in FIG. 69, the pixel region is associated with the pixel information in Invention Method 1 and the same processing is performed.

  FIG. 70 is a diagram for visually explaining the observation process of the inventive method 4, and FIG. 71 is an explanatory diagram visually showing the configuration of the observation equation of the inventive method 4.

As for the observation process, in Invention Method 4, as shown in FIG. 70, an area having a size of p × q is designated as a pixel area, and pixel information included in the designated pixel area is arranged to form a pq × primary pixel area. Generate a vector. For example, as shown in FIG. 70, in the case of a region of p = q = 2 and 3pq × 3pq, the pixel region vector y ₁ (n) at the upper left is the four pixel information y ₁ (n), y ₂ (n), y ₃ (n), y ₄ (n) is a 4 × 1 order vector, and the pixel area vector y ₉ (n) at the bottom right is This is a 4 × first-order vector having four pieces of pixel information y ₃₃ (n), y ₃₄ (n), y ₃₅ (n), and y ₃₆ (n) as elements. Then, the pixel region vectors y _k (n) (k = 1, 2,..., 9) corresponding to the nine pixel regions are arranged to express the observation vector y (n). That is, in Invention Method 4, as shown in FIG. 71, the pixel region is associated with the pixel information in Invention Method 1 and the same processing is performed.

  72 is a diagram showing a specific example of the observed transition matrix of the invention method 4 in the case of the region of p = q = 2 and 3pq × 3pq, and FIG. 73 shows the configuration of the observed transition matrix of FIG. It is explanatory drawing shown visually. FIG. 74 is a diagram showing a specific example of the coefficient assignment method constituting the observation transition matrix of FIG. 72, and FIG. 75 is another example of the coefficient assignment method constituting the observation transition matrix of FIG. 76 and 76 are still other diagrams showing a specific example of a method of assigning coefficients constituting the observation transition matrix of FIG.

Here, 3 × 3 (= 9) coefficients h _{r, s} (coordinates r, s = −1, 0, 1) constituting the observation transition matrix of the inventive method 4 are the same as those of the inventive method 1. Yes, the pixel information is assigned so that the same relationship as in the case of Invention Method 1 is established when viewed in units of pixels. Specific allocation methods for the coefficients h _{r, s} (r, s = −1, 0, 1) are shown in FIGS. 74 to 76 in accordance with the position of the observation amount y _i when the pixel information is used as a unit. As shown.

FIG. 77 is an explanatory diagram collectively showing the state equation and the observation equation of Invention Method 4, and FIG. 78 is a diagram for explaining the algorithm of Invention Method 4. As a notation of the expression, generally, the lower case letter “α _γ ” of the alphabet represents a vector having a size of γ × 1, and the upper case letter “の_γ ” of the alphabet represents a matrix having a size of γ × γ. ing.

In the state equation and the observation equation shown in FIG. 77 and the algorithm shown in FIG. 78, K ₄ is the number of pixels included in the peripheral pixel region (strictly, the processing target block), and m and n are respectively the vertical lengths of the peripheral pixel region. In the case of the length of the axis and the horizontal axis, it is represented by K ₄ = m × n. Here, m and n are represented by m = l _m × p and n = l _n × q, respectively. p and q are the lengths of the vertical axis and horizontal axis of the pixel area of interest, respectively, and l _m and l _n are respectively configured to have a size equivalent to p × q on the vertical axis and horizontal axis of the peripheral pixel area. It is a number.

  FIG. 79 is a flowchart showing a processing procedure for executing the algorithm of FIG.

Here, the sizes of the matrix and the vector are specified in correspondence with the notation of the algorithm of FIG. Specifically, the correlation value (matrix) (S5100) of the estimation error of n → (n + 1) is a matrix having a size of K ₄ × K ₄ , and the weight coefficient (matrix) (S 5200) is a size of K ₄ × K. ₄ matrix, optimal estimation value (vector) (S5300) of state quantity of n → (n + 1) is a vector of size K ₄ × 1, optimal estimation value (vector) of state quantity of (n + 1) → (n + 1) (S5400), the size vector of _{K 4 × 1, (n +} 1) → (n + 1) correlation value of the estimated error of (matrix) (S5600), the size of the matrix _{K 4} × _{K 4.}

  FIG. 80 is a diagram for explaining the calculation amount evaluation of Invention Method 4. Here, the conventional method 4 is an image restoration method using a two-dimensional Kalman filter (for example, see Non-Patent Document 8). However, since the method described in Non-Patent Document 8 can only deal with a degraded image in which noise is added to the original image, the present inventor seems to be able to cope with blur and noise in the method described in Non-Patent Document 8. The following various comparisons (including simulation) were performed.

  On the upper side of FIG. 80, the number of operations (number of multiplications) per update for Conventional Method 4 and Invention Method 4 is shown. Based on this, the amount of calculation (number of multiplications) required to restore one image of 256 × 256 size is obtained, and the obtained result is shown together with the result for the inventive method 1, the left side of the center of FIG. The bar graph shown in FIG. As is apparent from this bar graph, the inventive method 4 has a smaller amount of calculation than the conventional method 4, but has a larger amount of calculation than the inventive method 1. However, Invention Method 4 has a smaller number of updates since the pixel area of interest is wider than Invention Method 1. When viewed in terms of the effect of reducing the amount of calculation, the magnitude relationship is Conventional Method 4 <Invention Method 4 <Invention Method 1. Therefore, Invention Method 4 can efficiently restore an image by changing the sizes of the target pixel region and the peripheral pixel region, respectively.

  FIG. 81 is a diagram for explaining the restoration processing speed evaluation of Invention Method 4.

  Here, the measurement results of Conventional Method 4 and Invention Method 4 are added to the measurement results of Invention Method 1. As a result, the inventive method 4 is faster in processing speed than the conventional method 4 because it does not require the amount of calculation for AR coefficient estimation (derivation). In addition, the invention method 4 has a larger amount of calculation than the invention method 1 as described above, but the number of updates is small, so that the processing can be performed at a higher speed than the method 1 of the invention. When viewed in terms of processing speed, as shown in FIG. 81, the magnitude relationship is Conventional Method 4 <Invention Method 1 <Invention Method 4. Therefore, Invention Method 4 can perform high-speed image restoration processing.

  82 is a diagram showing a simulation result (visual evaluation) of Invention Method 4 for the original image “cameraman” in comparison with the comparison target method, and FIG. 83 is an enlarged view of the peripheral area of the broken-line circle in FIG. FIG. 84 is a diagram showing a simulation result (visual evaluation) of Invention Method 4 for the original image “Lena” in comparison with the comparison target method, and FIG. 85 shows the peripheral area of the broken-line circle in FIG. FIG. The simulation conditions are as shown in FIG. The comparison target methods here are the conventional method 3 and the invention method 1.

  As shown in FIGS. 82 to 85, it can be seen that Invention Method 4 clearly restores the original image faithfully compared to Conventional Method 3. Moreover, it can be seen that the image restoration capability of the invention method 4 is higher than that of the invention method 1.

  FIG. 86 is a diagram showing a simulation result (objective evaluation) of Invention Method 4 for the original image in comparison with the comparison target method.

  In this case, as shown in FIG. 86, for both the “cameraman” and “Lena” images, the invention method 4 has a slightly smaller PSNR value than the invention method 3, but the conventional method 3 and the invention method 1 and It can be confirmed that the value of PSNR is larger than that of Invention Method 2. Thus, it can be seen that Invention Method 4 has higher image restoration capability than Objective Method 3, Invention Method 1, and Invention Method 2 in terms of objective evaluation. As a result, as shown in FIG. 86, the magnitude relationship of the objective evaluation (evaluation by numerical values) is as follows: Conventional method 3 <Invention method 1 <Invention method 2 <Invention method 4 <Invention method 3.

  As described above, the invention method 4 realizes a high-speed image restoration process capable of demonstrating better image restoration ability than the invention method 1 and the invention method 2 although the calculation amount is larger than that of the invention method 1. I understand that.

<Invention Method 5>
FIG. 87 is a diagram for visually explaining the problem of the invention technique 4.

In Invention Method 4, as shown in FIG. 87, the change at time n is expressed using the transition of the pixel region (processing target block). At time n + 1, for example, using the state quantities (pixel information of the original image) included in the 9 (= 3 × 3) pixel areas one time before the past, 9 (= The state quantities x ₂₉ , x ₃₀ , x ₃₁ , and x ₃₂ of the target pixel area are restored by estimating the respective state quantities (original image pixel information) included in the 3 × 3) pixel areas. As described above, the invention method 4 estimates all the state quantities (elements) in the processing target block in order to restore each state quantity (element) included in one target pixel region. However, in the first place, in order to restore an element in one target pixel area, only the element in the target pixel area needs to be obtained, and estimation of other elements is unnecessary. Therefore, in Invention Method 5, a state space model that considers only the pixel region of interest is reconstructed. Specifically, in Invention Method 5, the observation equation of Invention Method 4 is reconfigured to extract only elements included in the target pixel region. As a result, the invention method 5 makes it possible to significantly reduce the amount of calculation compared to the invention method 4, and realizes high-speed (low calculation amount) degraded image restoration. It can be said that Invention Method 5 is an extension of Invention Method 2 to two dimensions.

  FIG. 88 is an explanatory diagram visually showing the configuration of the state equation of Invention Method 5, in particular, (A) is a diagram showing a specific example of the state equation of Invention Method 4, and (B) is a change in time. It is a figure which shows the change of the process target block accompanying with. As shown in FIG. 88, the state equation of Invention Method 5 is exactly the same as the state equation of Invention Method 4, and therefore the description thereof is omitted.

  FIG. 89 is an explanatory diagram visually showing the configuration of the observation equation of the inventive technique 5, and FIG. 90 is another explanatory diagram visually showing the configuration of the observation equation of the inventive technique 5.

In Invention Method 5, as shown in FIG. 89, only the elements (for example, pixel region vector y ₅ (n)) included in the pixel region of interest in the observation equation of Invention Method 4 are extracted to reconstruct the observation equation. Thereby, in Invention Method 5, as shown in FIG. 90, the observation vector y _p4 (n), the observation transition matrix M _p4 and the observation noise vector ε _p4 (n) in Invention Method 4 are reduced in size, respectively. , Observation vector y _p5 (n), observation transition matrix M _p5 , observation noise vector ε _p5 (n). As a result, since the size of the invention method 5 is reduced as compared with the invention method 4, the calculation amount is greatly reduced. The calculation amount reduction effect will be described in detail later.

FIG. 91 is an explanatory diagram collectively showing the state equation and the observation equation of Invention Method 5, and FIG. 92 is a diagram for explaining the algorithm of Invention Method 5. As a notation of the expression, generally, the lower case letter “α _γ ” of the alphabet represents a vector having a size of γ × 1, and the upper case letter “の_γ ” of the alphabet represents a matrix having a size of γ × γ. The capital letter “C _{(γ × ζ)} ” of the alphabet represents a matrix of size γ × ζ.

State equation and an observation equation shown in FIG. 91, as well as in the algorithm shown in FIG. 92, K ₅ is the number of pixels included in the peripheral pixel area (strictly, the target block), m, vertical n each peripheral pixel region In the case of the length of the axis and the horizontal axis, it is expressed by K ₅ = m × n. Here, m and n are represented by m = l _m × p and n = l _n × q, respectively. p and q are the lengths of the vertical axis and horizontal axis of the pixel area of interest, respectively, and l _m and l _n are respectively configured to have a size equivalent to p × q on the vertical axis and horizontal axis of the peripheral pixel area. It is a number. L ₅ is the number of pixels included in the target pixel region, and is represented by L ₅ = p × q.

  FIG. 93 is a flowchart showing a processing procedure for executing the algorithm of FIG.

As described above, in Invention Method 5, as shown in FIG. 90, the observation vector y _p4 (n), the observation transition matrix M _p4 , and the observation noise vector ε _p4 (n) in the Invention Method 4 each have a size of 9 pq. × 1 → pq × 1, 9pq × 9pq → pq × 9pq, 9pq × 1 → pq × 1 (that is, L ₅ <K ₅ ), observation vector y _p5 (n), observation transition matrix M _p5 , The observation noise vector ε _p5 (n) is obtained. Thus, in the flowchart of FIG. 93, the correlation value (S6100) of the estimation error of n → (n + 1) is a matrix of size K ₅ × K ₅ , and the weight coefficient (S6200) is of size K ₅ × L ₅ . The optimal estimation value (S6300) of the matrix, n → (n + 1) state quantity is a vector of size K ₅ × 1, and the optimal estimation value (S6400) of the state quantity (n + 1) → (n + 1) is size K The correlation value (S6600) of the estimation error of ₅ × 1 vector, (n + 1) → (n + 1) is a matrix of size K ₅ × K ₅ .

  FIG. 94 is an explanatory diagram comparing the algorithms of Invention Method 4 and Invention Method 5. FIG.

  As shown in FIG. 94, the algorithm of the inventive method 4 requires calculation of an inverse matrix having a size of 9pq × 9pq (see the iterative process, that is, procedure 2 of the update formula), and has a large amount of calculation. On the other hand, in Invention Method 5, due to the size reduction described above, the inverse matrix of size 9pq × 9pq becomes the inverse matrix of size pq × pq (refer to the iterative process, that is, procedure 2 of the update formula). Therefore, the amount of calculation of Invention Method 5 is significantly reduced compared to Invention Method 4.

  FIG. 95 is a diagram for explaining the calculation amount evaluation of Invention Method 5.

  On the upper side of FIG. 95, the number of operations (number of multiplications) per update for Invention Method 5 is shown. Based on this, the amount of computation (number of multiplications) required to restore one image of 256 × 256 size is obtained, and the obtained result is added to the bar graph shown in FIG. A bar graph is obtained. As is apparent from this bar graph, Invention Method 5 has a significantly smaller amount of calculation than Conventional Method 4 and Invention Method 4. When viewed in terms of the effect of reducing the amount of computation, the magnitude relationship is as follows: Conventional method 4 <Invention method 4 <Invention method 1 <Invention method 5. Therefore, Invention Method 5 can greatly reduce the calculation amount, and can perform high-speed (low calculation amount) image restoration.

  FIG. 96 is a diagram for explaining the restoration processing speed evaluation of Invention Method 5.

  Here, the measurement result of Invention Method 5 is added to the result of FIG. As a result, the inventive method 5 also has a higher processing speed than the conventional method 4 because the amount of calculation for AR coefficient estimation (derivation) is unnecessary. In addition, since the invention method 5 has a smaller amount of calculation than the invention method 1 and the invention method 4 as described above, the processing can be performed at a higher speed than the invention method 1 and the invention method 4. When viewed in terms of processing speed, as shown in FIG. 96, the magnitude relationship is Conventional Method 4 <Invention Method 1 <Invention Method 4 <Invention Method 5. Therefore, Invention Method 5 can perform high-speed image restoration processing.

  97 is a diagram showing a simulation result (visual evaluation) of Invention Method 5 for the original image “cameraman” in comparison with the comparison target method, and FIG. 98 is an enlargement of the peripheral area of the broken-line circle in FIG. FIG. 99 is a diagram showing the simulation result (visual evaluation) of Invention Method 5 for the original image “Lena” in comparison with the comparison target method, and FIG. 100 shows the peripheral area of the broken-line circle in FIG. FIG. The simulation conditions are as shown in FIG. The comparison target methods here are the conventional method 3 and the invention method 1.

  As shown in FIGS. 97 to 100, it can be seen that Invention Method 5 clearly restores the original image faithfully compared to Conventional Method 3. However, when Invention Method 1 and Invention Method 5 are compared, the image restoration capability is almost the same level.

  FIG. 101 is a diagram showing a simulation result (objective evaluation) of Invention Method 5 on the original image in comparison with the comparison target method.

  In this case, as shown in FIG. 101, for both “cameraman” and “Lena” images, Invention Method 5 has a smaller PSNR value than Invention Method 3, and has a larger PSNR value than Conventional Method 3. Thus, it can be confirmed that the invention methods 1, 2, and 4 have almost the same PSNR numerical value. Thus, it can be seen that Invention Method 5 has a higher image restoration capability than Object Method 3 also in terms of objective evaluation. As a result, as shown in FIG. 101, the magnitude relationship of the objective evaluation (evaluation by numerical values) is as follows: Conventional method 3 <Invention method 1 <Invention method 5 <Invention method 2 <Invention method 4 <Invention method 3> Become.

  As described above, Invention Method 5 realizes an image restoration method that is high in speed and improved in processing efficiency by reducing the amount of calculation while maintaining the image restoration capability substantially equivalent to Invention Method 4 expanded in two dimensions. I understand that.

<Invention Method 6>
FIG. 102 is a diagram for visually explaining the problem of the inventive method 5.

  As described above, the invention method 5 can realize a high-speed image restoration process by significantly reducing the calculation amount as compared with the invention method 4, but the image restoration ability is slightly lowered. This is considered to be because, as shown in FIG. 102, the observation equation of the inventive method 5 does not include a region where the influence of the colored drive source is large. Therefore, in Invention Method 6, a state space model is reconstructed considering not only the pixel region of interest but also the colored drive source. That is, in Invention Method 6, the state space model is reconfigured so as to maximize the influence of the colored drive source.

  FIG. 103 is an explanatory diagram visually showing the configuration of the observation equation of the inventive method 6, and FIG. 104 is a diagram for visually explaining the state space model of the inventive method 6. Note that the state equation of Invention Method 6 is exactly the same as the state equation of Invention Method 5 (and thus Invention Method 4), and thus the description thereof is omitted.

In Invention Method 6, as shown in FIG. 103, in the observation equation of Invention Method 4, in addition to the elements (for example, pixel region vector y ₅ (n)) included in the target pixel region, the influence of the chromatic driving source is considered. The observation equation is reconstructed by extracting the elements (for example, pixel region vectors y ₇ (n), y ₈ (n), y ₉ (n)) included in the receiving region. That is, the invention method 6 incorporates pixel region vectors y ₇ (n), y ₈ (n), and y ₉ (n) as compared with the invention method 5 in order to consider the influence of the colored drive source. . As a result, in the inventive method 6, the observation vector y _p4 (n), the observation transition matrix M _p4 , and the observation noise vector ε _p4 (n) in the inventive method 4 are reduced in size, and the observation vector y _p6 (n ), Observation transition matrix M _p6 , and observation noise vector ε _p6 (n). Thus, in Invention Method 6, as shown in FIG. 104, the observation equation is calculated from the target pixel region and the region affected by the chromatic drive source in order to maximize the influence of the chromatic drive source in the state equation. Reconfigure. Thus, although the size of the inventive method 6 is smaller than that of the inventive method 5, the size is reduced compared to the inventive method 4, so that the amount of calculation is greatly reduced.

FIG. 105 is an explanatory diagram collectively showing the state equation and the observation equation of Invention Method 6, and FIG. 106 is a diagram for explaining the algorithm of Invention Method 6. As a notation of the expression, generally, the lower case letter “α _γ ” of the alphabet represents a vector having a size of γ × 1, and the upper case letter “の_γ ” of the alphabet represents a matrix having a size of γ × γ. The capital letter “C _{(γ × ζ)} ” of the alphabet represents a matrix of size γ × ζ.

In the state equation and the observation equation shown in FIG. 105 and the algorithm shown in FIG. 106, K ₆ is the number of pixels included in the peripheral pixel region (strictly, the processing target block), and m and n are respectively the vertical lengths of the peripheral pixel region. In the case of the length of the axis and the horizontal axis, K ₆ = m × n. Here, m and n are represented by m = l _m × p and n = l _n × q, respectively. p and q are the lengths of the vertical axis and horizontal axis of the pixel area of interest, respectively, and l _m and l _n are respectively configured to have a size equivalent to p × q on the vertical axis and horizontal axis of the peripheral pixel area. It is a number. L ₆ is the number of pixels included in the combined region of the target pixel region and the region including the colored drive source, and is represented by L ₆ = p × q + m × q (L ₆ <K ₆ ). As shown in FIG. 106, the algorithm of the invention method 6 is roughly divided into an initialization process and an iteration process, and the iteration process is more than the case of the invention method 5 (see FIG. 92). Although the calculation amount is slightly increased, the calculation amount is reduced as compared with the case of Invention Method 4 (see FIG. 78), so that the calculation amount is reduced based on the new state space model (state equation and observation equation) shown in FIG. . In the iteration process, steps 1 to 6 are repeated sequentially.

  FIG. 107 is a flowchart showing a processing procedure for executing the algorithm of FIG.

As described above, in Invention Method 6, as shown in FIG. 103, the size of observation vector y _p4 (n), observation transition matrix M _p4 , and observation noise vector ε _p4 (n) in Invention Method 4 is reduced. (That is, L ₆ <K ₆ ), observation vector y _p6 (n), observation transition matrix M _p6 , and observation noise vector ε _p6 (n). Accordingly, in the flowchart of FIG. 107, the correlation value (S7100) of the estimation error of n → (n + 1) is a matrix of size K ₆ × K ₆ , and the weighting coefficient (S 7200) is of size K ₆ × L ₆ . The matrix, the optimal estimate (S7300) of the state quantity n → (n + 1) is a vector of size K ₆ × 1, and the optimal estimate (S7400) of the state quantity (n + 1) → (n + 1) is size K The correlation value (S7600) of the estimation error of ₆ × 1 vector, (n + 1) → (n + 1) is a matrix of size K ₆ × K ₆ .

  FIG. 108 is a diagram for explaining the calculation amount evaluation of Invention Method 6.

  The upper side of FIG. 108 shows the amount of computation (number of multiplications) required to restore one image of 256 × 256 size for Invention Method 6. Based on this, the amount of computation (number of multiplications) per update is obtained, and when the obtained result is added to the bar graph shown in FIG. 95, the bar graph shown in the center left side of FIG. 108 is obtained. As is apparent from this bar graph, the invention method 6 has a slightly larger calculation amount than the invention method 5, but the calculation amount is significantly smaller than that of the invention method 4. When viewed from the effect of reducing the amount of computation, the magnitude relationship is as follows: Conventional method 4 <Invention method 4 <Invention method 1 <Invention method 6 <Invention method 5. Therefore, Invention Method 6 achieves a good calculation amount reduction.

  FIG. 109 is a diagram for explaining the restoration processing speed evaluation of the inventive method 6.

  Here, the measurement result of Invention Method 6 is added to the result of FIG. As a result, the inventive method 6 also has a higher processing speed than the conventional method 4 because it does not require the amount of calculation for AR coefficient estimation (derivation). In addition, the invention method 6 has a much smaller calculation amount than the invention method 4 as described above, and therefore, the processing can be performed at a higher speed than the invention method 4. Invention method 6 is slightly slower than Invention method 5 in processing speed. When viewed in terms of processing speed, as shown in FIG. 109, the magnitude relationship is Conventional Method 4 <Invention Method 1 <Invention Method 4 <Invention Method 6 <Invention Method 5. Therefore, Invention Method 6 can also perform high-speed image restoration processing.

  110 is a diagram showing a simulation result (visual evaluation) of Invention Method 6 for the original image “cameraman” in comparison with the comparison target method, and FIG. 111 is an enlarged view of a peripheral area of a broken-line circle in FIG. FIG. FIG. 112 is a diagram showing a simulation result (visual evaluation) of Invention Method 6 for the original image “Lena” in comparison with the comparison target method, and FIG. 113 shows the peripheral region of the broken-line circle in FIG. FIG. The simulation conditions are as shown in FIG. The comparison target methods here are the conventional method 3 and the invention method 1.

  As shown in FIGS. 110 to 113, it can be seen that Invention Method 6 clearly restores the original image faithfully compared to Conventional Method 3. Moreover, it can be seen that the image restoration capability of the invention method 6 is higher than that of the invention method 1.

  FIG. 114 is a diagram showing a simulation result (objective evaluation) of Invention Method 6 for the original image in comparison with the comparison target method.

  In this case, as shown in FIG. 114, the invention method 6 has a PSNR value higher than that of any of the other methods (conventional method 3 and invention methods 1 to 5) for both “cameraman” and “Lena” images. It can be confirmed that it is large. Thus, it can be seen that Invention Method 6 has a higher image restoration capability than other methods (Conventional Method 3, Invention Methods 1 to 5) in terms of objective evaluation. As a result, as shown in FIG. 114, the magnitude relationship of the objective evaluation (evaluation by numerical values) is as follows. Conventional Method 3 <Invention Method 1 <Invention Method 5 <Invention Method 2 <Invention Method 4 <Invention Method 3 <Invention Method 6 is used.

  As described above, the inventive method 6 realizes a high-speed image restoration process capable of demonstrating a good image restoration capability higher than the inventive methods 1 to 5 while slightly increasing the amount of calculation compared to the inventive method 5. I understand that.

  Here, the features of the state space models (state equations and observation equations) of the inventive methods 1 to 6 will be summarized.

<Invention Method 1>
State equation, the time n + 1 of K ₁ piece of the original image (pixel information), and K ₁ piece of the original image at time n (pixel information), the colored driving source comprising m of the original image (pixel information) It is composed of In the observation equation, K ₁ original images (pixel information) at time n are composed of K ₁ degraded images (pixel information) due to the influence of blur and noise. With this state space model, Invention Method 1 predicts K ₁ original images (pixel information) at time n + 1 of the state equation using K ₁ degraded images (pixel information) of the observation equation. Yes.

<Invention Method 2>
In Invention Method 1, only one restored image (pixel information) is required. Therefore, in Invention Method 2, in order to reduce the amount of calculation, the observation equation of Invention Method 1 is reconstructed by paying attention to only one degraded image (pixel information) of the observation equation of Invention Method 1. However, the state equation of Invention Method 2 is the same as that of Invention Method 1. With such a state space model, the observation equation of Invention Method 2 uses only one deteriorated image (pixel information) at time n, so that the least number of operations can be performed without depending on the past deteriorated image (pixel information). One original image (pixel information) is predicted by the amount.

<Invention technique 3>
In order to obtain a high-performance image restoration capability with a small amount of computation, the observation equation of Invention Method 3 is one of the observation equations of Invention Method 1 consisting of the original image (pixel information) at time n including the influence of the colored drive source. And the observation equation of Invention Method 2. With such a state space model, Invention Method 3 considers the influence of the chromatic drive source to the maximum extent, and thus realizes high-performance image restoration while reducing the amount of calculation.

<Invention Method 4>
In order to improve the processing efficiency, the equation of state is extended to two dimensions, time n + 1 of the K ₄ pieces of the original image (pixel information), the time n of K ₄ pieces of the original image (pixel information), m × q And a colored drive source including original images (pixel information). In the observation equation expanded in two dimensions, K ₄ original images (pixel information) at time n are composed of K ₄ degraded images (pixel information) due to the influence of blur and noise. With this state space model, Invention Method 4 predicts K ₄ original images (pixel information) at time n + 1 of the state equation using K ₄ degraded images (pixel information) of the observation equation. Yes.

<Invention Method 5>
In Invention Method 4 extended in two dimensions, the required number of restored images (pixel information) is p × q. Therefore, in Invention Method 2, in order to reduce the amount of calculation, the observation equation of Invention Method 4 is reconstructed by focusing on only p × q degraded images (pixel information) of the observation equation of Invention Method 4. . However, the state equation of Invention Method 5 is the same as that of Invention Method 4. With such a state space model, the observation equation of Invention Method 5 uses only p × q degraded images (pixel information) at time n, so that it is the most independent of the past degraded images (pixel information). P × q original images (pixel information) are predicted with a small amount of calculation.

<Invention Method 6>
In order to obtain a high-performance image restoration capability with a small amount of computation, the observation equation of Invention Method 6 is one of the observation equations of Invention Method 4 consisting of the original image (pixel information) at time n including the influence of the colored drive source. And the observation equation of Invention Method 5. With such a state space model, Invention Method 6 considers the influence of the chromaticity driving source to the maximum extent, and thus realizes high-performance image restoration while reducing the amount of calculation.

  As described above, according to the inventive methods 1 to 6, in order to construct a new state space model (state equation and observation equation) that does not require an AR coefficient, image restoration is performed by a new prediction method of one-stage processing. Blur and noise can be removed simultaneously using only one deteriorated image.

  In addition, according to the inventive methods 1 to 6, any of them does not require an AR coefficient, so that the amount of calculation can be reduced.

  In addition, according to the inventive methods 4 to 6 in which the inventive methods 1 to 3 are extended to two dimensions, the sizes of the target pixel region and the peripheral pixel region can be arbitrarily changed. It is possible to use different image restoration methods based on new two-dimensional and two-dimensional prediction methods, thereby realizing efficient image restoration.

  In particular, Invention Methods 4 to 6 are degraded image restoration methods using a new prediction method expanded in two dimensions. The greatest features of the invention methods 4 to 6 are the invention method 1, the invention method 2 in which the calculation amount of the invention method 1 is reduced, and the invention method in which performance is prioritized according to the nature of the degraded image and the purpose of the user. 3 can be realized, and can be adaptively changed to Invention Method 4 expanded in two dimensions, Invention Method 5 in which the calculation amount of Invention Method 4 is reduced, and Invention Method 6 in which performance is prioritized.

(Embodiment 2)
The second embodiment is a case where a plurality of image restoration methods are adaptively switched and executed according to the situation.

  FIG. 115 is a block diagram showing a configuration of an image restoration apparatus according to Embodiment 2 of the present invention. The image restoration apparatus 200 has the same basic configuration as that of the image restoration apparatus 100 shown in FIG. 1, and the same components are denoted by the same reference numerals and the description thereof is omitted.

  An image restoration apparatus 200 shown in FIG. 115 has a plurality of built-in image restoration algorithms, and can switch and execute the built-in image restoration algorithms according to user selection or automatically according to circumstances. As a plurality of image restoration algorithms, for example, all or a part of the above-described inventive methods 1 to 6 can be used. As described above, the inventive methods 1 to 6 have characteristics in terms of restoration ability, processing speed (computation amount), processing efficiency, flexibility, and the like, and accordingly, the inventive method to be applied is appropriately selected depending on the situation. Thus, it is possible to perform optimum image restoration according to the situation. Note that the plurality of built-in image restoration algorithms are not limited to the inventive methods 1 to 6, but are any combination of the inventive methods 1 to 6 and the conventional methods (for example, the conventional methods 1 to 4 described above). May be.

  In order to realize such a function, the image restoration apparatus 200 includes a restoration mode designation unit 210 and an image restoration processing unit 220. The restoration mode designation unit 210 is provided in the operation unit 130, for example. The restoration mode designating unit 210 designates a restoration mode to be described later by a user input operation. The image restoration processing unit 220 includes a parameter setting unit 222 and a subject area determination unit 224, for example. In the storage unit 150, for example, the main storage device 152 includes a captured image memory 230, a subject area memory 232, a processing parameter memory 234, and a restored image memory 236, and the display memory 156 includes the captured image memory 240. A processing content memory 242 and a restored image memory 244.

  FIG. 116 is a diagram showing a specific example of restoration modes that can be designated by the restoration mode designating unit 210.

  The restoration modes that can be designated by the restoration mode designating unit 210 are given as the internal structure of the restoration mode designating unit 210 as shown in FIG. In the example shown in FIG. 116, the restoration mode is roughly classified into, for example, “camera mode”, “captured image”, and “quality”. In the camera mode, for example, modes such as “sport”, “landscape”, “night view”, “close-up”, “portrait”, and the like are provided. , “Landscape” mode is provided. In addition, quality is provided with modes such as “image quality priority”, “auto”, and “processing speed priority”. The image quality priority mode is further divided into three levels: “very fine”, “fine”, and “normal”. Each restoration mode is associated with an optimal image restoration algorithm in advance. For example, invention method 5 is associated with “processing speed priority” of “quality”, and invention method 3 or invention method 6 is associated with “very fine” of “quality”. Also, a plurality of restoration modes can be specified simultaneously from different items. For example, when “photo” is specified from “captured image” and “very fine” from “quality” to “image quality priority” is specified, an image restoration algorithm that performs restoration processing that matches these conditions is used. Executed.

  In the image restoration processing unit 220, the parameter setting unit 222 internally sets various parameter values input by the parameter setting unit 132 of the operation unit 130. The subject area determination unit 224 determines a subject area by a predetermined subject area determination algorithm. Here, the subject region determination algorithm is not particularly limited, and any known algorithm can be used.

  In the main storage device 152, the captured image memory 230 stores the image captured by the image input device 110, and the subject area memory 232 stores the area of the subject determined by the subject area determination unit 224. The subject area stored in the subject area memory 232 may be the subject image itself or the storage location of the subject image. The processing parameter memory 234 stores the contents of the restoration process set by the operation unit 130 (set parameters, restoration mode, etc.). The restored image memory 234 stores the image restored by the image restoration processing unit 220. In the display memory 156, the captured image memory 240 temporarily stores the captured image to be displayed among the captured images stored in the captured image memory 230 of the main storage device 152. The processing content memory 242 Of the contents of the restoration process stored in the process parameter memory 234 (set parameters, restoration mode, etc.), the contents of the process to be displayed are temporarily stored. The restored image memory 244 temporarily stores a restored image to be displayed among the restored images stored in the restored image memory 234 of the main storage device 152.

  Next, the operation of the image restoration apparatus 200 having the above configuration will be described using the flowchart shown in FIG. The flowchart shown in FIG. 117 is stored as a control program in the storage unit 150 (the main storage device 152 or the auxiliary storage device 154), and is executed by a CPU (not shown).

  First, in step S8000, image data (degraded image) to be restored is captured from the image input device 110 and stored in the captured image memory 230 of the main storage device 152.

  In step S8050, the image data captured in step S8000 is temporarily stored for at least one frame. Here, the captured image data (at least one frame) is stored in a memory (for example, a working memory (not shown) of the main storage device 152) that temporarily stores one or more frames of image data.

  In step S8100, the subject region determination unit 224 of the image restoration processing unit 220 performs prescanning (subject recognition) on the captured image data temporarily stored in step S8050 at a predetermined level. Specifically, a change point of a predetermined level or higher is extracted from the captured image data.

  In step S8150, using the result of step S8100, the position where the change point of a predetermined level or higher is extracted is stored in the subject area memory 232 of the main storage device 152 in association with the captured image data.

  In step S8200, it is determined whether or not to end the prescan. This determination is made based on whether there is a frame that has not been pre-scanned. That is, when there are a plurality of frames, the pre-scan is repeated by the number of frames. As a result of this determination, when pre-scanning is to be ended (S8200: YES), the process proceeds to step S8250, and when pre-scanning is not to be ended (S8200: NO), the process returns to step S8100.

  In step S8250, the restoration mode designation unit 210 and the area designation unit 134 of the operation unit 130 perform restoration mode setting and boundary area setting. The restoration modes that can be set are, for example, as shown in FIG. In setting the boundary area, the boundary area when the pre-scan is performed in step S8100 is confirmed.

  In step S8300, it is determined whether or not the subject area. Specifically, it is determined whether or not the processing target block (the target pixel and its surrounding pixels) includes a subject area. This determination can be made by, for example, automatically determining at which position in the deteriorated image the subject that the user is paying attention is based on the luminance value of the pixel at the stage of pre-scanning, and using this result. Alternatively, it is performed by allowing the user to selectively determine the subject area using the touch panel. As a result of this determination, if the processing target block includes the subject area (S8300: YES), the process proceeds to step S8350. If the processing target block does not include the subject area (S8300: NO), the process proceeds to step S8400.

  In step S8350, a one-dimensional image restoration method (for example, one of invention methods 1 to 3) corresponding to the restoration mode set in step S8250 is selected as an image restoration method to be executed. In step S8400, a two-dimensional image restoration method (for example, one of invention methods 4 to 6) corresponding to the restoration mode set in step S8250 is selected as the image restoration method to be executed. That is, in the present embodiment, whether to execute the one-dimensional restoration method or the two-dimensional restoration method is largely selected depending on whether or not the processing target block includes a subject area, and then the selected one-dimensional restoration method is selected. Or, select from 2D restoration methods, for example, depending on the restoration mode adapted to the captured image (photograph, text, etc.), quality (image quality, speed, etc.), or various uses (portrait, night view, etc.) The correct restoration method is selected.

  In step S8450, the image restoration processing unit 220 performs image restoration processing according to the set parameters using the image restoration method selected in step S8350 or step S8400. Here, the parameter setting is performed by the parameter setting unit 132 of the operation unit 130.

  In step S8500, it is determined whether image restoration processing has been completed for the frame. As a result of this determination, if the image restoration process has been completed for the frame (S8500: YES), the process proceeds to step S8550. If the image restoration process has not been completed for the frame (S8500: NO), the process proceeds to step S8300. Return.

  In step S8550, it is determined whether image restoration processing has been completed for all frames. As a result of this determination, if the image restoration processing has not been completed for all frames (S8550: NO), the process returns to step S8250, and if the image restoration processing has been completed for all frames (S8550: YES), The process ends. Therefore, when there are a plurality of frames, the image restoration process is repeated by the number of frames.

  As described above, according to the present embodiment, since a plurality of image restoration methods are adaptively switched and executed according to the situation, image restoration processing suitable for a pixel to be restored is performed with high accuracy and high speed with one chip. Can be done.

  The image restoration device and the image restoration method according to the present invention are useful as a simple and practical image restoration device and an image restoration method capable of improving the image restoration capability.

100, 200 Image restoration device 110 Image input device 112 Camera 114 Scanner 116, 186 Recording medium 118, 188 Modem 120 Input interface unit 130 Operation unit 132, 222 Parameter setting unit 134 Area designation unit 136, 210 Restore mode designation unit 140 Internal interface Unit 150 storage unit 152 main storage unit 154 auxiliary storage unit 156 display memory 160, 220 image restoration processing unit 160a first restoration processing unit 160b second restoration processing unit 162 initial setting unit 164 correlation calculation unit 166 weight coefficient calculation unit 168 optimal estimation Value calculation unit 170 Output interface unit 180 Image output device 182 Display 184 Printer 224 Subject area determination unit 230, 240 Captured image memory 232 Subject area memory 234 Processing parameter Meter memory 236, 244 Restored image memory 242 Processing content memory

Claims (22)

In the system to be observed that can be modeled, only from the observation signal that is the degraded image information in which blur and noise are mixed in the original image information, using a prediction method based on a state space model that includes the original image information as a drive source, An image restoration apparatus for estimating the original image information,
The state space model uses a processing target block including at least one or more target pixels to be restored and pixels around the target pixel as a processing unit. It is configured to move to the next processing target block so as to include pixels, and to estimate the state quantity of the processing target block from the observation amount corresponding to the processing target block at each time,
Against time n + 1 only observed signals, the first correlation calculation for calculating the correlation value of the estimated error when the state quantity of the target block in the observation time Ri by the signal time n + 1 at time n until the estimated And
Against time n + 1 only observed signals, using the correlation value calculated by the first correlation calculation unit, and the optimum estimated value of the state quantity at time n + 1 by the observation signal up to a time n + 1, until time n the optimum estimated value of the state quantity at time n + 1 by the observation signals, observables of the estimated error including an observation signal at time n + 1, and the weight coefficient calculation unit for calculating a weighting factor for defining the relationship,
Against time n + 1 only observed signals, a first optimum estimate calculation unit that issues calculate the optimal estimate of the state quantity at time n + 1 by the observation signal up to a time n,
Second optimum estimation for calculating an optimum estimated value of the state quantity at time n + 1 by the observation signal up to time n + 1, using the weighting factor calculated by the weighting factor calculation unit for the observation signal only at time n + 1 A value calculator,
A second correlation calculation unit that calculates a correlation value of an estimation error when the state quantity at time n + 1 is estimated from an observation signal up to time n + 1 with respect to an observation signal only at time n + 1;
An image restoration apparatus. The state space model uses the processing target block including the target pixel to be restored and pixels around the target pixel as a processing unit, and the state of the processing target block from all the observation amounts corresponding to the processing target block. Configured to estimate all quantities ,
Prior Symbol first correlation calculation unit,
Against time n + 1 only observed signal, calculates the correlation value matrix of the estimation error vector in the case of estimating a state quantity of the target block at time n + 1 containing the original image information by the observation signal up to a time n,
The weight coefficient calculation unit includes:
Against time n + 1 only observed signals, using the correlation value matrix of the estimation error vector calculated by said first correlation calculation section, the state quantity of the optimum estimated value at time n + 1 by the observation signal up to a time n + 1 vector and the optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to time n, the observed amount including an observed signal at time n + 1 estimation error vector and the weighting factor matrix for defining the relationship To calculate
The first optimal estimated value calculation unit includes:
Against time n + 1 only observed signals, calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n,
The second optimum estimated value calculation unit includes:
For the observation signal only at time n + 1 , using the weighting coefficient matrix calculated by the weighting coefficient calculation unit, calculate the optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to time n + 1,
The second correlation calculation unit includes:
Against time n + 1 only observed signals, calculates a correlation value matrix of the estimation error vector in the case of estimating the state quantity of the observation signal up to a time n + 1 at time n + 1,
The image restoration apparatus according to claim 1. The weight coefficient calculation unit includes:
The weighting coefficient matrix is calculated using a correlation value matrix of the estimated error vector calculated by the first correlation calculation unit, a given observation transition matrix, and a covariance matrix of the given observation noise vector.
The image restoration apparatus according to claim 2. The first correlation calculation unit includes:
By using a predetermined state transition matrix, a covariance matrix of a given drive source vector, and a correlation value matrix of a given or previously estimated error vector calculated by the second correlation calculation unit, Calculate the correlation value matrix,
The image restoration apparatus according to claim 2. In the state space model, the processing target block including the target pixel to be restored and pixels around the target pixel is used as a processing unit, and all the state quantities of the processing target block are calculated from the observation amounts corresponding to the target pixel. Configured to estimate ,
Prior Symbol first correlation calculation unit,
Against time n + 1 only observed signal, calculates the correlation value matrix of the estimation error vector in the case of estimating a state quantity of the target block at time n + 1 containing the original image information by the observation signal up to a time n,
The weight coefficient calculation unit includes:
Against time n + 1 only observed signals, using the correlation value matrix of the estimation error vector calculated by said first correlation calculation section, the state quantity of the optimum estimated value at time n + 1 by the observation signal up to a time n + 1 weighting coefficient vector for defining vectors and, and the optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to time n, the observation of the estimated error vector containing the observation signal at time n + 1, the relationship To calculate
The first optimal estimated value calculation unit includes:
Against time n + 1 only observed signals, calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n,
The second optimum estimated value calculation unit includes:
Against time n + 1 only observed signals, using the weight coefficient vector calculated by the weight coefficient calculation unit calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n + 1,
The second correlation calculation unit includes:
Against time n + 1 only observed signals, calculates a correlation value matrix of the estimation error vector in the case of estimating the state quantity of the observation signal up to a time n + 1 at time n + 1,
The image restoration apparatus according to claim 1. The weight coefficient calculation unit includes:
The weighting coefficient vector is calculated using the correlation value matrix of the estimated error vector calculated by the first correlation calculation unit, the given observation transition vector, and the given correlation noise correlation value scalar.
The image restoration apparatus according to claim 5. The first correlation calculation unit includes:
By using a predetermined state transition matrix, a covariance matrix of a given drive source vector, and a correlation value matrix of a given or previously estimated error vector calculated by the second correlation calculation unit, Calculate the correlation value matrix,
The image restoration apparatus according to claim 5. The state space model is based on an observation amount in a predetermined partial region included in the processing target block, with a processing target block including the target pixel to be restored and pixels around the target pixel as a processing unit. , Configured to estimate all the state quantities of the processing target block ,
Prior Symbol first correlation calculation unit,
Against time n + 1 only observed signal, calculates the correlation value matrix of the estimation error vector in the case of estimating a state quantity of the target block at time n + 1 containing the original image information by the observation signal up to a time n,
The weight coefficient calculation unit includes:
Against time n + 1 only observed signals, using the correlation value matrix of the estimation error vector calculated by said first correlation calculation section, the state quantity of the optimum estimated value at time n + 1 by the observation signal up to a time n + 1 vector and the optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to time n, the observed amount including an observed signal at time n + 1 estimation error vector and the weighting factor matrix for defining the relationship To calculate
The first optimal estimated value calculation unit includes:
Against time n + 1 only observed signals, calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n,
The second optimum estimated value calculation unit includes:
Against time n + 1 only observed signals, using the weight coefficient matrix calculated by the weight coefficient calculation unit calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n + 1,
The second correlation calculation unit includes:
Against time n + 1 only observed signals, calculates a correlation value matrix of the estimation error vector in the case of estimating the state quantity of the observation signal up to a time n + 1 at time n + 1,
The image restoration apparatus according to claim 1. The weight coefficient calculation unit includes:
The weighting coefficient matrix is calculated using a correlation value matrix of the estimated error vector calculated by the first correlation calculation unit, a given observation transition matrix, and a covariance matrix of the given observation noise vector.
The image restoration apparatus according to claim 8. The first correlation calculation unit includes:
By using a predetermined state transition matrix, a covariance matrix of a given drive source vector, and a correlation value matrix of a given or previously estimated error vector calculated by the second correlation calculation unit, Calculate the correlation value matrix,
The image restoration apparatus according to claim 8.   The image restoration apparatus according to claim 8, wherein the predetermined partial region includes an observation amount corresponding to the target pixel and an observation amount of a portion affected by a chromaticity driving source. The state space model has a processing target block including a target pixel area including at least one pixel to be restored and a pixel area around the target pixel area as a processing unit. It is configured to estimate all the state quantities of the processing target block from observed quantities ,
Prior Symbol first correlation calculation unit,
Against time n + 1 only observed signal, calculates the correlation value matrix of the estimation error vector in the case of estimating a state quantity of the target block at time n + 1 containing the original image information by the observation signal up to a time n,
The weight coefficient calculation unit includes:
Against time n + 1 only observed signals, using the correlation value matrix of the estimation error vector calculated by said first correlation calculation section, the state quantity of the optimum estimated value at time n + 1 by the observation signal up to a time n + 1 vector and the optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to time n, the observed amount including an observed signal at time n + 1 estimation error vector and the weighting factor matrix for defining the relationship To calculate
The first optimal estimated value calculation unit includes:
Against time n + 1 only observed signals, calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n,
The second optimum estimated value calculation unit includes:
Against time n + 1 only observed signals, using the weight coefficient matrix calculated by the weight coefficient calculation unit calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n + 1,
The second correlation calculation unit includes:
And to observe signals only the time n + 1, it calculates a correlation value matrix of the estimation error vector in the case of estimating the state amount at time n + 1 by the observation signal up to a time n + 1,
The image restoration apparatus according to claim 1. The weight coefficient calculation unit includes:
The weighting coefficient matrix is calculated using a correlation value matrix of the estimated error vector calculated by the first correlation calculation unit, a given observation transition matrix, and a covariance matrix of the given observation noise vector.
The image restoration apparatus according to claim 12. The first correlation calculation unit includes:
By using a predetermined state transition matrix, a covariance matrix of a given drive source vector, and a correlation value matrix of a given or previously estimated error vector calculated by the second correlation calculation unit, Calculate the correlation value matrix,
The image restoration apparatus according to claim 12. The state space model is a processing unit including a target pixel area including at least one pixel to be restored and a pixel area around the target pixel area as a processing unit. consists of observables to estimate all state variables of the processing target block,
Prior Symbol first correlation calculation unit,
Against time n + 1 only observed signal, calculates the correlation value matrix of the estimation error vector in the case of estimating a state quantity of the target block at time n + 1 containing the original image information by the observation signal up to a time n,
The weight coefficient calculation unit includes:
Against time n + 1 only observed signals, using the correlation value matrix of the estimation error vector calculated by said first correlation calculation section, the state quantity of the optimum estimated value at time n + 1 by the observation signal up to a time n + 1 vector and the optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to time n, the observed amount including an observed signal at time n + 1 estimation error vector and the weighting factor matrix for defining the relationship To calculate
The first optimal estimated value calculation unit includes:
Against time n + 1 only observed signals, calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n,
The second optimum estimated value calculation unit includes:
Against time n + 1 only observed signals, using the weight coefficient matrix calculated by the weight coefficient calculation unit calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n + 1,
The second correlation calculation unit includes:
Against time n + 1 only observed signals, calculates a correlation value matrix of the estimation error vector in the case of estimating the state quantity of the observation signal up to a time n + 1 at time n + 1,
The image restoration apparatus according to claim 1. The weight coefficient calculation unit includes:
The weighting coefficient matrix is calculated using a correlation value matrix of the estimated error vector calculated by the first correlation calculation unit, a given observation transition matrix, and a covariance matrix of the given observation noise vector.
The image restoration apparatus according to claim 15. The first correlation calculation unit includes:
By using a predetermined state transition matrix, a covariance matrix of a given drive source vector, and a correlation value matrix of a given or previously estimated error vector calculated by the second correlation calculation unit, Calculate the correlation value matrix,
The image restoration apparatus according to claim 15. The state space model is a predetermined target included in the processing target block, with a processing target block including a target pixel region including at least one pixel to be restored and a pixel region around the target pixel region as a processing unit. It is configured to estimate all the state quantities of the processing target block from observation amounts in some pixel regions ,
Prior Symbol first correlation calculation unit,
Against time n + 1 only observed signal, calculates the correlation value matrix of the estimation error vector in the case of estimating a state quantity of the target block at time n + 1 containing the original image information by the observation signal up to a time n,
The weight coefficient calculation unit includes:
Against time n + 1 only observed signals, using the correlation value matrix of the estimation error vector calculated by said first correlation calculation section, the state quantity of the optimum estimated value at time n + 1 by the observation signal up to a time n + 1 vector and the optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to time n, the observed amount including an observed signal at time n + 1 estimation error vector and the weighting factor matrix for defining the relationship To calculate
The first optimal estimated value calculation unit includes:
Against time n + 1 only observed signals, calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n,
The second optimum estimated value calculation unit includes:
Against time n + 1 only observed signals, using the weight coefficient matrix calculated by the weight coefficient calculation unit calculates an optimum estimated value vector of the state quantity at time n + 1 by the observation signal up to a time n + 1,
The second correlation calculation unit includes:
Against time n + 1 only observed signals, calculates a correlation value matrix of the estimation error vector in the case of estimating the state quantity of the observation signal up to a time n + 1 at time n + 1,
The image restoration apparatus according to claim 1. The weight coefficient calculation unit includes:
The weighting coefficient matrix is calculated using a correlation value matrix of the estimated error vector calculated by the first correlation calculation unit, a given observation transition matrix, and a covariance matrix of the given observation noise vector.
The image restoration device according to claim 18. The first correlation calculation unit includes:
By using a predetermined state transition matrix, a covariance matrix of a given drive source vector, and a correlation value matrix of a given or previously estimated error vector calculated by the second correlation calculation unit, Calculate the correlation value matrix,
The image restoration device according to claim 18.   The image restoration device according to claim 18, wherein the predetermined partial pixel region includes an observation amount in the target pixel region and an observation amount of a portion affected by a chromatic drive source. In the system to be observed that can be modeled, only from the observation signal that is the degraded image information in which blur and noise are mixed in the original image information, using a prediction method based on a state space model that includes the original image information as a drive source, An image restoration method for estimating the original image information,
The state space model uses a processing target block including at least one or more target pixels to be restored and pixels around the target pixel as a processing unit. It is configured to move to the next processing target block so as to include pixels, and to estimate the state quantity of the processing target block from the observation amount corresponding to the processing target block at each time,
Against time n + 1 only observed signals, the first correlation calculation for calculating the correlation value of the estimated error when the state quantity of the target block in the observation time Ri by the signal time n + 1 at time n until the estimated Process,
Against time n + 1 only observed signals, using the correlation value calculated by the first correlation operation step, the optimal estimate of the state quantity at time n + 1 by the observation signal up to a time n + 1, up to the time n the optimum estimated value of the state quantity at time n + 1 by the observation signal, the observation of the estimated error including an observation signal at time n + 1, and the weighting coefficient calculation step of calculating a weighting factor for defining the relationship,
Against time n + 1 only observed signals, a first optimum estimated value calculation step that gives calculate the optimal estimate of the state quantity at time n + 1 by the observation signal up to a time n,
A second optimum estimated value for calculating an optimum estimated value of the state quantity at time n + 1 from the observed signal up to time n + 1, using the weighting factor calculated in the weighting factor calculating step for the observation signal only at time n + 1. A calculation process;
A second correlation calculation step of calculating a correlation value of an estimation error when the state quantity at time n + 1 is estimated from an observation signal up to time n + 1 with respect to an observation signal only at time n + 1;
An image restoration method comprising:

Images