JP6552375B2 - IMAGE PROCESSING DEVICE, IMAGING DEVICE, AND IMAGE PROCESSING PROGRAM - Google Patents

Description

  The present invention relates to an image processing technique for performing image processing (image restoration processing) on an image generated by imaging.

  An image obtained by imaging a subject with an imaging device such as a digital camera has image degradation caused by spherical aberration, coma aberration, field curvature, astigmatism, etc. of an imaging optical system (hereinafter simply referred to as an optical system). The blur component as a component is included. Such a blur component is generated when a light beam emitted from one point of a subject to be collected again at one point on the imaging surface when an aberration is not caused and there is no influence of diffraction is formed by forming an image with a certain spread.

  The blur component referred to here is optically represented by a point spread function (PSF), which is different from blur due to defocus. In addition, color bleeding in a color image can also be said to be a difference in blurring for each wavelength of light with respect to axial chromatic aberration, chromatic spherical aberration, and chromatic coma aberration of the optical system.

An optical transfer function (OTF) obtained by Fourier-transforming a point spread function (PSF) is frequency component information of an aberration, and is represented by a complex number. The absolute value of the OTF, that is, the amplitude component is referred to as MTF (Modulation Transfer Function), and the phase component is referred to as PTF (Phase Transfer Function). MTF and PTF are frequency characteristics of an amplitude component and a phase component of image degradation due to aberration, respectively. Here, the phase component is expressed as the phase angle by the following equation. Re (OTF) and Im (OTF) represent the real part and the imaginary part of the OTF, respectively.
PTF = tan ^-1 (Im (OTF) / Re (OTF))
As described above, since the OTF of the optical system degrades the amplitude component and the phase component of the image, the degraded image is in a state where each point of the object is blurred asymmetrically like coma.

  The chromatic aberration of magnification occurs when the imaging position shifts due to the difference in imaging magnification for each wavelength of light, and this is acquired as a color component such as RGB according to the spectral characteristics of the imaging device. Therefore, in addition to the shift of the imaging position between RGB, the shift of the imaging position for each wavelength, that is, the spread of the image due to the phase shift occurs in each color component. For this reason, although the magnification chromatic aberration is not simply a color shift of parallel shift, the color shift is treated as the same as the magnification chromatic aberration unless otherwise described.

  As a method of correcting the deterioration of the amplitude component (MTF) and the deterioration of the phase component (PTF) in the deteriorated image (input image), it is known to use the information of the OTF of the optical system. This method is also referred to as image restoration and image restoration. Hereinafter, processing for correcting (reducing) a deteriorated image using the information of the OTF of the optical system is referred to as image restoration processing. And although details will be described later, as one of methods of image restoration processing, there is known a method of convolving an input image with an image restoration filter of real space having inverse characteristics of OTF.

  In order to effectively perform the image restoration process, it is necessary to obtain a more accurate OTF of the optical system. The OTF is an imaging parameter of the imaging device such as the type of the optical system, the focal length, the aperture value, and the distance to the subject. Varies depending on the combination. For this reason, it is necessary to change the image restoration filter or OTF used for the image restoration processing according to the combination of imaging parameters.

  However, if it is going to hold the data of the image restoration filter and OTF corresponding to the combination of all the imaging parameters, the amount of data to hold will be huge. In order to reduce the amount of data to be held, it is conceivable to limit the image restoration filter and the OTF as data to be held to those corresponding to discrete imaging parameters. In addition, holding data of coefficients of an approximate expression that approximates OTF can also reduce the amount of data compared to the case of directly holding an OTF or an image restoration filter. In this case, an OTF is obtained by substituting the coefficient into the approximate expression, and an image restoration filter is generated from the OTF.

  Although the amount of retained data can be reduced by these methods, as a further problem, when performing image restoration processing on a plurality of images obtained by imaging with different imaging parameters, an OTF is acquired for each image and the image is acquired. It is necessary to generate a recovery filter.

  Patent Document 1 discloses a method of holding an image restoration filter generated using OTF and determining whether or not the held image restoration filter can be reused from an imaging parameter at the time of imaging. In this method, when it is possible to reuse the image restoration filter, the image restoration filter that is held is reused. In Patent Document 2, the same image restoration filter (or an image restoration filter generated from the same OTF) is used for a plurality of images obtained by imaging within a certain imaging parameter range (imaging parameters slightly different from each other). A method for performing image restoration processing is disclosed.

JP, 2013-161278, A JP, 2012-156714, A

  However, in the method disclosed in Patent Document 1, in order to generate an image restoration filter for an image obtained by imaging with an imaging parameter different from the imaging parameter corresponding to the image restoration filter that is held, Then, processing is performed from acquisition of the imaging parameter. For this reason, it takes time to generate the image restoration filter.

  Further, in the method disclosed in Patent Document 2, since the same image restoration filter is used for images obtained by imaging with mutually different imaging parameters even if slightly, it is not necessarily good for all the images. Therefore, it is not always possible to perform an image restoration process.

  The present invention is an image processing capable of performing good image restoration processing on each image obtained by imaging with different imaging parameters while reducing the amount of held data and shortening the time required for the image restoration processing. Provide devices etc.

  An image processing apparatus according to one aspect of the present invention performs processing on an input image generated by an imaging unit whose imaging parameters are variable. The image processing apparatus acquires OTF data as data relating to an optical transfer function of an imaging unit according to an imaging parameter at the time of generation of an input image, and a filter generation unit that generates an image restoration filter using the OTF data; Image restoration processing means for performing image restoration processing using an image restoration filter on the input image, and OTF data used for generating the first filter which is an image restoration filter for the first image as the input image, Storage processing means for storing in the storage means in association with the imaging parameters at the time of generation of the first image. When generating the second filter that is an image restoration filter for the second image as the input image different from the first image, the filter generation means generates at least one of the OTF data stored in the storage means. When available OTF data exists as a result of searching for available OTF data as OTF data corresponding to an imaging parameter that matches or is less than or equal to the imaging parameter at the time of generating the second image Is characterized in that a second filter is generated using the available OTF data.

  An imaging apparatus having the imaging unit and the image processing apparatus also constitutes another aspect of the present invention.

  An image processing program according to another aspect of the present invention is a computer program that causes a computer to execute processing on an input image generated by an imaging unit whose imaging parameter is variable. The above process acquires OTF data as data relating to the optical transfer function of the imaging means according to the imaging parameters at the time of generation of the input image, and generates an image restoration filter using the OTF data; OTF data used to generate an image restoration process that performs an image restoration process using an image restoration filter on the first image and a first filter that is an image restoration filter for the first image as an input image Storage processing to be stored in the storage unit in association with the imaging parameter at the time of generation of the image. In the filter generation processing, when generating a second filter that is an image restoration filter for a second image as an input image different from the first image, the at least one OTF data stored in the storage unit is Search for available OTF data as OTF data associated with an imaging parameter that matches or differs from an imaging parameter at the time of generation of the image 2 when available OTF data exists Is characterized in that a second filter is generated using the available OTF data.

  According to the present invention, while reducing the amount of retained data and reducing the time required for generating an image restoration filter, that is, image restoration processing, it is favorable for each image obtained by imaging with different imaging parameters. Image restoration processing can be performed.

FIG. 1 is a block diagram showing the configuration of an imaging apparatus including an image processing apparatus according to an embodiment of the present invention. The figure which shows the RAW data in an Example. FIG. 1 is a block diagram showing a configuration of an image processing unit of an imaging device which is an embodiment. FIG. 6 is a diagram illustrating color components and image restoration components in an embodiment. The figure which shows the frequency characteristic according to color component in an Example. 6 is a flowchart showing image restoration processing in the embodiment. The figure which shows the image restoration filter in an Example. The figure which shows the image restoration filter in an Example. 3 is a flowchart showing processing in Embodiment 1. 6 is a flowchart showing processing in Example 2; 10 is a flowchart showing processing in the third embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

First, definitions of terms used in the embodiments described below and image restoration processing will be described.
"Input image"
The input image is a digital image generated using an output from an imaging device obtained by photoelectrically converting an object image formed by an imaging optical system in an imaging device, and is, for example, a RAW image (RAW data) having information of RGB color components It is. The input image is an image degraded by an optical transfer function (hereinafter referred to as OTF) including aberration of an imaging optical system including an optical element such as a lens or an optical filter.

  The imaging optical system may include a mirror (reflection surface) having a curvature. In addition, the imaging optical system may be detachable (replaceable) with respect to the imaging device. In an imaging device, an imaging system is configured by an imaging element and a signal processing unit that generates an input image using an output of the imaging element. The imaging device is constituted by a photoelectric conversion device such as a CMOS sensor or a CCD sensor.

The input image and the output image may be accompanied by imaging parameters such as a focal length and an aperture value of the imaging optical system and an imaging distance (subject distance), and various correction information for correcting the input image.
[Image recovery processing]
Let g (x, y) be the input image (degraded image) generated by the imaging device, f (x, y) be the original image (image not degraded), and have an optical transfer function (hereinafter referred to as OTF) A point spread function (hereinafter referred to as PSF) that is a Fourier pair is represented by h (x, y). In this case, the following equation (1) holds. Note that * indicates convolution (convolution integration or sum of products), and (x, y) indicates coordinates (position) on the input image.
g (x, y) = h (x, y) * f (x, y) (1)
When this equation (1) is Fourier transformed into a display format on the frequency plane, it becomes a product format for each frequency as in the following equation (2). H (u, v) is an OTF obtained by Fourier transform of h (x, y), which is a PSF. G (u, v) and F (u, v) are functions obtained by Fourier transforming g (x, y) and f (x, y), respectively. (U, v) indicate coordinates in a two-dimensional frequency plane, that is, frequency.
G (u, v) = H (u, v) · F (u, v) (2)
In order to obtain the original image from the degraded image, both sides of the above equation (2) may be divided by H (u, v) as in the following equation (3).
G (u, v) / H (u, v) = F (u, v) (3)
By performing an inverse Fourier transform of F (u, v), that is, G (u, v) / H (u, v) back to the real plane, a recovered image corresponding to the original image f (x, y) is obtained. It is done.
Here, assuming that the inverse Fourier transform of H ⁻¹ (u, v) is R, the original image is similarly processed by performing convolution processing on the image in the real plane as in the following equation (4). We can obtain a recovered image which is f (x, y).
g (x, y) * R (x, y) = f (x, y) (4)
R (x, y) in the equation (4) is an image restoration filter. When the input image is a two-dimensional image, generally the image restoration filter is also a two-dimensional filter having taps (cells) corresponding to the respective pixels of the two-dimensional image. Also, in general, the image restoration accuracy improves as the number of taps (cell number) of the image restoration filter increases, so this can be realized according to the required image quality as an output image, the image processing capability as an image processing apparatus, the spread width of PSF, etc. The number of taps is set.

  The image restoration filter needs to reflect at least the characteristics of the aberration, so it is completely different from the conventional horizontal and vertical three-tap edge emphasis filter (high pass filter) or the like. Further, since the image restoration filter is generated based on the OTF, it is possible to correct both the deterioration of the amplitude component and the phase component in the deteriorated image (input image) with high accuracy.

  However, the actual image includes a noise component. For this reason, when using the image restoration filter created by taking the perfect inverse of OTF as described above, not only the deteriorated image is recovered but also the noise component is significantly amplified. This is because the MTF is raised so that the MTF (amplitude component) of the imaging optical system is returned to 1 over the entire frequency in a state where the amplitude of noise is added to the amplitude component of the input image. The MTF, which is amplitude degradation due to the imaging optical system, returns to 1, but the power spectrum of the noise component also rises. As a result, the noise is amplified according to the degree to which the MTF is raised, that is, the recovery gain. Therefore, a good restored image cannot be obtained as an image for viewing from an input image containing a noise component.

  As a method of recovering an image including a noise component, for example, a method of controlling the degree of recovery according to the intensity ratio (SNR) of an image signal and a noise signal like a Wiener filter shown in Expression (5) It has been known.

M (u, v) indicates the frequency characteristic of the Wiener filter, and | H (u, v) | indicates the absolute value (MTF) of OTF. In an embodiment to be described later, M (u, v) in equation (5) represents the frequency characteristic of the image restoration filter. In this method, for each frequency, the recovery gain is suppressed as the MTF is small, and the recovery gain is increased as the MTF is large. Generally, the MTF of the imaging optical system is high on the low frequency side and low on the high frequency side, so that it is a method of suppressing the recovery gain on the high frequency side of the image signal substantially. A specific example of the image restoration filter will be described later.

  FIG. 1 shows the configuration of an imaging apparatus 100 equipped with an image processing apparatus according to a first embodiment of the present invention. In the image pickup apparatus 100, an image pickup optical system 101 including a diaphragm 101a and a focus lens 101b forms a light flux from an object (not shown) and forms an object image on the image pickup element 102. In this embodiment, the imaging optical system 101 is configured as an interchangeable lens, and is detachably attached to the imaging apparatus main body. However, the imaging optical system 101 may be integrally provided in the imaging apparatus 100.

  The aperture value of the imaging optical system 101 is set by the aperture 101a. A plurality of predetermined aperture values can be set by increasing or decreasing the aperture diameter of the aperture formed by the aperture 101a.

  The imaging element 102 photoelectrically converts the subject image formed by the imaging optical system 101 and outputs an analog imaging signal. An analog imaging signal output from the imaging element 102 is converted into a digital imaging signal by the A / D converter 103, and the digital imaging signal is input to the image processing unit 104. The imaging optical system 101 and the imaging element 102 constitute imaging means.

  The image processing unit 104 performs various image processing on the digital imaging signal to generate a captured image as a RAW image. Then, the image processing unit 104 as an image processing apparatus acquires the generated captured image as an input image for the above-described image restoration processing and other image processing. That is, the image processing unit 104 includes an image acquisition unit that acquires a captured image.

  Further, the image processing unit 104 acquires variable imaging parameter information in the imaging apparatus 100 from the state detection unit 107. The imaging parameters in this embodiment include the type of imaging optical system 101 (interchangeable lens) (hereinafter referred to as lens type), the aperture value of imaging optical system 101 at the time of imaging (hereinafter referred to as imaging aperture value), and the imaging optical system And the focal length when 101 is a zoom lens. Further, the imaging parameters include an imaging distance (or subject distance) corresponding to the position of the focus lens 101 b and the ISO sensitivity of the imaging element 102. The imaging parameters do not necessarily include all of the lens type, focal length, imaging distance, aperture value, and ISO sensitivity, and at least one of them may be included.

  The state detection unit 107 can directly acquire imaging parameter information from the system controller 110, and imaging parameter information regarding the imaging optical system 101 can be obtained from an optical system control unit 106 that controls the operation of the imaging optical system 101. It can also be acquired.

  The storage unit 108 stores (holds) OTF data for generating an image restoration filter or data indicating a result of arithmetic processing using the OTF (hereinafter, these data are collectively referred to as OTF data). ing. The storage unit 108 corresponds to a data storage unit.

  Furthermore, the image processing unit 104 includes an image restoration processing unit 111 and a correction processing unit 112. The image restoration processing unit 111 reads an image restoration filter corresponding to the imaging parameter from the storage unit 108, and generates an restored image by performing an image restoration process on the pickup image using the image restoration filter. When data related to the OTF is stored in the storage unit 108, the image recovery processing unit 111 reads out data related to the OTF corresponding to the imaging parameter, generates an image recovery filter corresponding to the imaging parameter, and performs image recovery processing. . In addition, the image restoration processing unit 111 causes the storage unit 108 to store data related to the OTF used for generating the image restoration filter. The image restoration processing unit 111 functions as a filter generation unit that performs filter generation processing, an image restoration processing unit that performs image restoration processing, and a storage processing unit that performs storage processing.

  The correction processing unit 112 performs image processing other than the image restoration processing on the captured image or the restored image before the image restoration processing. For example, since the restored image generated by the image restoration processing unit 111 is an image in which each pixel has only one color component among the three color components of RGB, the correction processing unit 112 applies pixels to the restored image. Interpolation processing produces an output image in which each pixel has all color components.

  The output image generated by the image processing unit 104 is recorded in a predetermined format on an image recording medium 109 such as a semiconductor memory or an optical disk. The display unit 105 performs predetermined display processing on the output image, and displays the display image after the display processing.

  All operations and processes in the imaging apparatus 100 configured as described above are controlled by the system controller 110. The optical system control unit 106 controls the drive of the diaphragm 101 a and the focus lens 101 b of the imaging optical system 101 based on an instruction from the system controller 110.

  The imaging optical system 101 may include an optical filter such as an optical low-pass filter or an infrared cut filter. When an optical filter that affects the OTF, such as an optical low-pass filter, is used, it is preferable to generate an image restoration filter corresponding to the influence. Even when the infrared cut filter is used, it affects each PSF of the RGB channel, particularly the PSF of the R channel, which is an integral value of the point spread function (PSF) of the spectral wavelength, and therefore, at the time of creating the image restoration filter. Consideration of In addition, since the shape of the pixel aperture of the image sensor 102 also affects the optical transfer function, it is preferable to generate the image restoration filter in consideration of the influence.

  FIG. 3 shows processing blocks in the image processing unit 104 of this embodiment. The captured image input to the image restoration processing unit 111 is RAW data having a color component of one color in each pixel, and an example thereof is shown in FIG. FIG. 2 shows RAW data of a Bayer array in which one pixel has R, G or B color components.

  In this embodiment, the G component is separated into G1 and G2 in the image restoration processing unit 111, and an image restoration filter is applied to four image restoration components of R, G1, G2 and B. As shown in FIG. 3, the four image restoration components are input to the four restoration filter application units 1110 to 1113, respectively.

  An example of each color component and each image restoration component in the RAW data is shown in FIG. FIGS. 4A, 4 B and 4 C respectively show G component, R component and B component which are three color components in RAW data. Pixels shown in white in the drawing represent respective color components. In this embodiment, the G component shown in FIG. 4A is divided into the G1 component and the G2 component shown in FIGS. 4D and 4E, and image restoration is performed for each of R, G1, G2, and B. Process. Thereby, the image restoration component R, the image restoration component B, the image restoration component G1 and the image restoration component G2 are generated.

  FIGS. 5A and 5B show the frequency characteristics of the pixel array for each color component in the imaging device 102. FIG. In these figures, a pixel (white) that can sense light of each color component shown in FIGS. 4A to 4E is set to 1, a pixel that can not detect light (black) is set to 0, and each pixel is set to m_G ( It is assumed that x, y), m_R (x, y), m_B (x, y), m_G1 (x, y), m_G2 (x, y). The frequency characteristics shown in FIGS. 5A and 5B are m_G (x, y), m_R (x, y), m_B (x, y), m_G1 (x, y), m_G2 (x, y). Is equivalent to the result of Fourier transform.

  FIG. 5A shows the frequency characteristic of the G component shown in FIG. This frequency characteristic is expressed as a comb-like function in which 1 is present only at positions indicated by ● in the figure. On the other hand, FIG. 5 (b) shows the frequency characteristics of the R component and the B component shown in FIGS. 4 (b) and 4 (c), and the frequency characteristics of the G component shown in FIG. The frequency characteristics are different. However, the frequency characteristics in the case where the G component is separated into the G1 component and the G2 component are as shown in FIG. 5 (b), similarly to the R component and the B component.

  The image restoration processing for the three color components R, G, and B has different frequency characteristics between the G component, the R component, and the B component as shown in FIGS. 5A and 5B. The false color which is not originally present in the image occurs in the area including. For this reason, in this embodiment, the G component is separated into the G1 component and the G2 component, so that the pixel arrays of the R, G1, G2, and B components exhibit the same frequency characteristics. As a result, since the frequency band to be subjected to the image restoration process is common, the generation of false colors due to the image restoration process can be suppressed.

  Even when image restoration processing is performed on three color components of R, G, and B, the frequency band of the G component is made to coincide with the frequency band of the R component and the B component depending on how to create the image restoration filter applied to the G component. It is possible. However, the frequency band recovered in that case is equivalent to the case where the G component is separated into the G1 component and the G2 component. As described above, since the image restoration filter is convoluted in the image restoration process, it is advantageous from the viewpoint of processing load to perform the image restoration process after separating the G component into the G1 component and the G2 component.

  Next, a basic flow of image restoration processing performed by the image processing unit 104 (image restoration processing unit 111) in the present embodiment will be described using the flowchart of FIG. An image processing unit 104, which is an image processing computer, executes this processing according to an image processing program as a computer program. The same applies to the image restoration processing shown in other flowcharts to be described later.

  In step S201, the image restoration processing unit 111 acquires the information of the imaging parameter described above from the state detection unit 107.

  Next, in step S202, the image recovery processing unit 111 generates the R, G1, G2 and B components shown in FIGS. 4B to 4E from the RAW data as the input image including the R, G and B components. Data (hereinafter referred to as color component data) is generated. At this time, R, G1, G2, and B are treated color components as data in which 0 is set to portions of color components other than the color components to be subjected to image restoration processing (hereinafter referred to as processing color components). Four color component data may be generated. In addition, four color component data may be generated with R, G1, G2, and B as processing target color components, respectively, as quarter-size data in which portions of color components other than the processing target color components are thinned out.

  Next, in step S203, the image restoration processing unit 111 is an image restoration filter corresponding to the imaging parameter acquired in step S201, and illustrates four image restoration filters to be applied to each of the R, G1, G2, and B components. 1 is read (selected) from the storage unit 108 shown in FIG.

  At this time, in order to reduce the amount of data stored in the storage unit 108, only the data of the image restoration filter corresponding to the plurality of imaging parameters discretely selected in advance may be stored in the storage unit 108. In this case, an image restoration filter corresponding to an actual imaging parameter is obtained by interpolation processing using data of an image restoration filter corresponding to two or more imaging parameters close to the actual imaging parameter among a plurality of discrete imaging parameters. May be generated. Further, when the data on the OTF for generating the image restoration filter is stored in the storage unit 108, the image restoration filter is generated from the data on the OTF corresponding to the imaging parameter.

  The image restoration filter will be described with reference to FIGS. 7 (a) and 7 (b). The image restoration filter shown in these figures is an example of the image restoration filter applied to each color component data (each color plane) of an image in which each pixel has three color components of RGB. The number of taps of the image restoration filter is determined according to the aberration characteristic of the imaging optical system and the required image restoration accuracy. In this example, an 11 × 11 tap 2D image restoration filter is shown. In this way, the image restoration filter is a two-dimensional filter having a tap number of 100 or more, so that it greatly spreads from the imaging position such as spherical aberration, coma aberration, axial chromatic aberration, off-axis color flare, etc. occurring in the imaging optical system. Deterioration due to aberration can be recovered satisfactorily.

  Although the values (coefficient values) in the respective taps are omitted in FIG. 7A, one cross section of the image restoration filter is shown in FIG. 7B. The distribution of the values in each tap of the image restoration filter serves to return the signal value (PSF) spatially spread by the aberration to an original one point ideally.

  In the image restoration process, the value of each tap of the image restoration filter is convoluted (also referred to as convolution integration or product-sum) with respect to a pixel corresponding to each tap in the input image. In the convolution process, in order to improve the signal value of a certain pixel, the pixel is matched with the center of the image restoration filter. Then, for each corresponding pixel of the input image and the image restoration filter, the product of the signal value of the input image and the value (coefficient value) of the tap of the image restoration filter is taken, and the sum is replaced as the signal value of the central pixel.

  The image restoration filter can be obtained by performing an inverse Fourier transform on a function designed based on the inverse function of the OTF of the photographing optical system. For example, in the case of using a Wiener filter, it is possible to create a real-space image restoration filter that is actually convoluted with the input image by performing inverse Fourier transform on Equation (5). In addition, the OTF for generating the image restoration filter can include not only the imaging optical system but also a factor that degrades the OTF with respect to the captured image input to the image processing unit 104. For example, the low pass filter suppresses high frequency components with respect to the frequency characteristic of OTF, and the shape and aperture ratio of the pixel aperture of the imaging device also affect the frequency characteristic of OTF. Besides, the spectral characteristics of the light source and the spectral characteristics of various wavelength filters affect the frequency characteristics of the OTF. It is desirable to create an image restoration filter based on OTF in a broad sense including these.

  The image restoration filters generated for each of the four color component data slightly differ from each other according to the chromatic aberration. That is, the cross-sectional view shown in FIG. 7B differs for each image restoration filter. Further, the tap arrangement of the image restoration filter does not necessarily have to be a square arrangement as shown in FIG. 7A, and can be set arbitrarily.

  FIGS. 8A and 8B show an example of an image restoration filter applied to RAW data in which each pixel has only one color component, with respect to the image restoration filter applied to each color plane. This image restoration filter is an image restoration filter that holds a filter coefficient other than 0 for pixels having a color component to be processed, and indicates a part holding a filter coefficient other than 0 in white in the figure, and a filter of 0 The part holding the coefficient is shown in black.

  FIG. 8A shows an image restoration filter applied to the R component and the B component when image restoration processing is performed on the three color components R, G, and B, and FIG. 8B shows the G component. Shows the image restoration filter applied to However, in this embodiment, since the G component is separated into the G1 component and the G2 component and the image restoration filter is applied to them, the image restoration filter shown in FIG. 8A is used for any of the R, G1, G2, and B components. Is used.

  In FIG. 6, next, in step S204, the image restoration processing unit 111 applies the image restoration filter for each color component (R, G1, G2, B) selected in step S203 to each pixel of the corresponding color component. Convolve. By this convolution processing, it is possible to remove or reduce an image degradation component due to the aberration generated in the imaging optical system 101. Also, chromatic aberration can be corrected by using an image restoration filter for each color component.

  The convolution process performed in this embodiment is a process for convolving the image restoration filter shown in FIG. 8A with respect to the four color component data shown in FIGS. 4B to 4E. In this process, the image restoration filter holding method and application method may be appropriately changed according to the type of each color component data separated in step S202. For example, when R, G1, G2, and B color component data in which 0 is set in the color component portion other than the processing target color component are prepared, the pixels to be subjected to convolution are limited to the processing target color component pixels. Thus, unnecessary calculations can be omitted. In addition, when 1 / 4-sized R, G1, G2, and B color component data obtained by thinning out color component portions other than the color component to be processed is prepared, image restoration is performed by thinning out coefficients other than the filter coefficients used. Prepare a filter. As a result, the image restoration filter can be directly applied to the color component data of ¼ size.

  The image restoration filter used in any of these cases is an image restoration filter applied to an image in which each pixel shown in FIG. 7A has three RGB color components, or an unseparated G component shown in FIG. 8B. The number of effective filter coefficients is smaller than the image restoration filter applied to. For this reason, the load of convolution processing can be reduced. Thus, the image restoration processing by the image restoration processing unit 111 is completed.

  Note that OTF changes according to the image height (field angle) even with the same imaging parameter. For this reason, it is desirable to perform image restoration processing by changing the image restoration filter for each image height. Specifically, in steps S203 and S204, it is desirable to select and apply a different image restoration filter for each pixel of each color component data.

  Further, although the application of the image restoration filter is described as the image restoration processing in this embodiment, other processing such as distortion correction processing, peripheral light amount correction processing and noise reduction processing may be performed before or after the processing shown in FIG. The entire process including the process may be inserted as an image restoration process.

  The four color component data on which the image restoration processing has been performed by the image restoration processing unit 111 are input to the correction processing unit 112. Since these four color component data are data for each color component in the Bayer array, the correction processing unit 112 performs interpolation processing on these color component data to obtain three colors of R, G, and B for each pixel. Generate image data having components. In addition to the interpolation processing, the correction processing unit 112 performs development processing such as gamma correction and color balance adjustment to generate data of a final output image.

  Next, image restoration processing (image restoration method) in the present embodiment will be described with reference to the flowchart shown in FIG. In step S301, the image recovery processing unit 111 applies an image recovery filter (second filter) to be applied to a captured image (second image: hereinafter referred to as a current process target image) which is a target of the current image recovery processing. Calculation processing of data related to OTF necessary for generation of. Specifically, the image restoration processing unit 111 calculates data related to the OTF (hereinafter referred to as necessary OTF data) necessary for generating an image restoration filter used in the image restoration process on color component data of the processing target image.

  Next, in step S302, the image restoration processing unit 111 performs data search processing. In this embodiment, when image restoration processing is performed on another captured image (first image) before, arithmetic processing necessary to generate an image restoration filter (first filter) is performed. Data related to the OTF (hereinafter referred to as processed OTF data) is stored in the storage unit 108. Arithmetic processing required to generate an image restoration filter (hereinafter referred to as filter generation arithmetic processing) is performed in step S306 described later. The image restoration processing unit 111 searches the used OTF data that can be reused (usable OTF data) in the storage unit 108 for the necessary OTF data calculated in step S301. In the present embodiment, whether or not processed OTF data can be reused is determined based on imaging parameters at the time of generation of a captured image (hereinafter, simply referred to as imaging parameters of a captured image).

  As described above, the imaging parameters in the present embodiment are the lens type, the focal length, the imaging distance, the aperture value, and the ISO sensitivity, and the processed OTF data stored in the storage unit 108 is the imaging of the other captured image. It is associated with the parameter. The image restoration processing unit 111 compares the imaging parameter associated with the processed OTF data with the imaging parameter of the current processing target image. Then, when these imaging parameters match, it is determined that the processed OTF data corresponding to the imaging parameter is reusable.

  In step S303, the image restoration processing unit 111 determines whether there is processed OTF data that can be reused in step S302. If there is, the process proceeds to step S304. The process proceeds to S305.

  In step S304, the image restoration processing unit 111 acquires, from the storage unit 108, processed OTF data corresponding to the imaging parameter of the current processing target image, that is, reusable. Then, the process proceeds to step S308.

  On the other hand, in step S305, the image restoration processing unit 111 acquires, from the storage unit 108, data that has not yet been subjected to filter generation calculation processing as data related to the OTF corresponding to the imaging parameter of the current processing target image. In the following description, the data related to the OTF that has not been subjected to the filter generation calculation process is referred to as unprocessed OTF data.

  Then, in the next step S306, the image restoration processing unit 111 performs filter generation calculation processing on the unprocessed OTF data acquired in step S305. Specifically, the filter generation calculation process weights each OTF data used for filter generation according to the imaging parameter.

  In step S307, the image restoration processing unit 111 stores the processed OTF data obtained as a result of the filter calculation process in step S306 in the storage unit 108 in association with the imaging parameter of the current processing target image. Then, the process proceeds to step S308.

  In step S308, the image restoration processing unit 111 determines whether or not all processed OTF data corresponding to the necessary OTF data for each of R, G1, G2, and B color component data has been acquired, that is, all used for image restoration processing. It is determined whether or not the image restoration filter can be generated. If the processed OTF data corresponding to all necessary OTF data has not been acquired, the process returns to step S302 and repeats the process of step SS304 or steps SS305 to S307. If the processed OTF data corresponding to all necessary OTF data is acquired, the process proceeds to step S309.

  As described above, in the present embodiment, the image restoration processing unit 111 performs processing for acquiring processed OTF data corresponding to one required OTF data in steps S302 to S308. Then, it is repeated until all of the processed OTF data corresponding to the required OTF data for each of the four color component data is acquired. On the other hand, necessary OTF data for all four color component data may be calculated in step S302, and processed OTF data corresponding to these necessary OTF data may be acquired in a lump in subsequent steps S303 to S307. .

  For example, eight necessary OTF data are calculated in step S302, there are reusable processed OTF data corresponding to some of them (here, four), and processed OTF data corresponding to the remaining four are stored. The case of newly acquiring will be described. In this case, four reusable processed OTF data are acquired from the storage unit 108 in step S304, and the remaining four unprocessed OTF data are processed at once in steps S305 to S307. As a result, the process can proceed to step S309 without repeating the steps S302 to S307 and determining whether the image restoration filter can be generated in the step S308.

  In step S309, the image restoration processing unit 111 generates all the image restoration filters necessary for the image restoration process using the processed OTF data acquired in step S304 and step S306.

  Then, in step S310, the image restoration processing unit 111 performs an image restoration process on the captured image (each color component data) using the image restoration filter generated in step S309. Thus, the image restoration process is ended.

  In the present embodiment, the processed OTF data used when the image restoration process for another captured image was previously performed is stored in the storage unit 108 in association with the imaging parameter, and this is stored in the current image restoration process. Reusable. By reusing the processed OTF data in the current image restoration process, the filter generation calculation process for the newly used unprocessed OTF data can be minimized, and the time required for the image restoration process can be shortened. .

  If the processed OTF data is not yet stored in the storage unit 108, such as when image restoration processing is performed for the first time, it is determined that there is no processed OTF data that can be reused in step S303, and steps S305 to S307 are performed. It is sufficient to carry out the processing. Thereby, the time required for the image restoration process for the next captured image can be shortened.

  Next, a second embodiment of the present invention will be described. In the present embodiment, the time required for the image restoration process is further shortened by determining the imaging parameters in more detail and reusing the image restoration filter itself as compared with the first embodiment. In the present embodiment, the components common to the components of the first embodiment are assigned the same reference numerals as the first embodiment.

  The flowchart in FIG. 10 shows the flow of image restoration processing performed by the image processing unit 104 (image restoration processing unit 111) in the present embodiment.

  Also in the present embodiment, processed OTF data used when an image restoration process was previously performed on another captured image is stored in the storage unit 108 in association with the imaging parameter of the captured image. In this embodiment, the image restoration filter generated in the previous image restoration process is also stored in the storage unit 108 in association with the imaging parameter. The data of the captured image includes information of lens type, focal length, imaging distance, aperture value, and imaging parameter which is ISO sensitivity.

  In step S <b> 401, the image restoration processing unit 111 compares the imaging parameter of the current processing target image with the imaging parameter associated with the processed OTF data and the image restoration filter stored in the storage unit 108. Then, the process proceeds to step S402.

  In step S402, the image restoration processing unit 111 stores in the storage unit 108 imaging parameters (hereinafter referred to as common imaging parameters) that match all of the lens type, focal length, imaging distance, aperture value, and ISO sensitivity in the comparison in step S401. To determine if it exists. In addition, the image restoration processing unit 111 sets an imaging parameter (hereinafter, referred to as a similar imaging parameter) in which all differences of the lens type, focal length, imaging distance, aperture value, and ISO sensitivity are equal to or less than a predetermined value It is also determined whether or not it exists in the storage unit 108. When the common imaging parameter or the similar imaging parameter exists in the storage unit 108, the process proceeds to step S403, and when none exists, the process proceeds to step S404.

  In step S403, the image restoration processing unit 111 acquires, from the storage unit 108, a reusable image restoration filter (usable filter) associated with the common or similar imaging parameter.

  On the other hand, in step S404, the image restoration processing unit 111 determines whether all of the imaging parameters other than ISO sensitivity match in the comparison in step S401 or whether the difference of imaging parameters other than ISO sensitivity is equal to or less than the predetermined value. judge. That is, it is determined whether or not there is an imaging parameter that is common or similar only for the lens type, focal length, imaging distance, and aperture value (hereinafter referred to as a partially common or partially similar imaging parameter) in the storage unit 108. If the partial common or partial similar imaging parameter exists in the storage unit 108, the process proceeds to step S405, and if not, the process proceeds to step S406.

  In step S405, the image restoration processing unit 111 acquires processed OTF data associated with the partial common or partial similar imaging parameter from the storage unit 108. The difference in ISO sensitivity relates only to the restoration strength of the image restoration filter, not to the data of the image restoration filter itself. Therefore, in this step, the image restoration processing unit 111 collectively acquires all processed OTF data necessary for generating an image restoration filter. After this, the process proceeds to step S414.

  In steps S406 to S413, the image restoration processing unit 111 performs the same processing as steps S301 to S308 in the first embodiment (FIG. 9). If processed OTF data corresponding to all necessary OTF data is not acquired in step S413, the process returns to step S406, and if processed OTF data corresponding to all necessary OTF data is acquired, the process proceeds to step S414. Also in this embodiment, necessary OTF data for all four color component data is calculated in step S406, and processed OTF data corresponding to these necessary OTF data is collectively acquired in subsequent steps S407 to S412. May be.

  In step S414, the image restoration processing unit 111 generates all image restoration filters necessary for the image restoration processing using the processed OTF data acquired in steps S405, S409, and S412.

  Next, in step S415, the image restoration processing unit 111 stores the image restoration filter generated in step S414 in the storage unit 108 in association with the imaging parameter of the current processing target image.

  In step S416, the image restoration processing unit 111 performs image restoration processing on the captured image (each color component data) using the reusable image restoration filter acquired in step S403 or the image restoration filter generated in step S414. Thus, the image restoration process is ended.

  In the present embodiment, the processed OTF data and the image restoration filter that were used when the image restoration process for another captured image was performed previously are stored in the storage unit 108 in association with the imaging parameter, and these are stored in the storage unit 108. Reusable in the image restoration process. By reusing the image restoration filter in the current image restoration process, the time required for the image restoration process can be shortened more than in the first embodiment. Further, when only the ISO sensitivity is different among the imaging parameters, all processed OTF data necessary for generating the image restoration filter can be acquired at a time, thereby reducing the time required for the determination and search processing. be able to.

  Next, Embodiment 3 of the present invention will be described. In this embodiment, the time required for the image restoration process can be further shortened by determining the presence or absence of the change of the interchangeable lens (lens type) having the imaging optical system 101. In the present embodiment, the components common to the components of the first and second embodiments are given the same reference numerals as the first and second embodiments.

  The flowchart of FIG. 11 shows the flow of image restoration processing performed by the image processing unit 104 (image restoration processing unit 111) in this embodiment. Also in this embodiment, the processed OTF data used when the image restoration processing was performed on another captured image before is stored in the storage unit 108 in association with the imaging parameter of the captured image. Further, in the present embodiment, the image restoration filter generated in the previous image restoration processing is also stored in the storage unit 108 in association with the imaging parameter. The captured image data includes information on imaging parameters such as lens type, focal length, imaging distance, aperture value, and ISO sensitivity.

  In step S501, the image restoration processing unit 111 sets the lens type included in the imaging parameter of the current processing target image to the imaging parameter associated with the processed OTF data and the image restoration filter stored in the storage unit 108. Compare with the lens types included in.

  Next, in step S502, the image restoration processing unit 111 determines whether the lens types do not match in the comparison of step S501, that is, whether the interchangeable lens has been changed (replaced). If the interchangeable lens has been changed, the process proceeds to step S503, and if not changed, the process proceeds to step S507.

  In step S503, the image recovery processing unit 111 stores the processed OTF data and the unnecessary storage data as the image recovery filter stored in the storage unit 108 when the image recovery process was previously performed on another captured image. Delete (discard) from the part 108. Since the OTF varies depending on the optical characteristics of the interchangeable lens, if the interchangeable lens is changed, the processed OTF data and the image restoration filter that are already stored in the storage unit 108 cannot be used. Therefore, in this embodiment, when the interchangeable lens is changed, unnecessary storage data stored in the storage unit 108 corresponding to the interchangeable lens before the change is deleted.

  However, the process of this step is not necessarily performed. For example, when the storage unit 108 has a capacity for storing processed OTF data and image restoration filters corresponding to a plurality of interchangeable lenses, the process of this step may be omitted. In this case, when performing an image restoration process on a captured image generated by imaging again using the removed interchangeable lens, reuse the stored processed OTF data and the image restoration filter. Can do.

  Next, in steps S504 to S506, the image restoration processing unit 111 performs the same processing as steps S305 to S307 in the first embodiment (FIG. 9) and steps S410 to S412 in the second embodiment (FIG. 10). Then, the process proceeds to step S520.

  On the other hand, in steps S507 to S519, the image restoration processing unit 111 performs the same processing as steps S401 to S413 in the second embodiment. If the processed OTF data corresponding to all necessary OTF data is not acquired in step S519, the process returns to step S513, and if the processed OTF data corresponding to all necessary OTF data is acquired, the process returns to step S520. move on. Note that when the lens type determined in step S501 indicates a single focus lens, the focal length is constant, so the focal length as an imaging parameter used for searching for processed OTF data may not be used.

  Also in the present embodiment, necessary OTF data for all four color component data is calculated in step S512, and processed OTF data corresponding to these necessary OTF data is collectively acquired in subsequent steps S513 to S518. May be.

  In step S520, the image restoration processing unit 111 generates all the image restoration filters necessary for the image restoration processing using the processed OTF data acquired in steps S511, S515, and S518.

  Next, in step S521, the image restoration processing unit 111 stores the image restoration filter generated in step S520 in the storage unit 108 in association with the imaging parameter of the current processing target image.

  In step S522, the image restoration processing unit 111 performs image restoration processing on the captured image (each color component data) using the reusable image restoration filter acquired in step S509 or the image restoration filter generated in step S520. Thus, the image restoration process is ended.

  In the present embodiment, prior to the search for the processed OTF data in step S513, the types of mounted interchangeable lenses are compared. Then, when the interchangeable lens is changed, the processed OTF data and the image restoration filter corresponding to another interchangeable lens stored in the storage unit 108 are discarded. As a result, when the interchangeable lens is changed, an extra search for the processed OTF data corresponding to another interchangeable lens in step S513 is not necessary, and the time required for the image restoration process can be further shortened.

In the first to third embodiments, the image pickup apparatus equipped with the image processing apparatus (the image processing unit 104) has been described. However, the image restoration process described in the first to third embodiments follows the image processing program installed in the personal computer It can also be done by a personal computer. In this case, the personal computer corresponds to the image processing apparatus as the fourth embodiment of the present invention. The personal computer acquires a captured image (input image) generated by the imaging device before the image restoration processing from the imaging device via wired / wireless communication or from another personal computer via a line such as the Internet. . The captured image may be acquired via an image recording medium in which the captured image is recorded. Then, the personal computer that has acquired the captured image performs an image restoration process according to an image processing program, and outputs an output image such as a restored image obtained as a result.
(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  The embodiments described above are only representative examples, and various modifications and changes can be made to the embodiments when the present invention is implemented.

101 Imaging optical system 104 Image processing unit 108 Storage unit 111 Image restoration processing unit

Claims (8)

An image processing apparatus that performs processing on an input image generated by an imaging unit whose imaging parameters are variable,
A filter generation unit that acquires OTF data as data on an optical transfer function of the imaging unit according to the imaging parameter at the time of generation of the input image, and generates an image restoration filter using the OTF data;
Image restoration processing means for performing an image restoration process using the image restoration filter on the input image;
The OTF data used to generate the first filter that is the image restoration filter for the first image as the input image is associated with the imaging parameter at the time of generation of the first image and stored in the storage unit Storage processing means for storing,
When the filter generation unit generates a second filter that is the image restoration filter for a second image as the input image different from the first image,
In at least one of the OTF data stored in the storage means, as the OTF data associated with an imaging parameter that matches or differs from the imaging parameter at the time of generation of the second image. Search for available OTF data,
An image processing apparatus characterized by generating the second filter using the available OTF data when the available OTF data exists.   The filter generation means includes a part of the plurality of OTF data used for generating the second filter as the usable OTF data, and the remaining OTF data does not exist as the usable OTF data. 2. The image processing apparatus according to claim 1, wherein the second filter is generated using the available OTF data and the newly acquired remaining OTF data.   The imaging parameter includes at least one of a type of an optical system included in the imaging unit, a focal length of the optical system, an aperture value of the optical system, a distance from the imaging unit to a subject, and ISO sensitivity of the imaging unit. The image processing apparatus according to claim 1, further comprising: The storage processing unit stores the first filter in the storage unit in association with the imaging parameter at the time of generation of the first image,
The filter generation unit
The at least one first filter stored in the storage means is associated with an imaging parameter that matches or differs from the imaging parameter at the time of generation of the second image. Search for available filters as one filter,
The image processing apparatus according to any one of claims 1 to 3, wherein when the available filter is present, the available filter is acquired as the second filter. The filter generation means includes
Determining the type of optical system of the imaging means;
When the storage data of the storage means is data for an optical system of a type different from the determined type of optical system, the OTF data used to generate the second filter is acquired without performing the search. The image processing apparatus according to any one of claims 1 to 4, characterized in that: The storage processing means includes
Determining the type of optical system of the imaging means;
The storage data is deleted when the storage data of the storage means is data for a type of optical system different from the type of the determined optical system, according to any one of claims 1 to 5, wherein the storage data is deleted. The image processing apparatus described. Imaging means with variable imaging parameters;
6. An imaging apparatus comprising: the image processing apparatus according to claim 1 that performs processing on an input image generated by the imaging unit. A computer program for causing a computer to execute processing on an input image generated by an imaging means whose imaging parameters are variable,
The process is
A filter generation process of acquiring OTF data as data relating to an optical transfer function of the imaging unit according to the imaging parameter at the time of generation of the input image, and generating an image restoration filter using the OTF data;
An image restoration process for performing an image restoration process using the image restoration filter on the input image;
The OTF data used to generate the first filter that is the image restoration filter for the first image as the input image is associated with the imaging parameter at the time of generation of the first image and stored in the storage unit Storage processing to be stored,
The filter generation process generates a second filter that is the image restoration filter for the second image as the input image different from the first image.
In at least one of the OTF data stored in the storage means, as the OTF data associated with an imaging parameter that matches or differs from the imaging parameter at the time of generation of the second image. Search for available OTF data,
An image processing program that generates the second filter using the available OTF data when the available OTF data exists.