JP2016092676A - Image processing apparatus, imaging apparatus, camera system and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an imaging apparatus, a camera system and an image processing system, capable of image restoration processing to make a data amount of an optical transfer function, which is used for image restoration processing, to be in conformity of the specification of the imaging apparatus.SOLUTION: The image processing apparatus includes: selection means for selecting partial optical transfer functions from supply means which stores a set of optical transfer functions proper to an imaging optical system; storage means for storing the partial optical transfer function; and image processing means for generating an image restoration filter corresponding to the imaging condition of the imaging optical system among the partial optical transmission functions and for performing image restoration processing using the image restoration filter.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, an imaging device, a camera system, and an image processing system, and more particularly to correction of a deteriorated image using image restoration processing.

  Along with the digitization of information, various correction processing methods have been proposed for captured images by handling images as signal values. When a subject is imaged with a digital camera and imaged, the obtained image is deteriorated not a little due to aberrations of the imaging optical system and light diffraction.

  The blur component of the image is caused by spherical aberration, coma, curvature of field, astigmatism, etc. of the optical system. When there is no aberration and no influence of diffraction, the light beam emitted from one point of the subject is essentially the imaging surface. What should be gathered again at one point above is the one that forms an image with a spread. Optically, this is called a point spread function (PSF), and it is called a blur component in an image. Speaking of image blur, for example, an out-of-focus image is blurred, but here it refers to an image that is blurred due to the effects of aberrations of the optical system, even when the image is in focus. In addition, regarding color bleeding in a color image caused by axial chromatic aberration, chromatic spherical aberration, and color coma in the optical system, it can be said that the blurring is different for each wavelength of light. In addition, when the lateral color shift is caused by the chromatic aberration of magnification of the optical system, it can be said that it is a position shift or a phase shift due to a difference in imaging magnification for each wavelength of light.

  An optical transfer function (OTF, Optical Transfer Function) obtained by Fourier transforming PSF is aberration frequency component information and is represented by a complex number. The absolute value of the OTF, that is, the amplitude component is called MTF (Modulation Transfer Function), and the phase component is called PTF (Phase Transfer Function). Therefore, MTF and PTF are frequency characteristics of an amplitude component and a phase component of image degradation due to aberration, respectively. Here, the phase component is expressed as the phase angle by the following formula. Re (OTF) and Im (OTF) represent the real part and the imaginary part of the OTF, respectively.

PTF = tan ⁻¹ (Im (OTF) / Re (OTF))
As described above, since the OTF of the imaging optical system deteriorates the amplitude component and the phase component of the image, the deteriorated image is in a state where each point of the subject is asymmetrically blurred like coma aberration.

  Further, the chromatic aberration of magnification is generated by shifting the imaging position due to the difference in imaging magnification for each wavelength of light, and acquiring this as, for example, RGB color components according to the spectral characteristics of the imaging device. Therefore, not only the image formation position is shifted between RGB, but also the image formation position shift for each wavelength, that is, the spread of the image due to the phase shift, occurs in each color component.

  There is known a method of correcting the deterioration of MTF and the deterioration of PTF using the information of the OTF of the imaging optical system. This method is called image restoration processing or image restoration processing. As an example of the image restoration processing, a method of convolving an image restoration filter having an inverse characteristic of OTF with an input image (convolution) is known. .

  In order to effectively perform the image restoration process, it is necessary to obtain more accurate OTF information of the imaging optical system. For example, if there is design value information of the imaging optical system, the OTF information can be obtained by calculation from the information. It can also be obtained by photographing a point light source and subjecting the intensity distribution to Fourier transform.

  The OTF of a general imaging optical system excluding the imaging optical system designed and manufactured with extremely high performance varies greatly depending on the image height (image position). For this reason, in order to perform image restoration processing of a captured image with high accuracy, it is necessary to use an image restoration filter generated based on a variation in OTF of each image height. In order to change the recovery characteristics in accordance with the position of the image, it is desirable to perform the image recovery process while switching the image recovery filter in the real space, rather than performing it all at once in the frequency space.

  Further, as a cause of blurring of the image, there is a light diffraction phenomenon depending on the aperture value (F number) of the imaging optical system. FIG. 12 is a diagram showing the relationship between the aperture value and the diffraction limit. The horizontal axis indicates the spatial frequency and the vertical axis indicates the MTF. As the aperture value increases, the cutoff frequency shifts to the low frequency side. For example, since the Nyquist frequency of an image sensor with a pixel size of 4 μm is 125 lines / mm, the influence is small when it is bright such as F2.8, but it is large when it is dark such as F16 and F32. Since the diffraction phenomenon can be described by OTF or PSF like the aberration, blur due to diffraction can be corrected by image restoration processing.

  In Patent Document 1, in an imaging device to which an imaging optical system can be attached and detached, an image restoration filter is generated in consideration of the characteristics of the OTF of the imaging optical system and the imaging unit. Patent Document 2 discloses an image processing apparatus that stores an OTF used for image restoration processing as a coefficient and performs image restoration processing corresponding to various imaging conditions of the imaging apparatus.

Japanese Patent No. 5162029 Japanese Patent No. 5153846

  However, since the OTF also changes depending on the aperture, the shooting distance, the shooting condition of the focal length of the zoom lens, and the image height in the screen, the total amount of highly accurate OTF data becomes very large. Therefore, when high-accuracy image restoration processing is performed by the image processing unit of the imaging apparatus, it is necessary to store OTF data having a large data amount in the storage unit of the imaging apparatus. Furthermore, in the case of an interchangeable lens camera, there are a plurality of lenses that can be attached to and detached from one camera, so that the capacity and cost of the storage unit increase in order to hold OTF data of the plurality of lenses. As a method of reducing the data amount of OTF, it is conceivable that the above-described shooting conditions and the number of data divisions of image height are coarsely used for interpolation, and the accuracy of OTF and the data amount are traded off.

  On the other hand, there are a plurality of camera models corresponding to one detachable lens. These cameras vary from those requiring higher image quality to those requiring lower cost. Therefore, it is preferable to change the roughness of the OTF data corresponding to the photographing condition and image height of a certain lens depending on the camera model.

  However, neither Patent Documents 1 and 2 describe that the amount of OTF data used is changed.

  In view of such a problem, the present invention provides an image processing apparatus, an imaging apparatus, and a camera capable of performing an image restoration process by making the data amount of an optical transfer function used for the image restoration process correspond to the specifications of the imaging apparatus. It is an object to provide a system and an image processing system.

  An image processing apparatus according to one aspect of the present invention includes a selection unit that selects a part of an optical transfer function from a supply unit that stores a set of optical transfer functions unique to an imaging optical system, and the part of the optical transfer function. An image restoration filter is generated using storage means for storing and an optical transfer function corresponding to a photographing condition of the imaging optical system among the partial optical transfer functions, and image restoration processing is performed using the image restoration filter And an image processing means.

  An image processing apparatus according to another aspect of the present invention selects an image restoration filter generated using a part of the optical transfer function from a supply unit that stores a set of optical transfer functions unique to the imaging optical system. Selection means, storage means for storing the image restoration filter, and image processing means for performing image restoration processing using an image restoration filter corresponding to a photographing condition of the imaging optical system among the image restoration filters. It is characterized by.

  An imaging apparatus according to another aspect of the present invention includes an imaging element that photoelectrically converts a subject image formed by an imaging optical system to generate a captured image, and a set of optical transfer functions unique to the imaging optical system. Corresponding to the imaging conditions of the imaging optical system among the partial optical transfer functions, the selection means for selecting a partial optical transfer function from the supply means for storing, the storage means for storing the partial optical transfer function, And image processing means for generating an image restoration filter using the optical transfer function to perform image restoration processing of the captured image using the image restoration filter.

  In addition, a camera system according to another aspect of the present invention includes an imaging optical system, an imaging element that photoelectrically converts a subject image formed by the imaging optical system to generate a captured image, and the imaging system. Selection means for selecting a part of the optical transfer function from supply means for storing a set of optical transfer functions, storage means for storing the part of the optical transfer function, and the imaging optics among the part of the optical transfer function Image processing means for generating an image restoration filter using an optical transfer function corresponding to a photographing condition of the system and performing image restoration processing of the photographed image using the image restoration filter.

  An image processing system according to another aspect of the present invention includes a supply unit that stores a set of optical transfer functions unique to the imaging optical system, a selection unit that selects a part of the optical transfer functions from the supply unit, The image recovery filter is generated using a storage unit that stores the partial optical transfer function and an optical transfer function corresponding to a photographing condition of the imaging optical system among the partial optical transfer functions, and the image recovery Image processing means for performing image restoration processing using a filter.

  According to the present invention, there is provided an image processing apparatus, an imaging apparatus, a camera system, and an image processing system capable of performing an image restoration process by making the data amount of an optical transfer function used for the image restoration process correspond to the specification of the imaging apparatus. Can be provided.

1 is a block diagram of a camera system according to an embodiment of the present invention. It is a flowchart until it stores the selected optical transfer function in a memory | storage part. It is a flowchart until it recovers and outputs a picked-up image using the stored optical transfer function. It is explanatory drawing of the modification of the image processing method of Example 1. FIG. It is explanatory drawing of an image restoration filter. It is explanatory drawing of an image restoration filter. It is explanatory drawing of a point spread function. It is explanatory drawing of the amplitude component and phase component of an optical transfer function. It is explanatory drawing of the example of selection of an optical transfer function. 6 is a flowchart of an image processing method according to the second embodiment. 1 is a block diagram of an image processing system according to an embodiment of the present invention. It is explanatory drawing of the relationship between an aperture value and a diffraction limit.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted. First, definitions of terms and image restoration processing described in this embodiment will be described. The image processing method described here is appropriately used in each embodiment described later.
[Input image]
The input image is a digital image (photographed image) obtained by receiving light with an image sensor through an imaging optical system, and an optical transfer function (diffraction and diffraction) of the imaging optical system including a lens and various optical filters. OTF). The imaging optical system may be configured using not only a lens but also a mirror (reflection surface) having a curvature.

  The color component of the input image has information on RGB color components, for example. As the color component, other commonly used color spaces such as brightness, hue, and saturation expressed in LCH, luminance expressed in YCbCr, and color difference signals can be selected and used. As other color spaces, XYZ, Lab, Yuv, and JCh can be used. Color temperature may also be used.

The input image and the output image can be accompanied by shooting conditions such as a focal length of the lens, an aperture value, and a shooting distance, and various correction information for correcting the image.
[Image recovery processing]
Next, an outline of the image restoration process will be described. The captured image (degraded image) is g (x, y), the original image is f (x, y), and the point spread function (PSF) that is the Fourier pair of the optical transfer function is h (x, y). When the following equation holds:

g (x, y) = h (x, y) * f (x, y) (1)
* Is convolution (convolution integration, sum of products), and (x, y) are coordinates on the photographed image.

  Further, when Formula (1) is Fourier-transformed and converted into a display format on the frequency plane, Formula (2) that can be expressed by a product for each frequency is obtained.

G (u, v) = H (u, v) · F (u, v) (2)
H is an OTF obtained by Fourier transform of a point spread function (PSF) h, and G and F are images obtained by Fourier transform of the degraded image g and the original image f, respectively. (U, v) is a coordinate on a two-dimensional frequency plane, that is, a frequency.

  In order to obtain the original image f from the photographed image g, both sides may be divided by OTF as shown in the following equation (3).

G (u, v) / H (u, v) = F (u, v) (3)
The original image f (x, y) is obtained as a restored image by performing inverse Fourier transform on F (u, v), that is, G (u, v) / H (u, v), and returning it to the actual surface. .

When R is the result of inverse Fourier transform of H− ¹ , the original image f (x, y) is similarly obtained by performing convolution processing on the actual image as in the following equation (4). Can be obtained.

g (x, y) * R (x, y) = f (x, y) (4)
R (x, y) is called an image restoration filter. When the image is a two-dimensional image, generally the image restoration filter R is also a two-dimensional filter having taps (cells) corresponding to each pixel of the image. In general, the greater the number of taps (number of cells) of the image recovery filter R, the higher the recovery accuracy. Therefore, the number of taps that can be realized is set according to the required image quality, image processing capability, aberration characteristics, and the like. Since the image restoration filter R needs to reflect at least aberration and diffraction characteristics, the image restoration filter R is different from a conventional edge enhancement filter (high-pass filter) of about 3 taps each in horizontal and vertical directions. Since the image restoration filter R is based on OTF, both deterioration of the amplitude component and the phase component can be corrected with high accuracy.

  In addition, since an actual image has a noise component, using the image restoration filter R created by taking the complete reciprocal of the OTF as described above greatly amplifies the noise component as the degraded image is restored. This is because the MTF is raised so that the MTF (amplitude component) of the optical system is returned to 1 over the entire frequency in a state where the amplitude of noise is added to the amplitude component of the image. The MTF, which is amplitude degradation due to the optical system, returns to 1, but at the same time, the noise power spectrum also rises. As a result, the noise is amplified according to the degree to which the MTF is raised (recovery gain).

  Therefore, when noise is included, a good image cannot be obtained as a viewing image. This can be expressed by the following equations (5-1) and (5-2).

G (u, v) = H (u, v) .F (u, v) + N (u, v) (5-1)
G (u, v) / H (u, v) = F (u, v) + N (u, v) / H (u, v)
(5-2)
N is a noise component.

  Regarding this point, for example, a method of controlling the degree of recovery according to the intensity ratio (SNR) of the image signal and the noise signal as in the Wiener filter shown in Expression (6) is known.

M (u, v) is the frequency characteristic of the Wiener filter, and | H (u, v) | is the absolute value (MTF) of the OTF.

  This method suppresses the recovery gain (recovery degree) as the MTF decreases for each frequency, and increases the recovery gain as the MTF increases. In general, since the MTF of the imaging optical system is high on the low frequency side and low on the high frequency side, it is a method of substantially suppressing the recovery gain on the high frequency side of the image.

  Subsequently, the image restoration filter will be described with reference to FIGS. The image restoration filter can determine the number of taps according to the spread of the PSF due to aberration and diffraction of the imaging optical system and the required restoration accuracy. In FIG. 5, for example, an 11 × 11 tap two-dimensional filter is used.

  Although values (coefficients) in each tap are omitted in FIG. 5, one section of the image restoration filter is shown in FIG. The distribution of the values (coefficient values) of each tap of the image restoration filter serves to return the signal value (PSF) spatially spread due to aberration and diffraction to the original one point ideally.

  Each tap of the image restoration filter is subjected to convolution processing (convolution integration, product sum) in the image restoration processing corresponding to each pixel of the image. In the convolution process, in order to improve the signal value of a predetermined pixel, the pixel coincides with the center of the image restoration filter. Then, the product of the signal value of the image and the coefficient value of the filter is taken for each corresponding pixel of the image and the image restoration filter, and the sum is replaced with the signal value of the central pixel.

  Next, characteristics of the image restoration in real space and frequency space will be described with reference to FIGS. FIG. 7 is an explanatory diagram of the PSF, where (a) shows the PSF before image recovery, and (b) shows the PSF after image recovery. FIG. 8 is an explanatory diagram of the amplitude component MTF (FIG. 8A) and the phase component PTF (FIG. 7B) of the OTF. The broken line (A) in FIG. 8A indicates the MTF before image restoration, and the alternate long and short dash line indicates the MTF after image restoration. Further, the broken line (A) in FIG. 8B indicates the PTF before image recovery, and the alternate long and short dash line indicates the PTF after image recovery. As shown in FIG. 7A, the PSF before image restoration has an asymmetric spread, and this asymmetry causes the PTF to have a non-linear value with respect to the frequency. Since the image restoration process amplifies the MTF and corrects the PTF to be zero, the PSF after the image restoration has a symmetrical and sharp shape.

  As described above, the image restoration filter can be obtained by performing inverse Fourier transform on a function designed based on the inverse function of the OTF of the imaging optical system. The image restoration filter used in this embodiment can be changed as appropriate. For example, the above-described Wiener filter can be used. When the Wiener filter is used, the real space image restoration filter that is actually convolved with the image can be created by performing inverse Fourier transform on Equation (6). In addition, since the OTF due to aberration changes in accordance with the image height (image position) of the imaging optical system even in one shooting state, the image restoration filter is changed and used in accordance with the image height. On the other hand, an OTF by diffraction whose influence becomes dominant as the aperture value increases can be treated as a uniform OTF with respect to the image height when the influence of the vignetting of the optical system is small.

  FIG. 1 is a block diagram of a camera system according to an embodiment of the present invention. The camera system includes an imaging device body and an imaging optical system 101 that is detachably attached to the imaging device body.

  The imaging optical system 101 includes a variable power lens (not shown), a diaphragm 101a, and a focus lens 101b, and forms an image of a light beam from a subject (not shown) on the imaging element 102. The diaphragm 101a adjusts the amount of light reaching the image sensor 102 by controlling the aperture diameter. The focus lens 101b has its lens position controlled by an autofocus (AF) mechanism (not shown) or a manual manual focus mechanism in order to adjust the focus according to the subject distance. For the imaging optical system 101, an optical element such as a low-pass filter or an infrared cut filter may be used. When an optical element that affects the characteristics of the OTF such as a low-pass filter is used, it may be necessary to consider at the time of creating the image restoration filter. Even in the infrared cut filter, since it affects each PSF of the RGB channel which is an integral value of the PSF of the spectral wavelength, particularly the PSF of the R channel, it may be necessary to consider at the time of creating the image restoration filter. The pixel aperture shape also affects the OTF and may be taken into consideration.

  An optical image (subject image) formed on the image sensor 102 is photoelectrically converted by the image sensor 102. An analog signal from the image sensor 102 is converted into a digital signal by the A / D converter 103 and input to the image processing unit (image processing means) 104.

  The image processing unit 104 performs image restoration processing together with predetermined processing. The display unit 105 displays an image obtained by performing predetermined processing for display on the recovered image. The display unit 105 may display an image obtained by performing simple processing for high-speed display on the recovered image.

  The state detection unit 107 sends information related to the imaging conditions of the imaging apparatus main body to the image processing unit 104. The state detection unit 107 may acquire information regarding the imaging conditions from the system controller 111, and may acquire information regarding the imaging conditions regarding the imaging optical system 101 from the imaging optical system control unit 106. The storage unit (storage means) 108 stores an OTF or a coefficient data group for generating an OTF. The image recording medium 109 stores the output image processed by the image processing unit 104 in a predetermined format. The selection unit 110 selects some OTFs from a supply unit (not shown) that stores a set of OTFs. The system controller 111 instructs the imaging optical system control unit 106 to mechanically drive the imaging optical system 101.

  Next, the case where the image processing flow of the present embodiment is applied to the camera system of FIG. 1 will be described. In this embodiment, the image restoration process is performed after a part of the OTF is selectively stored in the imaging apparatus according to the imaging apparatus used from the supply unit holding the OTF used for the image restoration process.

  First, the process until the OTF selected by the selection unit 110 is stored in the storage unit 108 will be described with reference to FIG.

  In step S11, image restoration processing is started, and in step S12, it is determined whether or not the OTF of the designated imaging optical system exists in the storage unit 108 of the imaging apparatus. If the OTF exists, the process proceeds to step S17, and if the OTF does not exist, the process proceeds to step S13. The imaging optical system may be specified by automatically determining the imaging optical system mounted on the imaging apparatus, or the user may specify the imaging optical system. Since the OTF is a complex number, it is spatially two-dimensional data composed of real part data and imaginary part data. However, the data to be stored may be a coefficient data group for reconfiguring the OTF. For example, if one of the OTFs is two-dimensional data of 11 × 11 in length and width, there are 121 pieces of data, but if this is polynomial-fitted with coefficients from 0 to 4th order in width and width, it becomes 25 pieces of data and data can be reduced. it can.

  In step S13, connection is made to a database (supply means) in which a set of OTFs is stored. The database exists in a data server capable of data communication with the imaging apparatus, and stores a set of OTFs unique to each of a plurality of types of imaging optical systems. As a data communication method, communication with the data server may be performed directly using the wireless communication function of the imaging apparatus, or communication may be performed via a computer. The database may be a storage medium that can be inserted into and removed from the imaging apparatus, or the imaging optical system 101.

  In step S14, the OTF set of the designated imaging optical system in the connected data server is specified.

  In step S15, the selection unit 110 selects a part of the OTF from the specified OTF set according to the model of the imaging apparatus. As a selection method, there is a method in which a range is selected according to shooting conditions with respect to a set of OTFs stored according to a combination of a focal length, an aperture value, and a shooting distance that can be shot by the imaging optical system. For convenience of explanation, assuming that the shooting conditions are two, that is, the focal length and the aperture value, the set of OTFs stored in the data server is as shown in FIG. A black circle is an imaging condition for which an OTF is prepared. The data of the real part and the imaginary part exists for each image height and color component of the image in one black circle. A device to be used is selected from the set of OTFs according to the model of the imaging apparatus. As an example of selection, every other data is selected in FIG. 9B except that all are selected and used. White circles indicate data that is not selected. When image restoration processing is performed for an unselected shooting condition, it can be used by interpolating from the selected shooting condition data. Further, in FIG. 9C, an OTF with an imaging condition range in which the aperture value is a predetermined value or less is selected. The reason for changing the presence or absence of the selection according to the aperture value is that the influence of aberration is dominant when the value is less than or equal to the predetermined value, and the effect of diffraction is dominant when the value is greater than or equal to the predetermined value. A method of image restoration processing for an imaging condition that is not selected will be described in a second embodiment.

  The determination of the data amount of the OTF to be selected may be specified by the user in addition to being automatically determined according to the model of the imaging apparatus. When storing OTF data of a plurality of imaging optical systems in one imaging device, the number of models is increased instead of reducing the amount of data by increasing the data amount of one imaging optical system instead of reducing the number of models of the imaging optical system to be stored. Can be increased. In this way, the user can select a trade-off between the required accuracy and the number of types of compatible optical systems for a finite storage capacity.

  In step S16, the selected OTF is stored in the storage unit 108, and the process ends in step S17.

  Next, with reference to FIG. 3, a description will be given of the process from when image recovery processing is performed on a captured image using the OTF stored in the storage unit 108 to when the recovered image is output.

  In step S21, a captured image is acquired as an input image. In particular, it is preferable to acquire a RAW image as input data for development processing.

  In step S22, shooting conditions are acquired. The shooting conditions include information for specifying the model of the lens or camera. The shooting conditions may be acquired directly from the imaging device or may be acquired from information attached to the image.

  In step S <b> 23, the image processing unit 104 acquires an OTF corresponding to the shooting condition from the storage unit 108. If there is no OTF that matches the shooting conditions, the OTF is generated by interpolation from nearby shooting conditions. As an interpolation processing method, for example, bilinear interpolation (linear interpolation), bicubic interpolation, or the like is known, but is not limited thereto.

  In step S24, the image processing unit 104 generates an image restoration filter using OTF.

  In step S25, the image processing unit 104 performs image recovery processing by convolving an image recovery filter with the captured image.

  A step S26 outputs a restored image.

  In step S15 in FIG. 2, the selection unit 110 may select a part of the data already converted into the image restoration filter, instead of selecting a part of the OTF stored in the database. In that case, processing is performed along the flow shown in FIG. In step S31, a captured image is acquired as an input image, and in step S32, a capturing condition is acquired. In step S <b> 33, the image processing unit 104 acquires an image restoration filter corresponding to the shooting condition from the storage unit 108. If the shooting conditions do not match the conditions stored in advance in the image restoration filter, interpolation generation is performed using an image restoration filter of a neighboring shooting condition. In step S34, the image processing unit 104 performs image recovery processing by convolving an image recovery filter with the captured image, and outputs a recovered image in step S35.

  In this embodiment, as shown in FIG. 9C, a case will be described in which an OTF in an imaging condition range whose aperture value is equal to or smaller than a predetermined value is selected. The reason for changing the presence or absence of the selection according to the aperture value is that the influence of aberration is dominant when the aperture value is less than a predetermined value, and the effect of diffraction is dominant when the aperture value is greater than or equal to the predetermined value.

  A case where the image processing flow of this embodiment is applied to the camera system of FIG. 1 will be described. Since the processing until the selected OTF is stored in the storage unit of the imaging apparatus is the same as that in the first embodiment, a description thereof will be omitted.

  FIG. 10 is a flowchart until the captured image is restored and output using the stored OTF.

  In step S41, the captured image is acquired as an input image, and in step S42, the imaging condition is acquired.

  In step S43, it is determined whether or not the aperture value is equal to or less than a predetermined value. If the aperture value is equal to or smaller than the predetermined value, it is determined that the influence of aberration is dominant, and an output image is generated in steps S44 to S47. Since the processing in steps S44 to S47 is the same as the processing in steps S23 to S26 described in the first embodiment, the description thereof is omitted. If it is greater than or equal to the predetermined value, it is determined that the influence of diffraction is dominant, and the process proceeds to step S48. The predetermined value can be appropriately changed according to the aberration amount of the imaging optical system to be used and the pixel pitch (pixel size) of the imaging element. Preferably, an aperture value in which the cutoff frequency due to diffraction determined by the aperture value is smaller than the sampling frequency determined by the pixel pitch of the image sensor is set as a predetermined value. This can be expressed by the following equation (7).

F ≧ p / λ (7)
F is the F number, p is the pixel pitch, and λ is the wavelength.

  For example, if the pixel pitch p is 0.006 mm, the sampling frequency is 167 lines / mm. Therefore, if the wavelength λ is 550 nm, the branching occurs depending on whether it is F11 or more.

  In step S48, an image restoration filter corresponding to the shooting condition is acquired. The cause of image degradation is greatly affected by diffraction and does not depend on the shooting conditions of the shooting distance and the focal length. Therefore, the image restoration filter may be selected only in accordance with the aperture value. The image restoration filter may be stored in advance in the storage unit of the imaging apparatus or may be acquired from a database. Then, the restored image is output through steps S46 and S47.

  Therefore, the data stored in the storage unit 108 is the coefficient data for generating the OTF or OTF when the aperture value is less than or equal to the predetermined value, and the image restoration filter when the aperture value is greater than or equal to the predetermined value. As a result, data relating to OTFs with an aperture value equal to or greater than a predetermined value is reduced, and generation of an image restoration filter for each image height becomes unnecessary. Further, the same image restoration filter can be used for the entire image without switching the image restoration filter in accordance with the image height in step S46.

  On the other hand, in the case of an imaging apparatus that performs high-accuracy image restoration processing even if the amount of data is large, an OTF that takes account of the influence of aberration is stored in the storage unit even under imaging conditions that are greatly affected by diffraction, and a restored image is generated. Can do. In this way, by changing the amount of OTF data used according to the specifications of the imaging apparatus, the OTF set of the imaging optical system can be handled in common.

  Here, the reason why the image restoration filter is generated from the OTF when the aperture value is equal to or smaller than the predetermined value and the image restoration filter is directly used when the aperture value is equal to or larger than the predetermined value will be described.

  When the aperture value is equal to or smaller than the predetermined value, the aberration becomes dominant. Therefore, it is necessary to change and use the image restoration filter according to the aberration for each image height in the image. In this case, using the rotational symmetry of the imaging optical system, an OTF can be prepared for each image height in a one-dimensional direction, and an OTF at each position of the image can be generated by rotating around the optical axis. Since the characteristic change of the OTF is not relatively steep as shown in FIG. 8, the OTF can be rearranged with little deterioration even if the rotation and the interpolation processing associated therewith are performed. Therefore, the OTF of the entire screen can be reproduced from a small amount of data in the one-dimensional direction. On the other hand, since the characteristics of the image restoration filter are steep as shown in FIG. 6, when the rotation and the interpolation processing associated therewith are performed, the characteristics change greatly. Therefore, it becomes necessary to hold an image restoration filter for the entire screen in advance. Further, not only in-screen interpolation but also when interpolating between shooting conditions, the amount of data can be reduced by treating OTF as initial data due to the same problem.

  When the aperture value is greater than or equal to a predetermined value, diffraction is dominant and a uniform image restoration filter can be used in the image. Therefore, when the aperture value is equal to or larger than the predetermined value, it is not necessary to convert the OTF into the image restoration filter as the initial data, and the processing load can be reduced by using the image restoration filter prepared in advance.

  In the present embodiment, a case where the supply unit side includes a selection unit will be described with reference to FIG. FIG. 11 is a block diagram of an image processing system according to the embodiment of the present invention.

  FIG. 11A shows an image processing system when the data server 306 has a selection unit 308. The imaging apparatus 301 includes a storage unit 302 that stores an OTF and an image processing unit 303 that performs image restoration processing. The imaging optical system 304 is detachably attached to the imaging device 301. The imaging device 301 is connected to a data server 306 via a computer 305 so that data communication is possible. Note that the communication unit (not shown) of the image pickup apparatus 301 may be connected to a state in which data communication can be performed directly with the data server 306 without using the computer 305 by wireless communication or the like. The data server 306 includes a database 307 that stores a set of OTFs of a plurality of types of imaging optical systems, and a selection unit 308 that selects a part from the set of OTFs according to the imaging apparatus. When the OTF is selected from the database 307 and stored in the storage unit 302, the imaging optical system 304 and the model information of the imaging device 301 are transmitted from the imaging device 301 side, and the selection unit 308 on the data server 306 side is suitable for this. Select OTF.

  FIG. 11B shows an image processing system when the computer 305 has the selection unit 308. When the OTF is selected from the database 307 and stored in the storage unit 302, the model information of the imaging optical system 304 and the imaging device 301 is transmitted from the imaging device 301 side. Then, the selection unit 308 of the computer 305 selects an OTF suitable for the transmitted information, acquires the OTF from the data server 306, and transmits it to the imaging apparatus 301. Further, the user may specify the amount of OTF data from the free capacity of the storage unit 302 of the imaging device 301 on the computer. In this way, the user can select a trade-off between the required accuracy and the number of types of compatible optical systems for a finite storage capacity.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

101 Imaging optical system 104 Image processing unit (image processing means)
108 Storage unit (storage means)
110 Selection means

Claims (11)

A selection means for selecting a part of the optical transfer function from a supply means for storing a set of optical transfer functions specific to the imaging optical system;
Storage means for storing the partial optical transfer function;
Image processing means for generating an image restoration filter using an optical transfer function corresponding to a photographing condition of the imaging optical system among the partial optical transfer functions, and performing image restoration processing using the image restoration filter; An image processing apparatus comprising: The set of optical transfer functions is stored according to a combination of a focal length that can be photographed by the imaging optical system, an aperture value, and a photographing distance,
The image processing apparatus according to claim 1, wherein the selection unit selects the partial optical transfer function based on at least one of a focal length, an aperture value, and an imaging distance.   The image processing apparatus according to claim 2, wherein the selection unit selects an optical transfer function in which the aperture value of the imaging optical system is a predetermined value or less from the set of optical transfer functions.   4. The image processing device according to claim 1, wherein the image processing unit performs image recovery processing using the generated image recovery filter when an aperture value is equal to or less than the predetermined value among the photographing conditions. 5. The image processing apparatus according to item.   5. The image processing apparatus according to claim 1, wherein the partial optical transfer function is a coefficient data group for generating an optical transfer function used by the image processing unit. . Selection means for selecting some image restoration filters from supply means for storing a set of image restoration filters unique to the imaging optical system;
Storage means for storing the partial image restoration filter;
An image processing apparatus comprising: an image processing unit that performs an image restoration process using an image restoration filter corresponding to a photographing condition of the imaging optical system among the partial image restoration filters. An image sensor that photoelectrically converts a subject image formed by the imaging optical system to generate a captured image; and
A selection means for selecting a part of the optical transfer function from a supply means for storing a set of optical transfer functions specific to the imaging optical system;
Storage means for storing the partial optical transfer function;
Image processing for generating an image restoration filter using an optical transfer function corresponding to a photographing condition of the imaging optical system among the partial optical transfer functions, and performing image restoration processing of the photographed image using the image restoration filter And an imaging device. An imaging optical system;
An imaging device that photoelectrically converts a subject image formed by the imaging optical system to generate a captured image;
A selection means for selecting a part of the optical transfer function from a supply means for storing a set of optical transfer functions specific to the imaging optical system;
Storage means for storing the partial optical transfer function;
Image processing for generating an image restoration filter using an optical transfer function corresponding to a photographing condition of the imaging optical system among the partial optical transfer functions, and performing image restoration processing of the photographed image using the image restoration filter And a camera system. Supply means for storing a set of optical transfer functions specific to the imaging optics;
Selection means for selecting a part of the optical transfer function from the supply means;
Storage means for storing the partial optical transfer function;
An image processing unit that generates an image restoration filter using an optical transfer function corresponding to a photographing condition of the imaging optical system among the partial optical transfer functions, and performs image restoration processing using the image restoration filter; An image processing system comprising:   The image processing system according to claim 9, wherein the supply unit includes the selection unit.   The image processing system according to claim 9, wherein the supply unit is any one of a data server, a storage medium, an imaging device, and the imaging optical system.