JP5611439B2 - Image processing method, image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing method for performing image restoration processing of a captured image.

  A captured image obtained by the imaging apparatus includes a blur component and is deteriorated due to the influence of each aberration such as spherical aberration, coma aberration, field curvature, astigmatism, and the like of the imaging optical system. The blur component of the image due to such aberrations means that when there is no aberration and there is no influence of diffraction, the light beam emitted from one point of the subject spreads out on the imaging surface and should be collected again at one point. It is expressed by a point spread function PSF (Point Spread Function).

  An optical transfer function OTF (Optical Transfer Function) obtained by Fourier transforming the point spread function PSF is frequency component information of aberration and is represented by a complex number. The absolute value of the optical transfer function OTF, that is, the amplitude component is referred to as MTF (Modulation Transfer Function), and the phase component is referred to as PTF (Phase Transfer Function). The amplitude component MTF and the phase component PTF are the frequency characteristics of the amplitude component and the phase component, respectively, of image degradation due to aberration, and are expressed by the following equations with the phase component as the phase angle.

PTF = tan ⁻¹ (Im (OTF) / Re (OTF))
Here, Re (OTF) and Im (OTF) represent the real part and the imaginary part of the optical transfer function OTF, respectively. As described above, since the optical transfer function OTF of the imaging optical system deteriorates the amplitude component MTF and the phase component PTF of the image, the deteriorated image is in a state where each point of the subject is asymmetrically blurred like coma. Yes. Further, the chromatic aberration of magnification is generated by shifting the imaging position due to the difference in imaging magnification for each wavelength of light, and acquiring this as, for example, RGB color components according to the spectral characteristics of the imaging device. Accordingly, the image formation position is shifted between RGB, and the image formation position shift for each wavelength, that is, the spread of the image due to the phase shift also occurs in each color component.

  As a method for correcting the deterioration of the amplitude component MTF and the phase component PTF, a method of correcting using the information of the optical transfer function OTF of the imaging optical system is known. This method is called “image restoration” or “image restoration”. Hereinafter, processing for correcting deterioration of a captured image using information of an optical transfer function (OTF) of the imaging optical system is referred to as image restoration processing. Although details will be described later, as one of image restoration methods, there is known a method that convolves an input image with an image restoration filter having an inverse characteristic of an optical transfer function (OTF).

  In order to effectively use image restoration, it is necessary to obtain more accurate OTF information of the imaging optical system. The OTF of a general imaging optical system varies greatly depending on the image height (image position). The optical transfer function OTF is two-dimensional data and is a complex number, and thus has a real part and an imaginary part. In addition, when performing image restoration processing for a color image having three color components of RGB, OTF data of one image height includes the number of vertical taps × the number of horizontal taps × 2 (real part, imaginary part). Part) × 3 (RGB). Here, the number of taps is the vertical and horizontal size of the OTF data. If these are held for all shooting conditions such as image height, F number (aperture value), zoom (focal length), and shooting distance, a huge amount of data is required.

  Patent Document 1 discloses a technique for performing image processing while retaining a filter coefficient for correcting deterioration of a captured image. However, since a recovery filter corresponding to the position in the screen is required, the amount of data becomes enormous. In Patent Document 2, although it is not an image restoration technique, the distance from the optical center is obtained in order to correct lateral chromatic aberration, and this is substituted into a cubic function to calculate the correction movement amounts of the R and B components. Thus, a method for determining a correction amount according to the position in the screen is disclosed.

JP 2010-56992 A JP 2004-241991 A

  However, since the image restoration filter is two-dimensional data, the image restoration filter corresponding to the position of the image is not determined only by the distance from the optical center (the center of the captured image or the optical axis of the imaging optical system). I need. Therefore, the method of Patent Document 2 cannot be applied to image restoration processing. Further, since the coefficient of the image restoration filter varies finely between taps, if the rotation according to the position of the image is performed, the value of the filter coefficient is greatly collapsed, and the effect of the image restoration process cannot be obtained.

  Therefore, the present invention provides an image processing method, an image processing apparatus, an imaging apparatus, and an image processing program capable of executing a highly accurate image restoration process while reducing the amount of information necessary for the image restoration process.

  An image processing method according to one aspect of the present invention is an image processing method for performing image restoration processing of a captured image, and uses coefficient data corresponding to a shooting condition of the captured image according to a position of the captured image. Generating a first optical transfer function of a plurality of imaging optical systems; and rotating the first optical transfer function around the center of the captured image or the optical axis of the imaging optical system to generate a plurality of second optical transfer functions. Generating an optical transfer function; generating an image restoration filter based on the first optical transfer function and the second optical transfer function; and image restoration processing of the captured image using the image restoration filter The process of performing.

  An image processing apparatus according to another aspect of the present invention is an image processing apparatus that performs image restoration processing of a captured image, and uses coefficient data corresponding to a capturing condition of the captured image according to a position of the captured image. First optical transfer function generating means for generating a first optical transfer function of a plurality of imaging optical systems, and the first optical transfer function around the center of the captured image or the optical axis of the imaging optical system Second optical transfer function generating means for generating a plurality of second optical transfer functions by rotating, and image recovery for generating an image recovery filter based on the first optical transfer function and the second optical transfer function A filter generation unit; and an image restoration unit that performs an image restoration process on the captured image using the image restoration filter.

  An imaging apparatus according to another aspect of the present invention is an imaging apparatus that performs image restoration processing of a captured image, and uses a plurality of coefficient data according to the shooting conditions of the captured image, and a plurality of images according to the position of the captured image. First optical transfer function generating means for generating a first optical transfer function of the imaging optical system, and rotating the first optical transfer function around the center of the captured image or the optical axis of the imaging optical system Second optical transfer function generating means for generating a plurality of second optical transfer functions, and image recovery filter generation for generating an image recovery filter based on the first optical transfer function and the second optical transfer function And image recovery means for performing image recovery processing of the captured image using the image recovery filter.

  An image processing program according to another aspect of the present invention is an image processing program for performing image restoration processing of a captured image, and uses coefficient data corresponding to the capturing condition of the captured image according to the position of the captured image. Generating a first optical transfer function of a plurality of imaging optical systems; and rotating the first optical transfer function around the center of the captured image or the optical axis of the imaging optical system to generate a plurality of second optical transfer functions. Generating an optical transfer function of the image, generating an image restoration filter based on the first optical transfer function and the second optical transfer function, and image restoration of the captured image using the image restoration filter And causing the information processing device to execute the process.

  Other objects and features of the present invention are illustrated in the following examples.

  According to the present invention, it is possible to provide an image processing method, an image processing apparatus, an imaging apparatus, and an image processing program capable of executing a highly accurate image recovery process while reducing the amount of information necessary for the image recovery process. it can.

3 is a flowchart of an image processing method in Embodiment 1. It is explanatory drawing of the image restoration filter in a present Example. It is explanatory drawing of the image restoration filter in a present Example. It is explanatory drawing of the point spread function PSF in a present Example. It is explanatory drawing of the amplitude component MTF and phase component PTF of the optical transfer function in a present Example. 6 is a diagram illustrating a method for generating an image restoration filter in Embodiment 1. FIG. 1 is an explanatory diagram of an image processing system in Embodiment 1. FIG. It is explanatory drawing of the coefficient calculation apparatus in Example 1. FIG. It is explanatory drawing of the coefficient data in Example 1. FIG. It is explanatory drawing of the tap number and frequency pitch in Example 1. FIG. It is explanatory drawing of the tap number and frequency pitch in Example 1. FIG. It is explanatory drawing of the OTF reconstruction part in Example 1. FIG. It is explanatory drawing of the reconfiguration | reconstruction OTF in Example 1. FIG. 6 is a flowchart of another image processing method according to the first exemplary embodiment. FIG. 6 is a configuration diagram of an imaging apparatus in Embodiment 2. FIG. 10 is a configuration diagram of an image processing system in Embodiment 3. It is explanatory drawing of the information set in Example 3. FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

First, definitions of terms and image restoration processing (image processing method) described in this embodiment will be described. The image processing method described here is appropriately used in each embodiment described later.
[Input image]
The input image is a digital image (photographed image) obtained by receiving light with an image pickup device via an image pickup optical system, and is deteriorated by an optical transfer function OTF due to aberration of the image pickup optical system including a lens and various optical filters. doing. The imaging optical system can be configured using not only a lens but also a mirror (reflection surface) having a curvature.

  The color component of the input image has information on RGB color components, for example. As the color component, other commonly used color spaces such as brightness, hue, and saturation expressed in LCH, luminance expressed in YCbCr, and color difference signals can be selected and used. As other color spaces, XYZ, Lab, Yuv, and JCh can be used. Further, a color temperature may be used.

The input image and the output image can be accompanied by shooting conditions such as the focal length of the lens, aperture value, shooting distance, and various correction information for correcting this image. When the image is transferred from the imaging apparatus to another image processing apparatus and correction processing is performed, it is preferable to add information regarding shooting conditions and correction to the captured image as described above. As another delivery method of information regarding imaging conditions and correction, the imaging apparatus and the image processing apparatus may be directly or indirectly connected to be delivered.
[Image recovery processing]
Next, an outline of the image restoration process will be described. When the captured image (degraded image) is g (x, y), the original image is f (x, y), and the point spread function PSF that is a Fourier pair of the optical transfer function OTF is h (x, y). The following formula (1) is established.

g (x, y) = h (x, y) * f (x, y) (1)
Here, * is convolution (convolution integration, sum of products), and (x, y) are coordinates on the captured image.

  Moreover, if Formula (1) is Fourier-transformed and converted into a display format on the frequency plane, Formula (2) represented by the product for each frequency is obtained.

G (u, v) = H (u, v) .F (u, v) (2)
Here, H is an optical transfer function OTF obtained by Fourier transforming the point spread function PSF (h), and G and F are obtained by Fourier transforming the degraded image g and the original image f, respectively. Function. (U, v) is a coordinate on a two-dimensional frequency plane, that is, a frequency.

  In order to obtain the original image f from the photographed degraded image g, both sides may be divided by the optical transfer function H as shown in the following equation (3).

G (u, v) / H (u, v) = F (u, v) (3)
Then, F (u, v), that is, G (u, v) / H (u, v) is subjected to inverse Fourier transform to return to the actual surface, whereby the original image f (x, y) is obtained as a restored image. can get.

When R is the result of inverse Fourier transform of H− ¹ , the original image f (x, y) is similarly obtained by performing convolution processing on the actual image as in the following equation (4). Can be obtained.

g (x, y) * R (x, y) = f (x, y) (4)
Here, R (x, y) is called an image restoration filter. When the image is a two-dimensional image, generally, the image restoration filter R is also a two-dimensional filter having taps (cells) corresponding to each pixel of the image. In general, the greater the number of taps (number of cells) of the image recovery filter R, the higher the recovery accuracy. Therefore, the number of taps that can be realized is set according to the required image quality, image processing capability, aberration characteristics, and the like. Since the image restoration filter R needs to reflect at least aberration characteristics, it is different from a conventional edge enhancement filter (high-pass filter) of about 3 taps each in horizontal and vertical directions. Since the image restoration filter R is set based on the optical transfer function OTF, it is possible to correct both the amplitude component and the deterioration of the phase component with high accuracy.

  In addition, since an actual image contains a noise component, using the image restoration filter R created by taking the complete inverse of the optical transfer function OTF as described above, the noise component is greatly amplified along with the restoration of the deteriorated image. Will be. This is because the MTF is raised so that the MTF (amplitude component) of the optical system is returned to 1 over the entire frequency in a state where the amplitude of noise is added to the amplitude component of the image. The MTF, which is amplitude degradation due to the optical system, returns to 1, but at the same time, the noise power spectrum also rises. As a result, the noise is amplified according to the degree to which the MTF is raised (recovery gain).

  Therefore, when noise is included, a good image cannot be obtained as a viewing image. This is expressed by the following formulas (5-1) and (5-2).

G (u, v) = H (u, v) .F (u, v) + N (u, v) (5-1)
G (u, v) / H (u, v) = F (u, v) + N (u, v) / H (u, v) (5-2)
Here, N is a noise component.

  For an image including a noise component, there is a method of controlling the degree of recovery according to the intensity ratio SNR between the image signal and the noise signal, for example, as in the Wiener filter expressed by the following formula (6).

Here, M (u, v) is the frequency characteristic of the Wiener filter, and | H (u, v) | is the absolute value (MTF) of the optical transfer function OTF. In this method, for each frequency, the smaller the MTF, the smaller the recovery gain (degree of recovery), and the larger the MTF, the larger the recovery gain. In general, since the MTF of the imaging optical system is high on the low frequency side and low on the high frequency side, this method substantially reduces the recovery gain on the high frequency side of the image.

  Next, the image restoration filter will be described with reference to FIGS. The number of taps of the image restoration filter is determined according to the aberration characteristics of the imaging optical system and the required restoration accuracy. The image restoration filter in FIG. 2 is an 11 × 11 tap two-dimensional filter as an example. In FIG. 2, values (coefficients) in each tap are omitted. FIG. 3 shows a cross section of this image restoration filter. The distribution of the values (coefficient values) of each tap of the image restoration filter has a function of returning the signal value (PSF) spatially spread due to aberration to the original one point ideally.

  Each tap of the image restoration filter is subjected to convolution processing (convolution integration, product sum) in the image restoration processing corresponding to each pixel of the image. In the convolution process, in order to improve the signal value of a predetermined pixel, the pixel is matched with the center of the image restoration filter. Then, the product of the signal value of the image and the coefficient value of the filter is taken for each corresponding pixel of the image and the image restoration filter, and the sum is replaced with the signal value of the central pixel.

  Next, with reference to FIGS. 4 and 5, the characteristics of the image restoration in the real space and the frequency space will be described. 4A and 4B are explanatory diagrams of the point spread function PSF. FIG. 4A shows the point spread function PSF before image restoration, and FIG. 4B shows the point spread function PSF after image restoration. . FIG. 5 is an explanatory diagram of the amplitude component MTF (FIG. 5A) and the phase component PTF (FIG. 5B) of the optical transfer function OTF. In FIG. 5A, the broken line (A) indicates the MTF before image recovery, and the alternate long and short dash line (B) indicates the MTF after image recovery. In FIG. 5B, the broken line (A) indicates the PTF before image recovery, and the alternate long and short dash line (B) indicates the PTF after image recovery. As shown in FIG. 4A, the point spread function PSF before image restoration has an asymmetric spread, and the phase component PTF has a non-linear value with respect to the frequency due to this asymmetry. Since the image restoration process amplifies the amplitude component MTF and corrects the phase component PTF to be zero, the point spread function PSF after the image restoration has a symmetrical and sharp shape.

  As described above, the image restoration filter can be obtained by performing inverse Fourier transform on a function designed based on the inverse function of the optical transfer function OTF of the imaging optical system. The image restoration filter used in this embodiment can be appropriately changed. For example, the above-described Wiener filter can be used. When using the Wiener filter, it is possible to create an image restoration filter in real space that is actually convolved with the image by performing inverse Fourier transform on Equation (6). In addition, the optical transfer function OTF changes according to the image height (image position) of the imaging optical system even in one shooting state. For this reason, the image restoration filter is changed and used according to the image height.

  Next, an image processing method according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a flowchart of an image processing method (image processing program) in this embodiment. The flowchart in FIG. 1 is executed based on a command from an image processing apparatus described later.

  First, in step S11, a captured image is acquired as an input image. The captured image can be acquired by connecting the imaging device and the image processing device by wire or wirelessly. The captured image can also be acquired via a storage medium. Subsequently, in step S12, photographing conditions are acquired. The shooting conditions are a focal length, an aperture value, a shooting distance, and the like. Further, in the case of an imaging apparatus in which a lens is replaceably attached to the camera body, the imaging conditions further include a lens ID and a camera ID. Information regarding the imaging conditions can be obtained directly from the imaging apparatus. This information can also be acquired from information attached to the image.

  Next, in step S13, coefficient data suitable for the shooting conditions is acquired. The coefficient data is data for reconstructing the optical transfer function OTF. For example, desired coefficient data is selected from a plurality of coefficient data held in advance according to the imaging conditions. In addition, when the shooting conditions such as the aperture, shooting distance, and focal length of the zoom lens are specific shooting conditions, the coefficient data corresponding to the shooting conditions is interpolated from the coefficient data of other shooting conditions held in advance. It can also be generated by processing. In this case, it is possible to reduce the amount of data stored in the image restoration filter. As the interpolation processing, for example, bilinear interpolation (linear interpolation), bicubic interpolation, or the like is used, but is not limited thereto. In step S14, the optical transfer function OTF is reconstructed from the coefficient data. That is, the first optical transfer function of the plurality of imaging optical systems corresponding to the position of the captured image is generated using coefficient data corresponding to the captured condition of the captured image. The first optical transfer function is generated by first optical transfer function generation means. Note that the imaging optical system may include an imaging element, an optical low-pass filter, and the like. Details of the reconstruction of the optical transfer function OTF will be described later.

  In step S15, the reconstructed optical transfer function OTF is rotated around the center of the screen (the center of the captured image) or the optical axis of the imaging optical system to develop the optical transfer function OTF (second optical transfer function). Generating means). Specifically, by interpolating the optical transfer function OTF corresponding to the pixel arrangement, the optical transfer function OTF is discretely arranged at a plurality of positions in the captured image. In step S16, the optical transfer function OTF is converted into an image restoration filter, that is, an image restoration filter is generated using the developed optical transfer function OTF (image restoration filter generation means). The image restoration filter is generated by creating a restoration filter characteristic in the frequency space based on the optical transfer function OTF and converting it into a real space filter (image restoration filter) by inverse Fourier transform.

  Here, with reference to FIG. 6 (a)-(e), step S15, S16 is explained in full detail. 6A to 6E are diagrams illustrating a method for generating an image restoration filter. As shown in FIG. 6A, the reconstructed optical transfer function OTF is a circumscribed circle of the image in one direction (vertical direction) passing through the center of the screen (center of the captured image) or the optical axis of the imaging optical system. Are arranged in the area (imaging area).

  In the present embodiment, as shown in FIG. 6A, the optical transfer function is developed on a straight line in step S14, but the present invention is not limited to this. For example, in the captured image plane, straight lines that pass through the center of the captured image and are orthogonal to each other are defined as a first straight line (y in FIG. 6A) and a second straight line (x in FIG. 6A). At this time, at least two of the optical transfer functions generated in step S14 may be optical transfer functions corresponding to the position (image height) on the straight line of the first straight line. That is, the reconstructed optical transfer function OTF is arranged at a plurality of positions (a plurality of positions in the captured image) arranged at different distances in the predetermined direction from the center of the screen or the optical axis of the imaging optical system. For example, it may not be arranged linearly in one direction. When there is no pixel including the center of the captured image, that is, when the center of the captured image is between the pixels, the optical transfer function generated in step S14 is the position of the pixel across the first straight line ( Any optical transfer function corresponding to (image height) may be used.

  Further, when the optical transfer functions OTF are arranged in one direction, the optical transfer function OTF is not limited to the vertical direction but may be arranged in another direction such as a horizontal direction. It is more preferable to linearly arrange the optical transfer functions OTF in either the vertical direction or the horizontal direction because the image processing of this embodiment can be performed more easily.

  Subsequently, the reconstructed optical transfer function OTF is rotated, and interpolation processing (various processes corresponding to the rotated pixel array) is performed as necessary. As shown in FIG. 6B, the optical transfer function OTF is performed. Rearrange. The interpolation process includes an interpolation process in the radial direction and an interpolation process accompanying rotation, and the optical transfer function OTF can be rearranged at an arbitrary position. Next, with respect to the optical transfer function OTF at each position, the frequency characteristics of the image restoration filter are calculated as shown in, for example, Expression (6), and inverse Fourier transform is performed, so that the real space is obtained as shown in FIG. Is converted to an image restoration filter.

  The imaging optical system of the present embodiment is a rotationally symmetric optical system. Therefore, by utilizing the symmetry, the optical transfer function OTF can be inverted as shown in FIG. 6D and developed over the entire screen (the entire domain of the optical transfer function). That is, in the captured image, straight lines that pass through the center of the captured image and are orthogonal to each other are defined as a first straight line (y in FIG. 6A) and a second straight line (x in FIG. 6A). The second region (61 in FIG. 6C) is a point-symmetric region with respect to the first region (63 in FIG. 6C) of the captured image with respect to the center of the captured image or the optical axis of the imaging optical system. A region that is line-symmetric with respect to the first region and the first straight line is defined as a third region (62 in FIG. 6C). At this time, the optical transfer function of the second region or the third region is generated using the optical transfer function of the first region. As a result, the Fourier transform process is reduced to approximately ¼ of the position where the rearrangement is finally performed. Further, the optical transfer function OTF in FIG. 6B and the image restoration filter in FIG. 6C are rearranged by rotation and interpolation processing as shown in FIG. If development is performed as shown in (d), the Fourier transform processing can be further reduced. Note that the arrangements (recovery filter arrangement densities) shown in FIGS. 6A to 6E are merely examples, and the arrangement interval can be arbitrarily set according to fluctuations in the optical transfer function OTF of the imaging optical system. .

  Next, in step S17 in FIG. 1, image recovery processing of the captured image is performed using the image recovery filter generated in step S16 (image recovery means). In other words, the image restoration process of the photographed image is performed by convolving the image restoration filter with the photographed image. In step S18, a recovered image is acquired from the result of the image recovery process in step S17.

  It should be noted that during convolution of the image restoration filter, pixels other than the position where the image restoration filter of FIG. 6D is arranged can be interpolated using a plurality of filters arranged in the vicinity. At this time, the image restoration filter includes a first image restoration filter at the first position of the photographed image and a second image restoration filter at the second position of the photographed image. The first image restoration filter is generated using the developed optical transfer function. The second image restoration filter is generated by interpolating using the first image restoration filter. By performing such interpolation processing, for example, it is possible to change the image restoration filter for each pixel.

  Next, an image processing system including an image processing apparatus that performs the above-described image processing method will be described with reference to FIG. FIG. 7 is an explanatory diagram of the image processing system 100 in the present embodiment. The image processing system 100 includes a coefficient calculation device 101, a camera 110 (imaging device), and an image restoration processing device 120 (image processing device).

  The coefficient calculation device 101 calculates coefficient data for reconstructing the optical transfer function OTF according to the shooting condition of the shot image. The coefficient calculation apparatus 101 performs a process of calculating the optical transfer function OTF from the design value or measurement value of the imaging optical system. The coefficient calculation device 101 converts the optical transfer function OTF into a coefficient, and determines the order of the coefficient used for the reconstruction of the optical transfer function OTF later according to the required accuracy. The coefficient calculation apparatus 101 determines the number of taps for each image height necessary for reproducing the optical transfer function OTF from the size of the spatial distribution of the point spread function PSF. The coefficient calculation device 101 calculates and outputs coefficient data up to a required order and information regarding the number of taps for combinations of various imaging lenses and camera imaging elements. The camera 110 has an image sensor 111 and an imaging lens 112. The camera 110 outputs the image captured by the imaging lens 112 with the lens ID of the imaging lens 112 and information on the imaging conditions (aperture, zoom, imaging distance, etc.) and the Nyquist frequency of the imaging element 111 added.

  The image restoration processing device 120 (image processing device) includes an image restoration information holding unit 121, an OTF reconstruction unit 122, and a filter processing unit 123 (generation unit). The image restoration processing device 120 holds information output from the coefficient calculation device 101 and the camera 110, and corrects a deteriorated image photographed by the imaging lens 112 using the information (performs image restoration processing of the photographed image). ).

  The image restoration information holding unit 121 includes coefficient data, the number of taps, a lens ID, shooting conditions, and the Nyquist frequency of the image sensor for each of the various combinations of the image pickup lens 112 and the image sensor 111 calculated by the coefficient calculator 101. Store information. As described above, the image restoration information holding unit 121 is a storage unit that stores coefficient data corresponding to the shooting conditions of a shot image.

  The OTF reconstruction unit 122 acquires Nyquist frequency information and a captured image of the image sensor of the image sensor 111 from the camera 110, and acquires information regarding the lens ID of the image pickup lens 112 and the shooting conditions. The OTF reconstruction unit 122 searches the coefficient data and the number of taps stored in the image recovery information holding unit 121 from the lens ID of the camera 110 used when the photographer took a picture and the shooting conditions, and responds accordingly. Get coefficient data and number of taps. The OTF reconstruction unit 122 reconstructs the optical transfer function OTF used by the filter processing unit 123 in the spatial frequency region up to the Nyquist frequency of the image sensor of the camera 110. That is, using the acquired coefficient data and the number of taps, the optical transfer function OTF of the imaging optical system (imaging lens 112) corresponding to the position of the captured image is reconstructed. As described above, the OTF reconstruction unit 122 is a reconstruction unit that reconstructs the optical transfer function OTF of the imaging optical system according to the position of the captured image using the coefficient data. The OTF reconstruction unit 122 is a developing unit that expands the optical transfer function OTF by rotating the optical transfer function OTF around the center of the captured image or the optical axis of the imaging optical system. Hereinafter, the optical transfer function OTF reconstructed by the OTF reconstruction unit 122 is referred to as a reconstructed OTF.

  The filter processing unit 123 uses the reconstruction OTF created by the OTF reconstruction unit 122 to create an image restoration filter that corrects the degradation of the captured image, and corrects the degradation of the image. That is, the filter processing unit 123 is a generation unit that generates an image restoration filter using the developed optical transfer function OTF. The filter processing unit 123 is a processing unit that performs image restoration processing on a captured image using an image restoration filter. Here, if the coefficient data and the number of taps calculated in advance by the coefficient calculation device 101 are held in the image restoration information holding unit 121, it is not necessary to provide the coefficient calculation device 101 to the user (photographer). The user can also download and use information necessary for image restoration processing such as coefficient data through a network or various storage media.

  Next, a coefficient calculation method in the coefficient calculation apparatus 101 will be described in detail. In this embodiment, the coefficient calculation apparatus 101 generates coefficient data by fitting a design value or a measured value of the optical transfer function OTF of the imaging optical system (imaging lens 112) with a predetermined function. That is, the coefficient calculation apparatus 101 generates coefficient data by approximating the optical transfer function OTF of the imaging optical system by fitting to a predetermined function. In the present embodiment, fitting is performed using a Legendre polynomial. However, the present embodiment is not limited to this, and fitting can be performed using other functions such as a Chebuschev polynomial. The Legendre polynomial is expressed by Equation (7).

Here, [x] is a maximum integer not exceeding x.

Further, since OTF is expressed in the form of z = f (x, y), it is necessary to calculate the coefficient a _ij in Expression (8).

Equation (8) is an orthogonal function, and the value of a _ij is determined without depending on the order at the time of fitting. By utilizing the property of the orthogonal function of Expression (8), when the fitting of the optical transfer function OTF can be performed with sufficiently high accuracy even at a low order, the amount of information related to coefficient data to be held in the apparatus is reduced. be able to.

  Next, the coefficient calculation apparatus 101 will be described in detail with reference to FIG. FIG. 8 is an explanatory diagram of the coefficient calculation apparatus 101 and shows a specific method for fitting the optical transfer function OTF using the equations (7) and (8). Fum and fvm shown in FIG. 8 are Nyquist frequencies in the meridional and sagittal directions of the optical transfer function OTF, respectively. Nx and Ny are odd tap numbers in the meridional and sagittal directions of the optical transfer function OTF, respectively. The coefficient calculation apparatus 101 calculates coefficient data by performing the above-described fitting on each of the real part and the imaginary part of the optical transfer function OTF.

  The real part of the optical transfer function OTF is symmetric in the meridional direction and the sagittal direction. The imaginary part of the optical transfer function OTF is symmetric in the meridional direction but is symmetric, and is symmetric in the sagittal direction. When such symmetry is used, it is necessary and sufficient if there is information on a quarter region (a part of the defined region) of the entire defined region as data of the optical transfer function OTF to be fitted. That is, the optical transfer function OTF can be developed over the entire domain using the symmetry with respect to the center of the screen or the optical axis of the imaging optical system. For the above reason, in this embodiment, in order to fit the optical transfer function OTF with high accuracy, a quarter of the entire domain is cut out from the optical transfer function OTF so that the DC component is included in both the real part and the imaginary part. Perform fitting. In this embodiment, the case where the data of the optical transfer function OTF is Nx (row) × Ny (column) tap has been described. From this data, 1 to [Nx / 2] +1 row, 1 to [Ny / 2]. Although +1 column data is cut out, the present invention is not limited to this.

  FIG. 9 is an explanatory diagram of coefficient data calculated by the above-described method. FIG. 9 shows an example in which the real part and imaginary part coefficients of the optical transfer function OTF are calculated up to the 10th order for each image height. One coefficient data (coefficient information) is completed by adding the lens ID, aperture, zoom, and subject distance information to the coefficient group for each image height. In the present embodiment, as an example, coefficient information for 10 image heights in imaging conditions of lens ID: No123, aperture: F2.8, zoom: WIDE, and imaging distance: close is shown. The coefficient information for the 10 image heights is used to reconstruct the 10 optical transfer functions OTF in FIG. In addition, the coefficient data created as described above may be further converted into a function between image heights for each order. The coefficient calculation apparatus 101 creates this information for all combinations of lens ID, aperture, zoom, and shooting distance information, and outputs the information to the image restoration processing apparatus 120.

  Next, a method for determining the number of taps of the reconstructed OTF will be described in detail. When filtering an image, the processing time greatly depends on the number of taps of the filter. For this reason, it is preferable that the number of taps of the filter is small as long as a desired image restoration effect can be obtained when the filter processing is performed and no adverse effects such as ringing occur.

  The image restoration filter used in the filter processing unit 123 of the image restoration processing apparatus 120 is a real space filter. Therefore, the number of taps necessary for the filter may be determined in real space. Since the image restoration filter is a filter that corrects image degradation due to the point spread function PSF, it is sufficient that an area equivalent to the area where the point spread function PSF is distributed in real space can be secured. That is, the number of taps necessary for the image restoration filter is the number of taps in the above-described region. Since the real space and the frequency space are reciprocal, the number of taps determined in the real space can be used in the frequency space.

  10 and 11 are explanatory diagrams of the number of taps and the frequency pitch. FIG. 10 shows a case where the number of taps is taken in an area that is sufficiently larger than the spatial distribution of the point spread function PSF. FIG. 11 shows a case where the number of taps is taken in a region substantially equivalent to the spatial distribution of the point spread function PSF with respect to the same point spread function PSF as in FIG. In FIG. 10, the number of taps in the real space corresponds to the minimum frequency pitch in the frequency space. As shown in FIG. 11, reducing the number of taps in the real space means that the frequency space is roughly sampled, and indicates that the minimum frequency pitch is increased. At this time, the value of the Nyquist frequency in the frequency space does not change.

  The image recovery information holding unit 121 stores coefficient information (coefficient data) output from the coefficient calculation apparatus 101, tap number information, lens ID, shooting conditions, and Nyquist frequency information of the image sensor. The OTF reconstruction unit 122 acquires from the camera 110 the lens ID at the time of shooting, shooting conditions, and Nyquist frequency information of the image sensor. In order to read out the tap number information, the lens ID, the imaging conditions, and the Nyquist frequency information of the image sensor corresponding to each of the above conditions from the image recovery information holding unit 121, and to create an image recovery filter using these information A reconstructed OTF used in the above is generated.

  Next, a method for creating the reconstructed OTF will be described in detail with reference to FIG. FIG. 12 is an explanatory diagram of the OTF reconstruction unit 122. In FIG. 12, it is assumed that the Nyquist frequencies in the meridional and sagittal directions necessary for creating the reconstructed OTF are fuc_rm and fvc_im, and the tap numbers in the meridional and sagittal directions are Mx and My. However, 0 <fum_n ≦ fum, 0 <fvm_n ≦ fvm, 0 <Mx ≦ Nx, 0 <My ≦ Ny with respect to fum and fvm, and Mx and My are odd numbers.

  Here, x and y in formulas (7) and (8) are replaced with u and m, and the domain of −fum_n / fum ≦ u ≦ 1 and −fvm_n / fvm ≦ v ≦ 1 is set to [Mx / 2] +1 , [My / 2] + 1 tap sampling. Then, when the coefficient is substituted into equation (8), a quarter region of the reconstructed OTF is created. The above procedure is performed for both the real part 122-1-1 and the imaginary part 122-2-1 of the reconstructed OTF.

  Next, from the reconstructed OTF of the ¼ area of the real part and the imaginary part created by the above method, the definition areas are −fum_n / fum ≦ u ≦ fum_n / fum, −fvm_n / fvm ≦ v ≦ fvm_n / fvm A reconstructed OTF having tap numbers Mx and My is created.

  First, the real part creation method of the reconstructed OTF will be described. Using the real part 122-1-1 of the reconstructed OTF, the real part of the reconstructed OTF is converted into an area of 1 to [Mx / 2] +1 rows and 1 to [My / 2] columns, and 1 to [Mx / 2] The area is divided into +1 row and [My / 2] +1 column. Next, as in the real part 122-1-2, the numerical data in the region of 1 to [Mx / 2] +1 rows and 1 to [My / 2] columns is changed to 1 to [Mx / 2] +1 rows, [My / 2] is substituted into the region of 1 to [Mx / 2] +1 rows and [My / 2] +2 to My columns so as to be line symmetric with respect to the region of +1 column.

  Further, like the real part 122-1-3, the reconstructed OTF of the ½ area created by the real part 122-1-2 is 1 to [Mx / 2] rows, 1 to My column area, and [Mx / 2] The area is divided into +1 row and 1 to My columns. Then, the numerical data of the area of 1 to [Mx / 2] rows and 1 to My columns is [Mx / 2] so as to be symmetrical with respect to the area of [Mx / 2] +1 rows and 1 to My columns. Substitute in the area of +2 rows and 1 to My columns. The imaginary part of the reconstructed OTF can be created in the same way as the real part. However, in the imaginary part 122-2-3, it is necessary to replace the positive and negative values. The creation method as described above is possible due to the characteristics of the optical transfer function OTF.

  FIG. 13 is a relationship diagram (reconstruction OTF) between the Nyquist frequency of the reconfiguration OTF and the number of taps. As described above, the Nyquist frequency is a parameter determined from the spatial resolution of the image sensor 111, and the number of taps is a parameter that depends on the point spread function PSF of the imaging lens 112. A desired reconstructed OTF is generated using these two parameters and the coefficient data described above. In FIG. 13, the Nyquist frequency is f_nyq1> f_nyq2, and the number of taps is N> M1> M2. As shown in FIG. 13, the Nyquist frequency and the number of taps can be controlled to desired values.

  As described above, the optical transfer function OTF corresponding to the combination of the imaging element and the lens and the imaging condition is converted into coefficient data and stored in the image recovery processing device 120, whereby the image recovery process corresponding to the imaging condition at the time of imaging is performed. Is possible. Further, as shown in FIG. 6, since the data of the entire image area can be recovered from a small amount of coefficient data with an appropriate number of taps, the amount of retained data can be reduced.

  Next, another image processing method (modified example) in the present embodiment will be described with reference to FIG. FIG. 14 is a flowchart of another image processing method in this embodiment. In the image processing method of FIG. 14, in step S26, the step of applying a rotationally asymmetric transfer function with respect to the center of the screen or the optical axis is added to the developed optical transfer function OTF. It is different from the processing method. Steps S21 to S25 and steps S27 to S29 in FIG. 14 are the same as steps S11 to S18 in FIG.

  Through the steps S21 to S25 in FIG. 14, for example, the optical transfer function OTF is rearranged as shown in FIG. 6B. Here, when considering a rotationally asymmetric transfer function such as a transfer function of an optical low-pass filter or a transfer function of a pixel aperture shape of the image sensor 111, each optical transfer function OTF in the state of FIG. Apply (add) a rotationally asymmetric transfer function. Further, in FIG. 6B, the optical transfer function OTF is developed in a quarter region of the image. However, for example, the optical transfer function OTF is developed over the entire image according to the symmetry of the transfer function. You may apply from. As shown in FIG. 14, in this embodiment, an image restoration filter can be generated using an optical transfer function OTF to which a rotationally asymmetric transfer function is applied.

  Next, with reference to FIG. 15, an imaging apparatus according to the second embodiment of the present invention will be described. FIG. 15 is a configuration diagram of the imaging apparatus 200 in the present embodiment. The image capturing apparatus 200 is installed with an image processing program for performing image restoration processing of the captured image (an image processing method similar to that in the first embodiment). This image restoration processing is performed by the image processing unit 204 (inside the image capturing apparatus 200). Executed by an image processing apparatus).

  The imaging apparatus 200 includes an imaging optical system 201 (lens) and an imaging apparatus main body (camera main body). The imaging optical system 201 includes a diaphragm 201a and a focus lens 201b, and is configured integrally with an imaging apparatus main body (camera main body). However, the present embodiment is not limited to this, and can also be applied to an imaging apparatus in which the imaging optical system 201 is attached to the imaging apparatus body in a replaceable manner.

  The image sensor 202 photoelectrically converts a subject image (imaging light) obtained via the imaging optical system 201 to generate a captured image. That is, the subject image is photoelectrically converted by the image sensor 202 and converted into an analog signal (electric signal). The analog signal is converted into a digital signal by the A / D converter 203, and the digital signal is input to the image processing unit 204.

  The image processing unit 204 (image processing apparatus) performs predetermined processing on the digital signal and performs the above-described image restoration processing. First, the image processing unit 204 acquires imaging condition information of the imaging apparatus from the state detection unit 207. The imaging condition information is information related to an aperture, a shooting distance, a focal length of a zoom lens, and the like. The state detection unit 207 can acquire the imaging condition information directly from the system controller 210, but is not limited to this. For example, the imaging condition information regarding the imaging optical system 201 can be acquired from the imaging optical system control unit 206. The processing flow (image processing method) of the image restoration process of the present embodiment is the same as that of Embodiment 1 described with reference to FIG. 1 or FIG.

  Coefficient data for generating the reconstructed OTF is held in the storage unit 208. The output image processed by the image processing unit 204 is stored in the image recording medium 209 in a predetermined format. The display unit 205 displays an image obtained by performing a predetermined display process on the image that has been subjected to the image restoration process of the present embodiment. However, the present invention is not limited to this, and an image that has been subjected to simple processing for high-speed display may be displayed on the display unit 205.

  A series of controls in this embodiment is performed by the system controller 210, and mechanical driving of the imaging optical system 201 is performed by the imaging optical system control unit 206 based on an instruction from the system controller 210. The imaging optical system control unit 206 controls the aperture diameter of the aperture 201a as the F number imaging state setting. In addition, the imaging optical system control unit 206 controls the position of the focus lens 201b by an unillustrated autofocus (AF) mechanism or manual manual focus mechanism in order to perform focus adjustment according to the subject distance. Note that functions such as aperture diameter control and manual focus of the aperture 201a may not be executed according to the specifications of the imaging apparatus 200.

  The imaging optical system 201 may include an optical element such as a low-pass filter or an infrared cut filter. However, when an element that affects the characteristics of an optical transfer function (OTF) such as a low-pass filter is used, an image restoration filter is used. It may be necessary to consider at the time of creation. The infrared cut filter also affects each PSF of the RGB channel, particularly the PSF of the R channel, which is the integral value of the point spread function (PSF) of the spectral wavelength, and therefore needs to be considered at the time of creating the image restoration filter. It may become. In this case, as described with reference to FIG. 14, a rotationally asymmetric transfer function is added after rearranging the optical transfer function (OTF).

  In the present embodiment, the coefficient data stored in the storage unit of the imaging apparatus is used. However, as a modification, for example, the imaging apparatus is configured to acquire coefficient data stored in a storage medium such as a memory card. May be.

  Next, with reference to FIG. 16, an image processing apparatus and an image processing system in Embodiment 3 of the present invention will be described. FIG. 16 is a configuration diagram of an image processing system 300 in the present embodiment. Note that the processing flow (image processing method) of the image restoration processing of the present embodiment is the same as that of the first embodiment described with reference to FIG. 1 or FIG.

  In FIG. 16, an image processing apparatus 301 is a computer device equipped with image processing software 306 for causing a computer (information processing apparatus) to execute the image processing method of this embodiment. The imaging device 302 is an imaging device such as a camera, a microscope, an endoscope, or a scanner. The storage medium 303 is a storage unit that stores captured images, such as a semiconductor memory, a hard disk, or a server on a network.

  The image processing apparatus 301 acquires captured image data from the imaging device 302 or the storage medium 303, and outputs the image data subjected to predetermined image processing to one or more of the output device 305, the imaging device 302, and the storage medium 303. Output. Further, the output destination can be stored in a storage unit built in the image processing apparatus 301. The output device 305 is a printer, for example.

  A display device 304 that is a monitor is connected to the image processing apparatus 301. Therefore, the user can perform the image processing work through the display device 304 and evaluate the corrected image. The image processing software 306 performs image restoration processing (image processing method) of the present embodiment, and performs development and other image processing as necessary.

Next, with reference to FIG. 17, data contents for performing the image processing method of the present embodiment and data transfer between devices will be described. FIG. 17 is an explanatory diagram of an information set (data contents). In the present embodiment, the information set includes the following information.
[Correction control information]
The correction control information is composed of setting information regarding which of the image processing apparatus 301, the imaging device 302, and the output device 305 performs correction processing, and selection information of data to be transmitted to other devices along with the setting information. Is done. For example, when generating a restored image in the imaging device 302, there is no need to transmit coefficient data. On the other hand, when an image before correction is acquired from the imaging device 302 and a restored image is generated by the image processing apparatus 301, coefficient data, tap number information, lens ID, imaging element Nyquist information, imaging conditions, and the like are transmitted. Alternatively, based on the imaging device information and imaging condition information of the imaging device 302 used for shooting, the coefficient data stored in the image processing apparatus 301 is selected in advance, and corrected and used as necessary. Also good.
[Imaging equipment information]
The imaging device information is identification information of the imaging device 302. When the lens is exchangeably attached to the camera body, the identification information includes the combination. For example, the lens ID described above corresponds to this.
[Imaging condition information]
The imaging condition information is information related to the state of the imaging device 302 at the time of shooting. For example, the focal length, aperture value, shooting distance, ISO sensitivity, or white balance setting.
[Imaging device individual information]
The imaging device individual information is identification information of each imaging device with respect to the above-described imaging device information. Due to variations in manufacturing errors, there are individual variations in the optical transfer function (OTF) of the imaging device 302. For this reason, the imaging device individual information is information that is effective for setting optimum correction parameters individually. The correction parameter is a set value such as coefficient data, a correction coefficient for the image restoration filter, a gain value for edge enhancement, distortion correction, or shading correction. When the state of the manufacturing error is known in advance, more accurate image restoration processing can be performed by correcting initial data used for image restoration filter generation.
[Coefficient data group]
The coefficient data group is a set of coefficient data used for correcting phase deterioration. When a device that performs image restoration processing does not have coefficient data, it is necessary to transmit coefficient data from another device.
[User setting information]
The user setting information is a correction parameter or a correction value of the correction parameter for correcting the sharpness (the degree of recovery) according to the user's preference. Although the user can variably set the correction parameter, using the user setting information can always obtain a desired output image as an initial value. Further, the user setting information is preferably updated to the sharpness most preferred by the user using the learning function based on the correction parameter history set by the user. Further, the provider (maker) of the imaging device 302 can provide preset values corresponding to some sharpness patterns via the network or the storage medium 303.

  The information set described above is preferably attached to individual image data. By attaching necessary correction information to the image data, correction processing can be performed by any device equipped with the image processing apparatus of this embodiment. Further, the contents of the information set can be appropriately selected automatically or manually as necessary.

  According to each of the above embodiments, the image is rearranged in the image in the state of the optical transfer function and then converted into the real space image restoration filter. A corresponding image restoration filter can be generated. In addition, since the image restoration filter can be generated with high accuracy, the amount of data related to the optical transfer function that needs to be held in advance can be reduced. Therefore, according to each embodiment, it is possible to provide an image processing method capable of acquiring a restored image corrected with high accuracy while reducing the amount of information used in the image restoration process.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

120: Image recovery processing device 121: Image recovery information holding unit 122: OTF reconstruction unit 123: Filter processing unit

Claims (12)

An image processing method for performing image restoration processing of a captured image,
Generating a first optical transfer function of a plurality of imaging optical systems according to the position of the captured image using coefficient data corresponding to the imaging condition of the captured image;
Rotating the first optical transfer function around the center of the captured image or the optical axis of the imaging optical system to generate a plurality of second optical transfer functions;
Generating an image restoration filter based on the first optical transfer function and the second optical transfer function;
And a step of performing image restoration processing of the photographed image using the image restoration filter.   2. The image processing according to claim 1, wherein at least two of the first optical transfer functions are optical transfer functions corresponding to positions on a straight line passing through a center of the captured image. Method.   The image processing method according to claim 2, wherein all of the plurality of first optical transfer functions are optical transfer functions corresponding to positions on the straight line. The image restoration filter includes a first image restoration filter at a first position of the photographed image, and a second image restoration filter at a second position of the photographed image;
The first image restoration filter is generated using the first and second optical transfer functions;
The image processing method according to claim 1, wherein the second image restoration filter is generated based on the first image restoration filter. Using the second optical transfer function and a transfer function that is rotationally asymmetric with respect to the center of the captured image or the optical axis of the imaging optical system, to generate a rotationally asymmetric optical transfer function;
The image processing method according to claim 1, wherein the image restoration filter is generated using the rotationally asymmetric optical transfer function.   The image processing method according to claim 5, wherein the rotationally asymmetric transfer function is a transfer function of an optical low-pass filter.   The image processing method according to claim 6, wherein the rotationally asymmetric transfer function is a transfer function of a pixel aperture shape of an image sensor.   The image processing method according to claim 1, wherein the coefficient data is coefficient data of a function that approximates a design value or a measurement value of an optical transfer function of the imaging optical system. . In the captured image, a region that is point-symmetric with respect to the center of the captured image or the optical axis of the imaging optical system is defined as a first region and a second region, and a straight line that passes through the first region and the center of the captured image. When the line-symmetric region is the third region,
9. The method according to claim 1, further comprising the step of generating an optical transfer function of the second region or the third region using an optical transfer function of the first region. The image processing method according to item. An image processing apparatus that performs image restoration processing of a captured image,
First optical transfer function generation means for generating first optical transfer functions of a plurality of imaging optical systems according to the positions of the captured images using coefficient data corresponding to the imaging conditions of the captured images;
Second optical transfer function generating means for generating a plurality of second optical transfer functions by rotating the first optical transfer function around the center of the captured image or the optical axis of the imaging optical system;
Image restoration filter generating means for generating an image restoration filter based on the first optical transfer function and the second optical transfer function;
And an image recovery means for performing image recovery processing of the photographed image using the image recovery filter. An imaging device that performs image restoration processing of a captured image,
First optical transfer function generation means for generating first optical transfer functions of a plurality of imaging optical systems according to the positions of the captured images using coefficient data corresponding to the imaging conditions of the captured images;
Second optical transfer function generating means for generating a plurality of second optical transfer functions by rotating the first optical transfer function around the center of the captured image or the optical axis of the imaging optical system;
Image restoration filter generating means for generating an image restoration filter based on the first optical transfer function and the second optical transfer function;
And an image recovery means for performing image recovery processing of the captured image using the image recovery filter. An image processing program for performing image restoration processing of a captured image,
Generating a first optical transfer function of a plurality of imaging optical systems according to the position of the captured image using coefficient data corresponding to the imaging condition of the captured image;
Rotating the first optical transfer function around the center of the captured image or the optical axis of the imaging optical system to generate a plurality of second optical transfer functions;
Generating an image restoration filter based on the first optical transfer function and the second optical transfer function;
An image processing program causing an information processing apparatus to execute an image restoration process of the captured image using the image restoration filter.